Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250903 20:05:04.190940 26885 test_util.cc:276] Using random seed: 517318060
2025-09-03T20:05:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-03T20:05:04Z Disabled control of system clock
I20250903 20:05:04.228973 26885 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskH7eH49/build/asan/bin/kudu
/tmp/dist-test-taskH7eH49/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.65.126:39015
--webserver_interface=127.26.65.126
--webserver_port=0
--builtin_ntp_servers=127.26.65.84:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.65.126:39015,127.26.65.125:44059,127.26.65.124:40911
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250903 20:05:04.559077 26893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 20:05:04.559401 26893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 20:05:04.559473 26893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 20:05:04.567853 26893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 20:05:04.567966 26893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 20:05:04.568024 26893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 20:05:04.568084 26893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 20:05:04.578365 26893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.65.84:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.26.65.126:39015,127.26.65.125:44059,127.26.65.124:40911
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.65.126:39015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.65.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 19:52:33 UTC on 24a791456cd2
build id 7884
ASAN enabled
I20250903 20:05:04.580114 26893 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 20:05:04.582036 26893 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 20:05:04.591444 26899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 20:05:04.591449 26898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 20:05:04.591653 26901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 20:05:04.592695 26893 server_base.cc:1047] running on GCE node
I20250903 20:05:04.593498 26893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 20:05:04.595139 26893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 20:05:04.596339 26893 hybrid_clock.cc:648] HybridClock initialized: now 1756929904596287 us; error 44 us; skew 500 ppm
I20250903 20:05:04.596704 26893 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 20:05:04.598937 26893 webserver.cc:480] Webserver started at http://127.26.65.126:33509/ using document root <none> and password file <none>
I20250903 20:05:04.599700 26893 fs_manager.cc:362] Metadata directory not provided
I20250903 20:05:04.599807 26893 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 20:05:04.600195 26893 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 20:05:04.602771 26893 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/data/instance:
uuid: "8857b5d4ea1e4f45aafb8e029837cca2"
format_stamp: "Formatted at 2025-09-03 20:05:04 on dist-test-slave-fs2w"
I20250903 20:05:04.603586 26893 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/wal/instance:
uuid: "8857b5d4ea1e4f45aafb8e029837cca2"
format_stamp: "Formatted at 2025-09-03 20:05:04 on dist-test-slave-fs2w"
I20250903 20:05:04.609249 26893 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250903 20:05:04.613066 26907 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 20:05:04.614523 26893 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250903 20:05:04.614740 26893 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/data,/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/wal
uuid: "8857b5d4ea1e4f45aafb8e029837cca2"
format_stamp: "Formatted at 2025-09-03 20:05:04 on dist-test-slave-fs2w"
I20250903 20:05:04.614951 26893 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 20:05:04.646256 26893 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 20:05:04.647261 26893 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 20:05:04.647591 26893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 20:05:04.668540 26893 rpc_server.cc:307] RPC server started. Bound to: 127.26.65.126:39015
I20250903 20:05:04.668581 26958 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.65.126:39015 every 8 connection(s)
I20250903 20:05:04.670301 26893 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/data/info.pb
I20250903 20:05:04.674660 26959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:04.678623 26885 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskH7eH49/build/asan/bin/kudu as pid 26893
I20250903 20:05:04.678946 26885 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/wal/instance
I20250903 20:05:04.682441 26885 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskH7eH49/build/asan/bin/kudu
/tmp/dist-test-taskH7eH49/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.65.125:44059
--webserver_interface=127.26.65.125
--webserver_port=0
--builtin_ntp_servers=127.26.65.84:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.65.126:39015,127.26.65.125:44059,127.26.65.124:40911
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250903 20:05:04.682766 26959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } has no permanent_uuid. Determining permanent_uuid...
I20250903 20:05:04.704881 26959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } has no permanent_uuid. Determining permanent_uuid...
W20250903 20:05:04.705977 26909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.65.125:44059: connect: Connection refused (error 111)
W20250903 20:05:04.707582 26959 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.26.65.125:44059: Network error: Client connection negotiation failed: client connection to 127.26.65.125:44059: connect: Connection refused (error 111)
I20250903 20:05:04.757009 26959 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } attempt: 1
W20250903 20:05:04.759146 26959 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.26.65.125:44059: Network error: Client connection negotiation failed: client connection to 127.26.65.125:44059: connect: Connection refused (error 111)
I20250903 20:05:04.827493 26959 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } attempt: 2
W20250903 20:05:04.829371 26959 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.26.65.125:44059: Network error: Client connection negotiation failed: client connection to 127.26.65.125:44059: connect: Connection refused (error 111)
I20250903 20:05:04.920684 26959 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } attempt: 3
W20250903 20:05:04.922569 26959 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.26.65.125:44059: Network error: Client connection negotiation failed: client connection to 127.26.65.125:44059: connect: Connection refused (error 111)
W20250903 20:05:05.015565 26961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 20:05:05.015900 26961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 20:05:05.015982 26961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 20:05:05.024948 26961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 20:05:05.025061 26961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 20:05:05.025118 26961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 20:05:05.025167 26961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 20:05:05.035336 26961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.65.84:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.26.65.126:39015,127.26.65.125:44059,127.26.65.124:40911
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.65.125:44059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.26.65.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 19:52:33 UTC on 24a791456cd2
build id 7884
ASAN enabled
I20250903 20:05:05.037081 26961 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 20:05:05.038981 26961 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 20:05:05.046808 26970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 20:05:05.046836 26969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 20:05:05.047767 26972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 20:05:05.048085 26961 server_base.cc:1047] running on GCE node
I20250903 20:05:05.048842 26961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 20:05:05.050391 26961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 20:05:05.051594 26961 hybrid_clock.cc:648] HybridClock initialized: now 1756929905051556 us; error 35 us; skew 500 ppm
I20250903 20:05:05.051940 26961 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 20:05:05.054134 26961 webserver.cc:480] Webserver started at http://127.26.65.125:41613/ using document root <none> and password file <none>
I20250903 20:05:05.054912 26961 fs_manager.cc:362] Metadata directory not provided
I20250903 20:05:05.055012 26961 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 20:05:05.055392 26961 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 20:05:05.057871 26961 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/data/instance:
uuid: "35dd69f3fa7a4fd8843b97f8a5d68296"
format_stamp: "Formatted at 2025-09-03 20:05:05 on dist-test-slave-fs2w"
I20250903 20:05:05.058691 26961 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/wal/instance:
uuid: "35dd69f3fa7a4fd8843b97f8a5d68296"
format_stamp: "Formatted at 2025-09-03 20:05:05 on dist-test-slave-fs2w"
I20250903 20:05:05.064167 26961 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250903 20:05:05.065838 26959 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } attempt: 4
W20250903 20:05:05.067862 26959 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.26.65.125:44059: Network error: Client connection negotiation failed: client connection to 127.26.65.125:44059: connect: Connection refused (error 111)
I20250903 20:05:05.068087 26978 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 20:05:05.069424 26961 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20250903 20:05:05.069587 26961 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/data,/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/wal
uuid: "35dd69f3fa7a4fd8843b97f8a5d68296"
format_stamp: "Formatted at 2025-09-03 20:05:05 on dist-test-slave-fs2w"
I20250903 20:05:05.069777 26961 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 20:05:05.101047 26961 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 20:05:05.102033 26961 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 20:05:05.102344 26961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 20:05:05.122897 26961 rpc_server.cc:307] RPC server started. Bound to: 127.26.65.125:44059
I20250903 20:05:05.122933 27029 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.65.125:44059 every 8 connection(s)
I20250903 20:05:05.124531 26961 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/data/info.pb
I20250903 20:05:05.128651 27030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:05.132117 26885 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskH7eH49/build/asan/bin/kudu as pid 26961
I20250903 20:05:05.132272 26885 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/wal/instance
I20250903 20:05:05.133327 26885 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskH7eH49/build/asan/bin/kudu
/tmp/dist-test-taskH7eH49/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.65.124:40911
--webserver_interface=127.26.65.124
--webserver_port=0
--builtin_ntp_servers=127.26.65.84:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.65.126:39015,127.26.65.125:44059,127.26.65.124:40911
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250903 20:05:05.134898 27030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } has no permanent_uuid. Determining permanent_uuid...
I20250903 20:05:05.150581 27030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } has no permanent_uuid. Determining permanent_uuid...
I20250903 20:05:05.160821 27030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } has no permanent_uuid. Determining permanent_uuid...
W20250903 20:05:05.162031 26980 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.65.124:40911: connect: Connection refused (error 111)
W20250903 20:05:05.163821 27030 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.26.65.124:40911: Network error: Client connection negotiation failed: client connection to 127.26.65.124:40911: connect: Connection refused (error 111)
I20250903 20:05:05.213200 27030 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } attempt: 1
W20250903 20:05:05.215328 27030 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.26.65.124:40911: Network error: Client connection negotiation failed: client connection to 127.26.65.124:40911: connect: Connection refused (error 111)
I20250903 20:05:05.283622 27030 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } attempt: 2
W20250903 20:05:05.285578 27030 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.26.65.124:40911: Network error: Client connection negotiation failed: client connection to 127.26.65.124:40911: connect: Connection refused (error 111)
I20250903 20:05:05.367136 26959 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } attempt: 5
I20250903 20:05:05.373976 26959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } has no permanent_uuid. Determining permanent_uuid...
W20250903 20:05:05.375748 26959 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.26.65.124:40911: Network error: Client connection negotiation failed: client connection to 127.26.65.124:40911: connect: Connection refused (error 111)
I20250903 20:05:05.376796 27030 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } attempt: 3
W20250903 20:05:05.378428 27030 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.26.65.124:40911: Network error: Client connection negotiation failed: client connection to 127.26.65.124:40911: connect: Connection refused (error 111)
I20250903 20:05:05.427065 26959 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } attempt: 1
W20250903 20:05:05.429000 26959 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.26.65.124:40911: Network error: Client connection negotiation failed: client connection to 127.26.65.124:40911: connect: Connection refused (error 111)
W20250903 20:05:05.479651 27032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 20:05:05.480015 27032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 20:05:05.480131 27032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 20:05:05.488456 27032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 20:05:05.488580 27032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 20:05:05.488648 27032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 20:05:05.488739 27032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 20:05:05.497269 26959 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } attempt: 2
W20250903 20:05:05.499228 26959 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.26.65.124:40911: Network error: Client connection negotiation failed: client connection to 127.26.65.124:40911: connect: Connection refused (error 111)
I20250903 20:05:05.499488 27032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.65.84:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.26.65.126:39015,127.26.65.125:44059,127.26.65.124:40911
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.65.124:40911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.26.65.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 19:52:33 UTC on 24a791456cd2
build id 7884
ASAN enabled
I20250903 20:05:05.501204 27032 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 20:05:05.503108 27032 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 20:05:05.512020 27041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 20:05:05.512169 27043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 20:05:05.511994 27040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 20:05:05.513284 27032 server_base.cc:1047] running on GCE node
I20250903 20:05:05.514073 27032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 20:05:05.515560 27032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 20:05:05.516762 27032 hybrid_clock.cc:648] HybridClock initialized: now 1756929905516717 us; error 43 us; skew 500 ppm
I20250903 20:05:05.517086 27032 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 20:05:05.519235 27032 webserver.cc:480] Webserver started at http://127.26.65.124:33543/ using document root <none> and password file <none>
I20250903 20:05:05.519986 27032 fs_manager.cc:362] Metadata directory not provided
I20250903 20:05:05.520092 27032 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 20:05:05.520503 27032 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 20:05:05.521651 27030 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } attempt: 4
W20250903 20:05:05.523473 27030 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.26.65.124:40911: Network error: Client connection negotiation failed: client connection to 127.26.65.124:40911: connect: Connection refused (error 111)
I20250903 20:05:05.523214 27032 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-2/data/instance:
uuid: "35bdb0090d8747059b5ee103e786e979"
format_stamp: "Formatted at 2025-09-03 20:05:05 on dist-test-slave-fs2w"
I20250903 20:05:05.524106 27032 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-2/wal/instance:
uuid: "35bdb0090d8747059b5ee103e786e979"
format_stamp: "Formatted at 2025-09-03 20:05:05 on dist-test-slave-fs2w"
I20250903 20:05:05.530252 27032 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.000s sys 0.006s
I20250903 20:05:05.533973 27049 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 20:05:05.535444 27032 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20250903 20:05:05.535660 27032 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-2/data,/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-2/wal
uuid: "35bdb0090d8747059b5ee103e786e979"
format_stamp: "Formatted at 2025-09-03 20:05:05 on dist-test-slave-fs2w"
I20250903 20:05:05.535876 27032 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 20:05:05.551147 27032 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 20:05:05.552048 27032 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 20:05:05.552311 27032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 20:05:05.573624 27032 rpc_server.cc:307] RPC server started. Bound to: 127.26.65.124:40911
I20250903 20:05:05.573645 27100 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.65.124:40911 every 8 connection(s)
I20250903 20:05:05.575369 27032 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-2/data/info.pb
I20250903 20:05:05.579563 27101 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:05.583472 26885 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskH7eH49/build/asan/bin/kudu as pid 27032
I20250903 20:05:05.583717 26885 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-2/wal/instance
I20250903 20:05:05.587471 27101 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } has no permanent_uuid. Determining permanent_uuid...
I20250903 20:05:05.604420 27101 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } has no permanent_uuid. Determining permanent_uuid...
I20250903 20:05:05.605638 26959 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } attempt: 3
I20250903 20:05:05.613200 27101 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } has no permanent_uuid. Determining permanent_uuid...
I20250903 20:05:05.623236 26959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2: Bootstrap starting.
I20250903 20:05:05.625579 27101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Bootstrap starting.
I20250903 20:05:05.626349 26959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:05.627960 26959 log.cc:826] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2: Log is configured to *not* fsync() on all Append() calls
I20250903 20:05:05.629005 27101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:05.629989 27101 log.cc:826] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Log is configured to *not* fsync() on all Append() calls
I20250903 20:05:05.630607 26959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2: No bootstrap required, opened a new log
I20250903 20:05:05.632324 27101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: No bootstrap required, opened a new log
I20250903 20:05:05.636438 26959 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } }
I20250903 20:05:05.636744 26959 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:05.636857 26959 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8857b5d4ea1e4f45aafb8e029837cca2, State: Initialized, Role: FOLLOWER
I20250903 20:05:05.637516 26959 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } }
I20250903 20:05:05.637674 27101 raft_consensus.cc:357] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } }
I20250903 20:05:05.638020 27101 raft_consensus.cc:383] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:05.638139 27101 raft_consensus.cc:738] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35bdb0090d8747059b5ee103e786e979, State: Initialized, Role: FOLLOWER
I20250903 20:05:05.638561 27112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:05.638900 27112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 20:05:05.638893 27101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } }
I20250903 20:05:05.639891 27113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:05.640267 27113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 20:05:05.640817 26959 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 20:05:05.641124 27101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [sys.catalog]: configured and running, proceeding with master startup.
W20250903 20:05:05.650492 26913 tablet.cc:2378] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 20:05:05.655956 27136 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20250903 20:05:05.657508 27128 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250903 20:05:05.657658 27128 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250903 20:05:05.652081 26893 master_runner.cc:186] Error getting master registration for 127.26.65.125:44059: OK, instance_id { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" instance_seqno: 1756929905117090 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250903 20:05:05.660398 26893 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 20:05:05.662911 27032 master_runner.cc:186] Error getting master registration for 127.26.65.125:44059: OK, instance_id { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" instance_seqno: 1756929905117090 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250903 20:05:05.666857 27032 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 20:05:05.764245 27113 raft_consensus.cc:491] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:05.764550 27113 raft_consensus.cc:513] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } }
I20250903 20:05:05.766068 27113 leader_election.cc:290] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8857b5d4ea1e4f45aafb8e029837cca2 (127.26.65.126:39015), 35dd69f3fa7a4fd8843b97f8a5d68296 (127.26.65.125:44059)
I20250903 20:05:05.766980 26934 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35bdb0090d8747059b5ee103e786e979" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" is_pre_election: true
I20250903 20:05:05.767582 26934 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35bdb0090d8747059b5ee103e786e979 in term 0.
I20250903 20:05:05.768255 27053 leader_election.cc:304] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35bdb0090d8747059b5ee103e786e979, 8857b5d4ea1e4f45aafb8e029837cca2; no voters:
I20250903 20:05:05.768716 27113 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:05.768887 27113 raft_consensus.cc:491] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:05.769008 27113 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:05.771495 27113 raft_consensus.cc:513] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } }
I20250903 20:05:05.772614 26934 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35bdb0090d8747059b5ee103e786e979" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8857b5d4ea1e4f45aafb8e029837cca2"
I20250903 20:05:05.772864 26934 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:05.772892 27113 leader_election.cc:290] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [CANDIDATE]: Term 1 election: Requested vote from peers 8857b5d4ea1e4f45aafb8e029837cca2 (127.26.65.126:39015), 35dd69f3fa7a4fd8843b97f8a5d68296 (127.26.65.125:44059)
I20250903 20:05:05.776492 26934 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35bdb0090d8747059b5ee103e786e979 in term 1.
I20250903 20:05:05.777051 27053 leader_election.cc:304] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35bdb0090d8747059b5ee103e786e979, 8857b5d4ea1e4f45aafb8e029837cca2; no voters:
I20250903 20:05:05.777433 27113 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:05.777978 27113 raft_consensus.cc:695] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [term 1 LEADER]: Becoming Leader. State: Replica: 35bdb0090d8747059b5ee103e786e979, State: Running, Role: LEADER
I20250903 20:05:05.778628 27113 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } }
I20250903 20:05:05.781705 27137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35bdb0090d8747059b5ee103e786e979. Latest consensus state: current_term: 1 leader_uuid: "35bdb0090d8747059b5ee103e786e979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:05.782049 27137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [sys.catalog]: This master's current role is: LEADER
I20250903 20:05:05.783208 27139 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 20:05:05.790140 27139 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 20:05:05.802408 26934 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [term 1 FOLLOWER]: Refusing update from remote peer 35bdb0090d8747059b5ee103e786e979: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 20:05:05.803546 27137 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250903 20:05:05.804776 27055 tablet.cc:2378] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 20:05:05.822696 27030 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } attempt: 5
I20250903 20:05:05.830132 27112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35bdb0090d8747059b5ee103e786e979. Latest consensus state: current_term: 1 leader_uuid: "35bdb0090d8747059b5ee103e786e979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:05.833968 27112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 20:05:05.843673 27030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296: Bootstrap starting.
I20250903 20:05:05.844216 27137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "35bdb0090d8747059b5ee103e786e979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:05.844467 27137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [sys.catalog]: This master's current role is: LEADER
I20250903 20:05:05.850239 27030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:05.851895 27030 log.cc:826] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296: Log is configured to *not* fsync() on all Append() calls
I20250903 20:05:05.853477 27112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "35bdb0090d8747059b5ee103e786e979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:05.853725 27112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 20:05:05.854476 27139 catalog_manager.cc:1349] Generated new cluster ID: 07b8834eda114abdb992ba4c63cadc24
I20250903 20:05:05.854597 27139 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 20:05:05.874344 27030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296: No bootstrap required, opened a new log
I20250903 20:05:05.877027 27139 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 20:05:05.878983 27139 catalog_manager.cc:1506] Loading token signing keys...
I20250903 20:05:05.899353 27030 raft_consensus.cc:357] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } }
I20250903 20:05:05.899930 27030 raft_consensus.cc:383] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:05.900095 27030 raft_consensus.cc:738] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35dd69f3fa7a4fd8843b97f8a5d68296, State: Initialized, Role: FOLLOWER
I20250903 20:05:05.900979 27030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } }
I20250903 20:05:05.911134 27139 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Generated new TSK 0
I20250903 20:05:05.912353 27139 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 20:05:05.918975 27147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:05.919425 27147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 20:05:05.920725 27030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 20:05:05.920804 27004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35bdb0090d8747059b5ee103e786e979" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296"
I20250903 20:05:05.921337 27004 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:05.922528 27005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35bdb0090d8747059b5ee103e786e979" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" is_pre_election: true
I20250903 20:05:05.927376 27004 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35bdb0090d8747059b5ee103e786e979 in term 1.
I20250903 20:05:05.928366 27003 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 1 FOLLOWER]: Refusing update from remote peer 35bdb0090d8747059b5ee103e786e979: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 20:05:05.930513 27137 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:05.942118 26885 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskH7eH49/build/asan/bin/kudu
/tmp/dist-test-taskH7eH49/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.65.65:0
--local_ip_for_outbound_sockets=127.26.65.65
--webserver_interface=127.26.65.65
--webserver_port=0
--tserver_master_addrs=127.26.65.126:39015,127.26.65.125:44059,127.26.65.124:40911
--builtin_ntp_servers=127.26.65.84:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250903 20:05:05.969976 27147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35bdb0090d8747059b5ee103e786e979. Latest consensus state: current_term: 1 leader_uuid: "35bdb0090d8747059b5ee103e786e979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:05.970317 27147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 20:05:05.970790 27152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "35bdb0090d8747059b5ee103e786e979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:05.971187 27152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 20:05:06.109855 27164 catalog_manager.cc:1261] Loaded cluster ID: 07b8834eda114abdb992ba4c63cadc24
I20250903 20:05:06.109987 27164 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296: loading cluster ID for follower catalog manager: success
I20250903 20:05:06.114159 27164 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296: acquiring CA information for follower catalog manager: success
I20250903 20:05:06.117260 27164 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250903 20:05:06.377961 27137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "35bdb0090d8747059b5ee103e786e979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:06.378294 27137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [sys.catalog]: This master's current role is: LEADER
I20250903 20:05:06.419083 27166 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 35bdb0090d8747059b5ee103e786e979)
I20250903 20:05:06.419518 27166 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } }
I20250903 20:05:06.422276 27005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" is_pre_election: true
I20250903 20:05:06.424176 27076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "35bdb0090d8747059b5ee103e786e979" is_pre_election: true
I20250903 20:05:06.425215 26909 leader_election.cc:304] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8857b5d4ea1e4f45aafb8e029837cca2; no voters: 35bdb0090d8747059b5ee103e786e979, 35dd69f3fa7a4fd8843b97f8a5d68296
I20250903 20:05:06.426753 27166 leader_election.cc:290] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 35dd69f3fa7a4fd8843b97f8a5d68296 (127.26.65.125:44059), 35bdb0090d8747059b5ee103e786e979 (127.26.65.124:40911)
I20250903 20:05:06.427211 27166 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250903 20:05:06.448824 27151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 20:05:06.449167 27151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 20:05:06.449311 27151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 20:05:06.457980 27151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 20:05:06.458287 27151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.65.65
I20250903 20:05:06.468538 27151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.65.84:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.65.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.65.65
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.26.65.126:39015,127.26.65.125:44059,127.26.65.124:40911
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.65.65
--log_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 19:52:33 UTC on 24a791456cd2
build id 7884
ASAN enabled
I20250903 20:05:06.470288 27151 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 20:05:06.472822 27151 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 20:05:06.483412 27173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 20:05:06.483752 27172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 20:05:06.485052 27175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 20:05:06.485641 27151 server_base.cc:1047] running on GCE node
I20250903 20:05:06.486335 27151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 20:05:06.487388 27151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 20:05:06.488603 27151 hybrid_clock.cc:648] HybridClock initialized: now 1756929906488537 us; error 63 us; skew 500 ppm
I20250903 20:05:06.488919 27151 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 20:05:06.491243 27151 webserver.cc:480] Webserver started at http://127.26.65.65:44579/ using document root <none> and password file <none>
I20250903 20:05:06.492053 27151 fs_manager.cc:362] Metadata directory not provided
I20250903 20:05:06.492177 27151 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 20:05:06.492527 27151 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 20:05:06.495146 27151 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/data/instance:
uuid: "89f8f0ee68b940db9ce250c0604f934a"
format_stamp: "Formatted at 2025-09-03 20:05:06 on dist-test-slave-fs2w"
I20250903 20:05:06.495922 27151 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/instance:
uuid: "89f8f0ee68b940db9ce250c0604f934a"
format_stamp: "Formatted at 2025-09-03 20:05:06 on dist-test-slave-fs2w"
I20250903 20:05:06.501801 27151 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250903 20:05:06.505584 27181 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 20:05:06.507184 27151 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250903 20:05:06.507370 27151 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal
uuid: "89f8f0ee68b940db9ce250c0604f934a"
format_stamp: "Formatted at 2025-09-03 20:05:06 on dist-test-slave-fs2w"
I20250903 20:05:06.507583 27151 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 20:05:06.523073 27151 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 20:05:06.523952 27151 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 20:05:06.524236 27151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 20:05:06.525226 27151 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 20:05:06.527091 27151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 20:05:06.527195 27151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 20:05:06.527320 27151 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 20:05:06.527398 27151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 20:05:06.567118 27151 rpc_server.cc:307] RPC server started. Bound to: 127.26.65.65:42989
I20250903 20:05:06.567175 27293 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.65.65:42989 every 8 connection(s)
I20250903 20:05:06.568648 27151 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/data/info.pb
I20250903 20:05:06.575500 26885 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskH7eH49/build/asan/bin/kudu as pid 27151
I20250903 20:05:06.575711 26885 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/instance
I20250903 20:05:06.605520 26885 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskH7eH49/build/asan/bin/kudu
/tmp/dist-test-taskH7eH49/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.65.66:0
--local_ip_for_outbound_sockets=127.26.65.66
--webserver_interface=127.26.65.66
--webserver_port=0
--tserver_master_addrs=127.26.65.126:39015,127.26.65.125:44059,127.26.65.124:40911
--builtin_ntp_servers=127.26.65.84:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250903 20:05:06.623564 27295 heartbeater.cc:344] Connected to a master server at 127.26.65.126:39015
I20250903 20:05:06.623911 27295 heartbeater.cc:461] Registering TS with master...
I20250903 20:05:06.625486 27295 heartbeater.cc:507] Master 127.26.65.126:39015 requested a full tablet report, sending...
I20250903 20:05:06.628916 26924 ts_manager.cc:194] Registered new tserver with Master: 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:06.629455 27294 heartbeater.cc:344] Connected to a master server at 127.26.65.124:40911
I20250903 20:05:06.629616 27294 heartbeater.cc:461] Registering TS with master...
I20250903 20:05:06.630199 27294 heartbeater.cc:507] Master 127.26.65.124:40911 requested a full tablet report, sending...
I20250903 20:05:06.632014 27066 ts_manager.cc:194] Registered new tserver with Master: 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:06.632030 27296 heartbeater.cc:344] Connected to a master server at 127.26.65.125:44059
I20250903 20:05:06.632186 27296 heartbeater.cc:461] Registering TS with master...
I20250903 20:05:06.632755 27296 heartbeater.cc:507] Master 127.26.65.125:44059 requested a full tablet report, sending...
I20250903 20:05:06.634498 27066 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.65.65:44885
I20250903 20:05:06.634793 26995 ts_manager.cc:194] Registered new tserver with Master: 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:06.659690 27128 catalog_manager.cc:1261] Loaded cluster ID: 07b8834eda114abdb992ba4c63cadc24
I20250903 20:05:06.659803 27128 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2: loading cluster ID for follower catalog manager: success
I20250903 20:05:06.662572 27128 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2: acquiring CA information for follower catalog manager: success
I20250903 20:05:06.664449 27128 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250903 20:05:06.969838 27304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 20:05:06.970166 27304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 20:05:06.970292 27304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 20:05:06.978664 27304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 20:05:06.978899 27304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.65.66
I20250903 20:05:06.988768 27304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.65.84:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.65.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.65.66
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.26.65.126:39015,127.26.65.125:44059,127.26.65.124:40911
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.65.66
--log_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 19:52:33 UTC on 24a791456cd2
build id 7884
ASAN enabled
I20250903 20:05:06.990483 27304 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 20:05:06.992375 27304 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 20:05:07.002354 27309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 20:05:07.002405 27310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 20:05:07.002429 27312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 20:05:07.003455 27304 server_base.cc:1047] running on GCE node
I20250903 20:05:07.004067 27304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 20:05:07.005118 27304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 20:05:07.006330 27304 hybrid_clock.cc:648] HybridClock initialized: now 1756929907006252 us; error 55 us; skew 500 ppm
I20250903 20:05:07.006719 27304 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 20:05:07.008977 27304 webserver.cc:480] Webserver started at http://127.26.65.66:33669/ using document root <none> and password file <none>
I20250903 20:05:07.009734 27304 fs_manager.cc:362] Metadata directory not provided
I20250903 20:05:07.009828 27304 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 20:05:07.010141 27304 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 20:05:07.012872 27304 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/data/instance:
uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
format_stamp: "Formatted at 2025-09-03 20:05:07 on dist-test-slave-fs2w"
I20250903 20:05:07.013659 27304 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/instance:
uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
format_stamp: "Formatted at 2025-09-03 20:05:07 on dist-test-slave-fs2w"
I20250903 20:05:07.019124 27304 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20250903 20:05:07.022861 27318 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 20:05:07.024310 27304 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250903 20:05:07.024513 27304 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal
uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
format_stamp: "Formatted at 2025-09-03 20:05:07 on dist-test-slave-fs2w"
I20250903 20:05:07.024718 27304 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 20:05:07.049623 27304 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 20:05:07.050585 27304 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 20:05:07.051002 27304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 20:05:07.052601 27304 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 20:05:07.054652 27304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 20:05:07.054764 27304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 20:05:07.054885 27304 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 20:05:07.054967 27304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 20:05:07.088775 27304 rpc_server.cc:307] RPC server started. Bound to: 127.26.65.66:38449
I20250903 20:05:07.088848 27430 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.65.66:38449 every 8 connection(s)
I20250903 20:05:07.090338 27304 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/data/info.pb
I20250903 20:05:07.097960 26885 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskH7eH49/build/asan/bin/kudu as pid 27304
I20250903 20:05:07.098188 26885 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/instance
I20250903 20:05:07.103207 26885 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskH7eH49/build/asan/bin/kudu
/tmp/dist-test-taskH7eH49/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.65.67:0
--local_ip_for_outbound_sockets=127.26.65.67
--webserver_interface=127.26.65.67
--webserver_port=0
--tserver_master_addrs=127.26.65.126:39015,127.26.65.125:44059,127.26.65.124:40911
--builtin_ntp_servers=127.26.65.84:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250903 20:05:07.109970 27431 heartbeater.cc:344] Connected to a master server at 127.26.65.124:40911
I20250903 20:05:07.110054 27433 heartbeater.cc:344] Connected to a master server at 127.26.65.125:44059
I20250903 20:05:07.110520 27431 heartbeater.cc:461] Registering TS with master...
I20250903 20:05:07.110523 27433 heartbeater.cc:461] Registering TS with master...
I20250903 20:05:07.110603 27432 heartbeater.cc:344] Connected to a master server at 127.26.65.126:39015
I20250903 20:05:07.110796 27432 heartbeater.cc:461] Registering TS with master...
I20250903 20:05:07.111835 27431 heartbeater.cc:507] Master 127.26.65.124:40911 requested a full tablet report, sending...
I20250903 20:05:07.113235 27432 heartbeater.cc:507] Master 127.26.65.126:39015 requested a full tablet report, sending...
I20250903 20:05:07.113725 27066 ts_manager.cc:194] Registered new tserver with Master: 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:07.113898 27433 heartbeater.cc:507] Master 127.26.65.125:44059 requested a full tablet report, sending...
I20250903 20:05:07.114423 26924 ts_manager.cc:194] Registered new tserver with Master: 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:07.114859 27066 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.65.66:53547
I20250903 20:05:07.115146 26995 ts_manager.cc:194] Registered new tserver with Master: 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:07.137339 27294 heartbeater.cc:499] Master 127.26.65.124:40911 was elected leader, sending a full tablet report...
I20250903 20:05:07.336087 27439 raft_consensus.cc:491] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 35bdb0090d8747059b5ee103e786e979)
I20250903 20:05:07.336300 27439 raft_consensus.cc:513] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } }
I20250903 20:05:07.337507 27439 leader_election.cc:290] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8857b5d4ea1e4f45aafb8e029837cca2 (127.26.65.126:39015), 35bdb0090d8747059b5ee103e786e979 (127.26.65.124:40911)
I20250903 20:05:07.338079 27076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "35bdb0090d8747059b5ee103e786e979" is_pre_election: true
I20250903 20:05:07.338106 26933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" is_pre_election: true
I20250903 20:05:07.338832 26982 leader_election.cc:304] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35dd69f3fa7a4fd8843b97f8a5d68296; no voters: 35bdb0090d8747059b5ee103e786e979, 8857b5d4ea1e4f45aafb8e029837cca2
I20250903 20:05:07.339171 27439 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:07.348389 27440 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 35bdb0090d8747059b5ee103e786e979)
I20250903 20:05:07.348575 27440 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } }
I20250903 20:05:07.349206 27440 leader_election.cc:290] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 35dd69f3fa7a4fd8843b97f8a5d68296 (127.26.65.125:44059), 35bdb0090d8747059b5ee103e786e979 (127.26.65.124:40911)
I20250903 20:05:07.349797 27005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" is_pre_election: true
I20250903 20:05:07.350035 27005 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8857b5d4ea1e4f45aafb8e029837cca2 in term 1.
I20250903 20:05:07.350255 27076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "35bdb0090d8747059b5ee103e786e979" is_pre_election: true
I20250903 20:05:07.350531 26909 leader_election.cc:304] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35dd69f3fa7a4fd8843b97f8a5d68296, 8857b5d4ea1e4f45aafb8e029837cca2; no voters:
I20250903 20:05:07.350857 27440 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250903 20:05:07.350987 27440 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 35bdb0090d8747059b5ee103e786e979)
I20250903 20:05:07.351097 27440 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [term 1 FOLLOWER]: Advancing to term 2
I20250903 20:05:07.354439 27440 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } }
I20250903 20:05:07.355283 27440 leader_election.cc:290] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [CANDIDATE]: Term 2 election: Requested vote from peers 35dd69f3fa7a4fd8843b97f8a5d68296 (127.26.65.125:44059), 35bdb0090d8747059b5ee103e786e979 (127.26.65.124:40911)
I20250903 20:05:07.355893 27076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "35bdb0090d8747059b5ee103e786e979"
I20250903 20:05:07.355873 27005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296"
I20250903 20:05:07.356099 27005 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 1 FOLLOWER]: Advancing to term 2
I20250903 20:05:07.358038 27005 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8857b5d4ea1e4f45aafb8e029837cca2 in term 2.
I20250903 20:05:07.358474 26909 leader_election.cc:304] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 35dd69f3fa7a4fd8843b97f8a5d68296, 8857b5d4ea1e4f45aafb8e029837cca2; no voters: 35bdb0090d8747059b5ee103e786e979
I20250903 20:05:07.358821 27440 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [term 2 FOLLOWER]: Leader election won for term 2
I20250903 20:05:07.359309 27440 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [term 2 LEADER]: Becoming Leader. State: Replica: 8857b5d4ea1e4f45aafb8e029837cca2, State: Running, Role: LEADER
I20250903 20:05:07.359781 27440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } }
I20250903 20:05:07.361516 27441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8857b5d4ea1e4f45aafb8e029837cca2. Latest consensus state: current_term: 2 leader_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:07.361799 27441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [sys.catalog]: This master's current role is: LEADER
I20250903 20:05:07.362457 27443 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 20:05:07.364198 27443 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 20:05:07.364964 27443 catalog_manager.cc:1261] Loaded cluster ID: 07b8834eda114abdb992ba4c63cadc24
I20250903 20:05:07.365049 27443 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 20:05:07.366118 27443 catalog_manager.cc:1506] Loading token signing keys...
I20250903 20:05:07.366894 27443 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2: Loaded TSK: 0
I20250903 20:05:07.367726 27443 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 20:05:07.379868 26933 raft_consensus.cc:1238] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [term 2 LEADER]: Rejecting Update request from peer 35bdb0090d8747059b5ee103e786e979 for earlier term 1. Current term is 2. Ops: []
I20250903 20:05:07.380611 27444 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250903 20:05:07.381157 27444 consensus_queue.cc:1230] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [LEADER]: Peer 8857b5d4ea1e4f45aafb8e029837cca2 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250903 20:05:07.381310 27445 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [term 1 LEADER]: Stepping down as leader of term 1
I20250903 20:05:07.381390 27445 raft_consensus.cc:738] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 35bdb0090d8747059b5ee103e786e979, State: Running, Role: LEADER
I20250903 20:05:07.381793 27445 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } }
I20250903 20:05:07.382308 27445 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [term 1 FOLLOWER]: Advancing to term 2
W20250903 20:05:07.448725 27438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 20:05:07.449030 27438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 20:05:07.449184 27438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250903 20:05:07.457387 27438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 20:05:07.457610 27438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.65.67
I20250903 20:05:07.467623 27438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.65.84:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.65.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.65.67
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.26.65.126:39015,127.26.65.125:44059,127.26.65.124:40911
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.65.67
--log_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 19:52:33 UTC on 24a791456cd2
build id 7884
ASAN enabled
I20250903 20:05:07.469326 27438 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 20:05:07.471298 27438 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 20:05:07.481506 27453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 20:05:07.481371 27451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 20:05:07.481326 27450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 20:05:07.482161 27438 server_base.cc:1047] running on GCE node
I20250903 20:05:07.482779 27438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 20:05:07.483788 27438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 20:05:07.484977 27438 hybrid_clock.cc:648] HybridClock initialized: now 1756929907484933 us; error 36 us; skew 500 ppm
I20250903 20:05:07.485275 27438 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 20:05:07.487324 27438 webserver.cc:480] Webserver started at http://127.26.65.67:41219/ using document root <none> and password file <none>
I20250903 20:05:07.488044 27438 fs_manager.cc:362] Metadata directory not provided
I20250903 20:05:07.488139 27438 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 20:05:07.488487 27438 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 20:05:07.490864 27438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/data/instance:
uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
format_stamp: "Formatted at 2025-09-03 20:05:07 on dist-test-slave-fs2w"
I20250903 20:05:07.491616 27438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/instance:
uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
format_stamp: "Formatted at 2025-09-03 20:05:07 on dist-test-slave-fs2w"
I20250903 20:05:07.496966 27438 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250903 20:05:07.500504 27459 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 20:05:07.502020 27438 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:07.502259 27438 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal
uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
format_stamp: "Formatted at 2025-09-03 20:05:07 on dist-test-slave-fs2w"
I20250903 20:05:07.502509 27438 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 20:05:07.514320 27438 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 20:05:07.515185 27438 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 20:05:07.515456 27438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 20:05:07.516402 27438 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 20:05:07.518075 27438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 20:05:07.518175 27438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 20:05:07.518342 27438 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 20:05:07.518417 27438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 20:05:07.553500 27438 rpc_server.cc:307] RPC server started. Bound to: 127.26.65.67:46231
I20250903 20:05:07.553567 27571 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.65.67:46231 every 8 connection(s)
I20250903 20:05:07.555054 27438 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/data/info.pb
I20250903 20:05:07.556878 26885 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskH7eH49/build/asan/bin/kudu as pid 27438
I20250903 20:05:07.557067 26885 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/instance
I20250903 20:05:07.573978 27573 heartbeater.cc:344] Connected to a master server at 127.26.65.126:39015
I20250903 20:05:07.574021 27572 heartbeater.cc:344] Connected to a master server at 127.26.65.124:40911
I20250903 20:05:07.574466 27573 heartbeater.cc:461] Registering TS with master...
I20250903 20:05:07.574471 27572 heartbeater.cc:461] Registering TS with master...
I20250903 20:05:07.575805 27573 heartbeater.cc:507] Master 127.26.65.126:39015 requested a full tablet report, sending...
I20250903 20:05:07.575800 27572 heartbeater.cc:507] Master 127.26.65.124:40911 requested a full tablet report, sending...
I20250903 20:05:07.577978 26924 ts_manager.cc:194] Registered new tserver with Master: e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:07.578119 27066 ts_manager.cc:194] Registered new tserver with Master: e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:07.579953 27574 heartbeater.cc:344] Connected to a master server at 127.26.65.125:44059
I20250903 20:05:07.580096 27574 heartbeater.cc:461] Registering TS with master...
I20250903 20:05:07.580217 26924 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.65.67:50135
I20250903 20:05:07.580632 27574 heartbeater.cc:507] Master 127.26.65.125:44059 requested a full tablet report, sending...
I20250903 20:05:07.581889 26995 ts_manager.cc:194] Registered new tserver with Master: e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:07.586378 26885 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250903 20:05:07.616217 26924 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35604:
name: "default.Table_be7928a2be8d4fc4b231e9abe440eca5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:07.616653 26922 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35604:
name: "default.Table_3f2071686e39440ba06b91e1a618f481"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:07.617432 27431 heartbeater.cc:499] Master 127.26.65.124:40911 was elected leader, sending a full tablet report...
I20250903 20:05:07.617467 26923 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35604:
name: "default.Table_96fa5ba3295a4ce8b95fc6d8b7790ba6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:07.618494 26921 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35604:
name: "default.Table_6c9710ec5f82499491a18c8079989dda"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:07.619125 26921 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c9710ec5f82499491a18c8079989dda in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:07.619926 26919 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35604:
name: "default.Table_aef2db5af9534175a391212d53a41cd3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:07.620186 26918 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35604:
name: "default.Table_1b35497e01d941dd9d9048a7f0d9c87a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:07.620657 26919 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aef2db5af9534175a391212d53a41cd3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:07.621063 26918 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b35497e01d941dd9d9048a7f0d9c87a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:07.622567 26920 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35604:
name: "default.Table_b9ca517d9a064036b68ca6f7e6c0fb41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:07.623347 26920 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9ca517d9a064036b68ca6f7e6c0fb41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:07.627163 26923 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96fa5ba3295a4ce8b95fc6d8b7790ba6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:07.629330 26922 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f2071686e39440ba06b91e1a618f481 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:07.630165 26917 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35604:
name: "default.Table_ebbff76084974404af90b4a1639f9747"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:07.631286 26917 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ebbff76084974404af90b4a1639f9747 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:07.631299 26924 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be7928a2be8d4fc4b231e9abe440eca5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:07.633560 26916 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35604:
name: "default.Table_a3d6b06b1c504fadafda3fa411f811b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:07.634368 26916 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3d6b06b1c504fadafda3fa411f811b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:07.641045 27005 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 2 FOLLOWER]: Refusing update from remote peer 8857b5d4ea1e4f45aafb8e029837cca2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20250903 20:05:07.641374 27076 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [term 2 FOLLOWER]: Refusing update from remote peer 8857b5d4ea1e4f45aafb8e029837cca2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20250903 20:05:07.642814 27440 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250903 20:05:07.643071 27441 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250903 20:05:07.644565 26915 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35604:
name: "default.Table_f2f3992244514b6787e00498c3d95341"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:07.645280 26915 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2f3992244514b6787e00498c3d95341 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:07.658077 27445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8857b5d4ea1e4f45aafb8e029837cca2. Latest consensus state: current_term: 2 leader_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:07.658417 27445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 20:05:07.659485 27439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8857b5d4ea1e4f45aafb8e029837cca2. Latest consensus state: current_term: 2 leader_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:07.659710 27439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 20:05:07.660715 27595 mvcc.cc:204] Tried to move back new op lower bound from 7196384901683924992 to 7196384900549210112. Current Snapshot: MvccSnapshot[applied={T|T < 7196384901683924992}]
I20250903 20:05:07.661432 27136 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250903 20:05:07.662833 27599 mvcc.cc:204] Tried to move back new op lower bound from 7196384901683924992 to 7196384900549210112. Current Snapshot: MvccSnapshot[applied={T|T < 7196384901683924992}]
I20250903 20:05:07.666895 27598 mvcc.cc:204] Tried to move back new op lower bound from 7196384901683924992 to 7196384900549210112. Current Snapshot: MvccSnapshot[applied={T|T < 7196384901683924992}]
I20250903 20:05:07.700258 27439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:07.700580 27439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 20:05:07.700547 27445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:07.700827 27445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 20:05:07.713433 26919 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35604:
table { table_name: "default.Table_aef2db5af9534175a391212d53a41cd3" } new_table_name: "default.Table_b762f9202b4141ccb73d262c8b6314a4" modify_external_catalogs: true
I20250903 20:05:07.715780 26918 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35604:
table { table_name: "default.Table_1b35497e01d941dd9d9048a7f0d9c87a" } new_table_name: "default.Table_78d83e20caac4177a194a3c3434d4ffc" modify_external_catalogs: true
I20250903 20:05:07.722776 26923 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35604:
table { table_name: "default.Table_b9ca517d9a064036b68ca6f7e6c0fb41" } new_table_name: "default.Table_b616c8470952474b8bdef7d06f6da59c" modify_external_catalogs: true
I20250903 20:05:07.725142 26920 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35604:
table { table_name: "default.Table_96fa5ba3295a4ce8b95fc6d8b7790ba6" } new_table_name: "default.Table_cbebd771513c4bfaa06d0776b4a10303" modify_external_catalogs: true
I20250903 20:05:07.736088 27615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:07.736594 27615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [sys.catalog]: This master's current role is: LEADER
I20250903 20:05:07.737069 27615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:07.737325 27615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [sys.catalog]: This master's current role is: LEADER
I20250903 20:05:07.755777 26921 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35604:
table { table_name: "default.Table_6c9710ec5f82499491a18c8079989dda" } modify_external_catalogs: true
I20250903 20:05:07.756108 26921 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35604:
table { table_name: "default.Table_6c9710ec5f82499491a18c8079989dda" } modify_external_catalogs: true
I20250903 20:05:07.757135 26924 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35604:
table { table_name: "default.Table_a3d6b06b1c504fadafda3fa411f811b6" } modify_external_catalogs: true
I20250903 20:05:07.757364 26924 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35604:
table { table_name: "default.Table_a3d6b06b1c504fadafda3fa411f811b6" } modify_external_catalogs: true
I20250903 20:05:07.758351 26922 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35604:
table { table_name: "default.Table_3f2071686e39440ba06b91e1a618f481" } new_table_name: "default.Table_3e4560347ecf4f09a263e7d0d562105d" modify_external_catalogs: true
I20250903 20:05:07.762924 26916 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35604:
table { table_name: "default.Table_be7928a2be8d4fc4b231e9abe440eca5" } modify_external_catalogs: true
I20250903 20:05:07.763162 26916 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35604:
table { table_name: "default.Table_be7928a2be8d4fc4b231e9abe440eca5" } modify_external_catalogs: true
I20250903 20:05:07.764465 26917 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35604:
table { table_name: "default.Table_ebbff76084974404af90b4a1639f9747" } modify_external_catalogs: true
I20250903 20:05:07.764698 26917 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35604:
table { table_name: "default.Table_ebbff76084974404af90b4a1639f9747" } modify_external_catalogs: true
I20250903 20:05:07.765769 26915 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35604:
table { table_name: "default.Table_f2f3992244514b6787e00498c3d95341" } modify_external_catalogs: true
I20250903 20:05:07.765980 26915 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35604:
table { table_name: "default.Table_f2f3992244514b6787e00498c3d95341" } modify_external_catalogs: true
W20250903 20:05:07.772782 26919 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4efd72d607f34650b4bcc832770fa2f7 has no consensus state
W20250903 20:05:07.775092 26918 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9d37a5d51a184c76bb8496679dfe3991 has no consensus state
W20250903 20:05:07.777110 26923 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fff44282202c4dfcb74743e5e8d09fc5 has no consensus state
I20250903 20:05:07.779672 26921 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet be2d301423154934a3d1774aeced70f3
I20250903 20:05:07.780966 26924 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet c973864ea6ad4114a82b6c6e1043da87
W20250903 20:05:07.782430 26920 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet af116a66fda74949836bda28300565f3 consensus state has no leader
W20250903 20:05:07.790234 26922 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 01bef73ab62d4d10a88c3478d4bd3b0c has no consensus state
I20250903 20:05:07.796949 26916 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 9101376c1fd8431e908af8967f4a8f19
I20250903 20:05:07.797377 26917 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 3f5ea90cc93944caaf2925f42c094caa
I20250903 20:05:07.797825 26915 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet ed7d7ea8d7914102b0f278f2ad67b4e4
W20250903 20:05:07.817878 26908 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 01bef73ab62d4d10a88c3478d4bd3b0c (table default.Table_3e4560347ecf4f09a263e7d0d562105d [id=a85e4c476fbe422d8aaee54022e6dfde], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 01bef73ab62d4d10a88c3478d4bd3b0c consensus state has no leader
I20250903 20:05:07.821352 26885 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskH7eH49/build/asan/bin/kudu pid 26893
I20250903 20:05:07.823077 26954 generic_service.cc:196] Checking for leaks (request via RPC)
I20250903 20:05:07.822484 27229 tablet_service.cc:1468] Processing CreateTablet for tablet af116a66fda74949836bda28300565f3 (DEFAULT_TABLE table=default.Table_96fa5ba3295a4ce8b95fc6d8b7790ba6 [id=cbae62c266f24526affd029472f41b94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:07.824199 27229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af116a66fda74949836bda28300565f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:07.822638 27228 tablet_service.cc:1468] Processing CreateTablet for tablet fff44282202c4dfcb74743e5e8d09fc5 (DEFAULT_TABLE table=default.Table_b616c8470952474b8bdef7d06f6da59c [id=8622aba89ccc48c5ac6a28e038fa9db5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:07.826845 27228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fff44282202c4dfcb74743e5e8d09fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:07.827569 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 9d37a5d51a184c76bb8496679dfe3991 (DEFAULT_TABLE table=default.Table_78d83e20caac4177a194a3c3434d4ffc [id=374b9abdf6a54b38876a96d8ec4a0368]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:07.828907 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d37a5d51a184c76bb8496679dfe3991. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:07.822542 27227 tablet_service.cc:1468] Processing CreateTablet for tablet 4efd72d607f34650b4bcc832770fa2f7 (DEFAULT_TABLE table=default.Table_b762f9202b4141ccb73d262c8b6314a4 [id=0ab13735ce35424286be08f4f966ddc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:07.830286 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4efd72d607f34650b4bcc832770fa2f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:07.830411 27503 tablet_service.cc:1468] Processing CreateTablet for tablet 01bef73ab62d4d10a88c3478d4bd3b0c (DEFAULT_TABLE table=default.Table_3e4560347ecf4f09a263e7d0d562105d [id=a85e4c476fbe422d8aaee54022e6dfde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:07.830925 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01bef73ab62d4d10a88c3478d4bd3b0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:07.833220 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 01bef73ab62d4d10a88c3478d4bd3b0c (DEFAULT_TABLE table=default.Table_3e4560347ecf4f09a263e7d0d562105d [id=a85e4c476fbe422d8aaee54022e6dfde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:07.833809 27226 tablet_service.cc:1468] Processing CreateTablet for tablet 9d37a5d51a184c76bb8496679dfe3991 (DEFAULT_TABLE table=default.Table_78d83e20caac4177a194a3c3434d4ffc [id=374b9abdf6a54b38876a96d8ec4a0368]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:07.833961 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01bef73ab62d4d10a88c3478d4bd3b0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:07.834461 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d37a5d51a184c76bb8496679dfe3991. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:07.838459 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 4efd72d607f34650b4bcc832770fa2f7 (DEFAULT_TABLE table=default.Table_b762f9202b4141ccb73d262c8b6314a4 [id=0ab13735ce35424286be08f4f966ddc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:07.839201 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4efd72d607f34650b4bcc832770fa2f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:07.841432 27507 tablet_service.cc:1468] Processing CreateTablet for tablet af116a66fda74949836bda28300565f3 (DEFAULT_TABLE table=default.Table_96fa5ba3295a4ce8b95fc6d8b7790ba6 [id=cbae62c266f24526affd029472f41b94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:07.842206 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af116a66fda74949836bda28300565f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:07.850239 27506 tablet_service.cc:1468] Processing CreateTablet for tablet fff44282202c4dfcb74743e5e8d09fc5 (DEFAULT_TABLE table=default.Table_b616c8470952474b8bdef7d06f6da59c [id=8622aba89ccc48c5ac6a28e038fa9db5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:07.850927 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fff44282202c4dfcb74743e5e8d09fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:07.858194 27642 tablet_bootstrap.cc:492] T 01bef73ab62d4d10a88c3478d4bd3b0c P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:07.860817 27643 tablet_bootstrap.cc:492] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:07.864101 27642 tablet_bootstrap.cc:654] T 01bef73ab62d4d10a88c3478d4bd3b0c P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:07.864770 27643 tablet_bootstrap.cc:654] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:07.865521 27642 log.cc:826] T 01bef73ab62d4d10a88c3478d4bd3b0c P 89f8f0ee68b940db9ce250c0604f934a: Log is configured to *not* fsync() on all Append() calls
I20250903 20:05:07.866406 27643 log.cc:826] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f: Log is configured to *not* fsync() on all Append() calls
I20250903 20:05:07.873414 27642 tablet_bootstrap.cc:492] T 01bef73ab62d4d10a88c3478d4bd3b0c P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:07.873574 27643 tablet_bootstrap.cc:492] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:07.873737 27642 ts_tablet_manager.cc:1397] T 01bef73ab62d4d10a88c3478d4bd3b0c P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20250903 20:05:07.873857 27643 ts_tablet_manager.cc:1397] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20250903 20:05:07.879184 27642 raft_consensus.cc:357] T 01bef73ab62d4d10a88c3478d4bd3b0c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:07.879557 27642 raft_consensus.cc:383] T 01bef73ab62d4d10a88c3478d4bd3b0c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:07.879674 27642 raft_consensus.cc:738] T 01bef73ab62d4d10a88c3478d4bd3b0c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:07.880625 27642 consensus_queue.cc:260] T 01bef73ab62d4d10a88c3478d4bd3b0c P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:07.881016 27643 raft_consensus.cc:357] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:07.881410 27643 raft_consensus.cc:383] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:07.881527 27643 raft_consensus.cc:738] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:07.882378 27643 consensus_queue.cc:260] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:07.883234 27642 ts_tablet_manager.cc:1428] T 01bef73ab62d4d10a88c3478d4bd3b0c P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.009s user 0.005s sys 0.005s
I20250903 20:05:07.883548 27295 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
I20250903 20:05:07.884734 27642 tablet_bootstrap.cc:492] T 9d37a5d51a184c76bb8496679dfe3991 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:07.886638 27642 tablet_bootstrap.cc:654] T 9d37a5d51a184c76bb8496679dfe3991 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:07.895584 27642 tablet_bootstrap.cc:492] T 9d37a5d51a184c76bb8496679dfe3991 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:07.895872 27642 ts_tablet_manager.cc:1397] T 9d37a5d51a184c76bb8496679dfe3991 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.003s
I20250903 20:05:07.896767 27642 raft_consensus.cc:357] T 9d37a5d51a184c76bb8496679dfe3991 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:07.896951 27642 raft_consensus.cc:383] T 9d37a5d51a184c76bb8496679dfe3991 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:07.897025 27642 raft_consensus.cc:738] T 9d37a5d51a184c76bb8496679dfe3991 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:07.897272 27642 consensus_queue.cc:260] T 9d37a5d51a184c76bb8496679dfe3991 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:07.898536 27643 ts_tablet_manager.cc:1428] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.025s user 0.013s sys 0.006s
I20250903 20:05:07.899871 27642 ts_tablet_manager.cc:1428] T 9d37a5d51a184c76bb8496679dfe3991 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250903 20:05:07.900205 27642 tablet_bootstrap.cc:492] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:07.900758 27643 tablet_bootstrap.cc:492] T fff44282202c4dfcb74743e5e8d09fc5 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:07.900789 27573 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
I20250903 20:05:07.901966 27642 tablet_bootstrap.cc:654] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:07.902549 27643 tablet_bootstrap.cc:654] T fff44282202c4dfcb74743e5e8d09fc5 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:07.904843 27643 tablet_bootstrap.cc:492] T fff44282202c4dfcb74743e5e8d09fc5 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:07.905014 27643 ts_tablet_manager.cc:1397] T fff44282202c4dfcb74743e5e8d09fc5 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250903 20:05:07.905972 27643 raft_consensus.cc:357] T fff44282202c4dfcb74743e5e8d09fc5 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:07.906181 27643 raft_consensus.cc:383] T fff44282202c4dfcb74743e5e8d09fc5 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:07.906291 27643 raft_consensus.cc:738] T fff44282202c4dfcb74743e5e8d09fc5 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:07.906603 27643 consensus_queue.cc:260] T fff44282202c4dfcb74743e5e8d09fc5 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:07.907146 27643 ts_tablet_manager.cc:1428] T fff44282202c4dfcb74743e5e8d09fc5 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250903 20:05:07.907451 27643 tablet_bootstrap.cc:492] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:07.909111 27643 tablet_bootstrap.cc:654] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:07.912273 27642 tablet_bootstrap.cc:492] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:07.912438 27642 ts_tablet_manager.cc:1397] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.009s
I20250903 20:05:07.913230 27642 raft_consensus.cc:357] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:07.913419 27642 raft_consensus.cc:383] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:07.913493 27642 raft_consensus.cc:738] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:07.913690 27642 consensus_queue.cc:260] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:07.914155 27642 ts_tablet_manager.cc:1428] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:07.915175 27642 tablet_bootstrap.cc:492] T 4efd72d607f34650b4bcc832770fa2f7 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:07.915467 27643 tablet_bootstrap.cc:492] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:07.915630 27643 ts_tablet_manager.cc:1397] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20250903 20:05:07.916378 27643 raft_consensus.cc:357] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:07.916556 27643 raft_consensus.cc:383] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:07.916640 27643 raft_consensus.cc:738] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:07.916870 27643 consensus_queue.cc:260] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:07.917140 27642 tablet_bootstrap.cc:654] T 4efd72d607f34650b4bcc832770fa2f7 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:07.918514 27643 ts_tablet_manager.cc:1428] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:07.918792 27643 tablet_bootstrap.cc:492] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:07.920555 27643 tablet_bootstrap.cc:654] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:07.931309 27643 tablet_bootstrap.cc:492] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:07.931318 27642 tablet_bootstrap.cc:492] T 4efd72d607f34650b4bcc832770fa2f7 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:07.931505 27642 ts_tablet_manager.cc:1397] T 4efd72d607f34650b4bcc832770fa2f7 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.009s
I20250903 20:05:07.931524 27643 ts_tablet_manager.cc:1397] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20250903 20:05:07.932204 27643 raft_consensus.cc:357] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:07.932263 27642 raft_consensus.cc:357] T 4efd72d607f34650b4bcc832770fa2f7 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:07.932379 27643 raft_consensus.cc:383] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:07.932459 27643 raft_consensus.cc:738] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:07.932468 27642 raft_consensus.cc:383] T 4efd72d607f34650b4bcc832770fa2f7 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:07.932546 27642 raft_consensus.cc:738] T 4efd72d607f34650b4bcc832770fa2f7 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:07.932724 27643 consensus_queue.cc:260] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:07.932812 27642 consensus_queue.cc:260] T 4efd72d607f34650b4bcc832770fa2f7 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:07.933302 27642 ts_tablet_manager.cc:1428] T 4efd72d607f34650b4bcc832770fa2f7 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:07.933669 27642 tablet_bootstrap.cc:492] T af116a66fda74949836bda28300565f3 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:07.935367 27642 tablet_bootstrap.cc:654] T af116a66fda74949836bda28300565f3 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:07.937058 27643 ts_tablet_manager.cc:1428] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250903 20:05:07.938656 27643 tablet_bootstrap.cc:492] T af116a66fda74949836bda28300565f3 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:07.941191 27643 tablet_bootstrap.cc:654] T af116a66fda74949836bda28300565f3 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:07.943852 27642 tablet_bootstrap.cc:492] T af116a66fda74949836bda28300565f3 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:07.944005 27642 ts_tablet_manager.cc:1397] T af116a66fda74949836bda28300565f3 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20250903 20:05:07.944757 27642 raft_consensus.cc:357] T af116a66fda74949836bda28300565f3 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } }
I20250903 20:05:07.944933 27642 raft_consensus.cc:383] T af116a66fda74949836bda28300565f3 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:07.945005 27642 raft_consensus.cc:738] T af116a66fda74949836bda28300565f3 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:07.945245 27642 consensus_queue.cc:260] T af116a66fda74949836bda28300565f3 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } }
I20250903 20:05:07.947316 27642 ts_tablet_manager.cc:1428] T af116a66fda74949836bda28300565f3 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250903 20:05:07.950294 27643 tablet_bootstrap.cc:492] T af116a66fda74949836bda28300565f3 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:07.950480 27643 ts_tablet_manager.cc:1397] T af116a66fda74949836bda28300565f3 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20250903 20:05:07.951121 27643 raft_consensus.cc:357] T af116a66fda74949836bda28300565f3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } }
I20250903 20:05:07.951273 27643 raft_consensus.cc:383] T af116a66fda74949836bda28300565f3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:07.951375 27643 raft_consensus.cc:738] T af116a66fda74949836bda28300565f3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:07.951608 27643 consensus_queue.cc:260] T af116a66fda74949836bda28300565f3 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } }
I20250903 20:05:07.952147 27643 ts_tablet_manager.cc:1428] T af116a66fda74949836bda28300565f3 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:08.005252 27647 raft_consensus.cc:491] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:08.005558 27647 raft_consensus.cc:513] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:08.007405 27647 leader_election.cc:290] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:08.014310 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01bef73ab62d4d10a88c3478d4bd3b0c" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:08.014897 27249 raft_consensus.cc:2466] T 01bef73ab62d4d10a88c3478d4bd3b0c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:08.015587 27462 leader_election.cc:304] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:08.016027 27647 raft_consensus.cc:2802] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:08.016188 27647 raft_consensus.cc:491] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:08.016286 27647 raft_consensus.cc:3058] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:08.017640 27462 leader_election.cc:343] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 01bef73ab62d4d10a88c3478d4bd3b0c
I20250903 20:05:08.019344 27647 raft_consensus.cc:513] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:08.020015 27647 leader_election.cc:290] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:08.020545 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01bef73ab62d4d10a88c3478d4bd3b0c" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:08.020746 27249 raft_consensus.cc:3058] T 01bef73ab62d4d10a88c3478d4bd3b0c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:08.020799 27462 leader_election.cc:343] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 01bef73ab62d4d10a88c3478d4bd3b0c
I20250903 20:05:08.023630 27249 raft_consensus.cc:2466] T 01bef73ab62d4d10a88c3478d4bd3b0c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:08.024061 27462 leader_election.cc:304] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:08.026010 26922 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35604:
name: "default.Table_8d1a53fac8c24b9c8b2b4549fa8d1dbc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:08.026584 26885 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskH7eH49/build/asan/bin/kudu with pid 26893
I20250903 20:05:08.027961 27647 raft_consensus.cc:2802] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:08.028406 27647 raft_consensus.cc:695] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:08.029209 27647 consensus_queue.cc:237] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:08.050159 27573 heartbeater.cc:646] Failed to heartbeat to 127.26.65.126:39015 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.26.65.126:39015 (error 108)
W20250903 20:05:08.050730 27582 connection.cc:537] client connection to 127.26.65.126:39015 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 20:05:08.050935 27295 heartbeater.cc:646] Failed to heartbeat to 127.26.65.126:39015 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.26.65.126:39015 (error 108)
W20250903 20:05:08.050940 27639 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 20:05:07.807344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54894 (local address 127.26.65.66:38449)
0903 20:05:07.819988 (+ 12644us) server_negotiation.cc:207] Beginning negotiation
0903 20:05:07.819998 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0903 20:05:07.820034 (+ 36us) server_negotiation.cc:408] Connection header received
0903 20:05:07.820146 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0903 20:05:07.820155 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0903 20:05:07.820549 (+ 394us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0903 20:05:07.820779 (+ 230us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0903 20:05:07.821726 (+ 947us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0903 20:05:07.823978 (+ 2252us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0903 20:05:08.049507 (+225529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54894: BlockingRecv error: recv got EOF from 127.0.0.1:54894 (error 108)
Metrics: {"server-negotiator.queue_time_us":12467,"thread_start_us":13628,"threads_started":1}
W20250903 20:05:08.051628 27582 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.26.65.126:39015)
W20250903 20:05:08.052079 27582 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.26.65.126:39015)
W20250903 20:05:08.052413 27582 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.26.65.126:39015)
W20250903 20:05:08.052687 27582 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.26.65.126:39015)
W20250903 20:05:08.058501 27575 tablet.cc:2378] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250903 20:05:08.105623 27303 tablet.cc:2378] T 01bef73ab62d4d10a88c3478d4bd3b0c P 89f8f0ee68b940db9ce250c0604f934a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 20:05:08.127779 27432 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
W20250903 20:05:08.129407 27432 heartbeater.cc:646] Failed to heartbeat to 127.26.65.126:39015 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.26.65.126:39015: connect: Connection refused (error 111)
I20250903 20:05:08.249120 27647 raft_consensus.cc:491] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:08.249370 27647 raft_consensus.cc:513] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:08.249929 27647 leader_election.cc:290] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:08.250497 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4efd72d607f34650b4bcc832770fa2f7" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:08.250725 27249 raft_consensus.cc:2466] T 4efd72d607f34650b4bcc832770fa2f7 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
W20250903 20:05:08.250892 27462 leader_election.cc:343] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 4efd72d607f34650b4bcc832770fa2f7
I20250903 20:05:08.251142 27462 leader_election.cc:304] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:08.251413 27647 raft_consensus.cc:2802] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:08.251505 27647 raft_consensus.cc:491] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:08.251607 27647 raft_consensus.cc:3058] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:08.253430 27647 raft_consensus.cc:513] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:08.253965 27647 leader_election.cc:290] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:08.254405 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4efd72d607f34650b4bcc832770fa2f7" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:08.254593 27249 raft_consensus.cc:3058] T 4efd72d607f34650b4bcc832770fa2f7 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:08.254742 27462 leader_election.cc:343] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 4efd72d607f34650b4bcc832770fa2f7
I20250903 20:05:08.256403 27249 raft_consensus.cc:2466] T 4efd72d607f34650b4bcc832770fa2f7 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:08.256742 27462 leader_election.cc:304] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:08.256986 27647 raft_consensus.cc:2802] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:08.257134 27647 raft_consensus.cc:695] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:08.257445 27647 consensus_queue.cc:237] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:08.311761 27662 raft_consensus.cc:491] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8857b5d4ea1e4f45aafb8e029837cca2)
I20250903 20:05:08.311959 27662 raft_consensus.cc:513] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } }
I20250903 20:05:08.313113 27662 leader_election.cc:290] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8857b5d4ea1e4f45aafb8e029837cca2 (127.26.65.126:39015), 35bdb0090d8747059b5ee103e786e979 (127.26.65.124:40911)
I20250903 20:05:08.313251 27076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: false dest_uuid: "35bdb0090d8747059b5ee103e786e979" is_pre_election: true
W20250903 20:05:08.315200 26982 leader_election.cc:336] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8857b5d4ea1e4f45aafb8e029837cca2 (127.26.65.126:39015): Network error: Client connection negotiation failed: client connection to 127.26.65.126:39015: connect: Connection refused (error 111)
I20250903 20:05:08.315404 26982 leader_election.cc:304] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35dd69f3fa7a4fd8843b97f8a5d68296; no voters: 35bdb0090d8747059b5ee103e786e979, 8857b5d4ea1e4f45aafb8e029837cca2
I20250903 20:05:08.315689 27662 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250903 20:05:08.316325 27665 raft_consensus.cc:491] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8857b5d4ea1e4f45aafb8e029837cca2)
I20250903 20:05:08.316457 27665 raft_consensus.cc:513] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } }
I20250903 20:05:08.316946 27665 leader_election.cc:290] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8857b5d4ea1e4f45aafb8e029837cca2 (127.26.65.126:39015), 35dd69f3fa7a4fd8843b97f8a5d68296 (127.26.65.125:44059)
I20250903 20:05:08.317386 27004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35bdb0090d8747059b5ee103e786e979" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: false dest_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" is_pre_election: true
I20250903 20:05:08.317590 27004 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35bdb0090d8747059b5ee103e786e979 in term 2.
W20250903 20:05:08.317780 27053 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.65.126:39015: connect: Connection refused (error 111)
I20250903 20:05:08.318020 27051 leader_election.cc:304] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35bdb0090d8747059b5ee103e786e979, 35dd69f3fa7a4fd8843b97f8a5d68296; no voters:
I20250903 20:05:08.318354 27665 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250903 20:05:08.318493 27665 raft_consensus.cc:491] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8857b5d4ea1e4f45aafb8e029837cca2)
I20250903 20:05:08.318610 27665 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [term 2 FOLLOWER]: Advancing to term 3
W20250903 20:05:08.319211 27053 leader_election.cc:336] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8857b5d4ea1e4f45aafb8e029837cca2 (127.26.65.126:39015): Network error: Client connection negotiation failed: client connection to 127.26.65.126:39015: connect: Connection refused (error 111)
I20250903 20:05:08.320633 27665 raft_consensus.cc:513] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } }
I20250903 20:05:08.321098 27665 leader_election.cc:290] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [CANDIDATE]: Term 3 election: Requested vote from peers 8857b5d4ea1e4f45aafb8e029837cca2 (127.26.65.126:39015), 35dd69f3fa7a4fd8843b97f8a5d68296 (127.26.65.125:44059)
I20250903 20:05:08.321539 27004 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35bdb0090d8747059b5ee103e786e979" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: false dest_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296"
I20250903 20:05:08.321753 27004 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 2 FOLLOWER]: Advancing to term 3
W20250903 20:05:08.322311 27053 leader_election.cc:336] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8857b5d4ea1e4f45aafb8e029837cca2 (127.26.65.126:39015): Network error: Client connection negotiation failed: client connection to 127.26.65.126:39015: connect: Connection refused (error 111)
I20250903 20:05:08.323800 27004 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35bdb0090d8747059b5ee103e786e979 in term 3.
I20250903 20:05:08.324183 27051 leader_election.cc:304] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 35bdb0090d8747059b5ee103e786e979, 35dd69f3fa7a4fd8843b97f8a5d68296; no voters: 8857b5d4ea1e4f45aafb8e029837cca2
I20250903 20:05:08.324447 27665 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [term 3 FOLLOWER]: Leader election won for term 3
I20250903 20:05:08.324771 27665 raft_consensus.cc:695] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [term 3 LEADER]: Becoming Leader. State: Replica: 35bdb0090d8747059b5ee103e786e979, State: Running, Role: LEADER
I20250903 20:05:08.325014 27665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } }
I20250903 20:05:08.325786 27668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35bdb0090d8747059b5ee103e786e979. Latest consensus state: current_term: 3 leader_uuid: "35bdb0090d8747059b5ee103e786e979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:08.326031 27668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [sys.catalog]: This master's current role is: LEADER
I20250903 20:05:08.326591 27669 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 20:05:08.328758 27669 catalog_manager.cc:671] Loaded metadata for table default.Table_b762f9202b4141ccb73d262c8b6314a4 [id=0ab13735ce35424286be08f4f966ddc4]
I20250903 20:05:08.329493 27669 catalog_manager.cc:671] Loaded metadata for table default.Table_78d83e20caac4177a194a3c3434d4ffc [id=374b9abdf6a54b38876a96d8ec4a0368]
I20250903 20:05:08.329985 27669 catalog_manager.cc:671] Loaded metadata for table default.Table_b616c8470952474b8bdef7d06f6da59c [id=8622aba89ccc48c5ac6a28e038fa9db5]
I20250903 20:05:08.330605 27669 catalog_manager.cc:671] Loaded metadata for table default.Table_3e4560347ecf4f09a263e7d0d562105d [id=a85e4c476fbe422d8aaee54022e6dfde]
I20250903 20:05:08.331069 27669 catalog_manager.cc:671] Loaded metadata for table default.Table_cbebd771513c4bfaa06d0776b4a10303 [id=cbae62c266f24526affd029472f41b94]
I20250903 20:05:08.332780 27669 tablet_loader.cc:96] loaded metadata for tablet 01bef73ab62d4d10a88c3478d4bd3b0c (table default.Table_3e4560347ecf4f09a263e7d0d562105d [id=a85e4c476fbe422d8aaee54022e6dfde])
I20250903 20:05:08.333263 27669 tablet_loader.cc:96] loaded metadata for tablet 4efd72d607f34650b4bcc832770fa2f7 (table default.Table_b762f9202b4141ccb73d262c8b6314a4 [id=0ab13735ce35424286be08f4f966ddc4])
I20250903 20:05:08.333698 27669 tablet_loader.cc:96] loaded metadata for tablet 9d37a5d51a184c76bb8496679dfe3991 (table default.Table_78d83e20caac4177a194a3c3434d4ffc [id=374b9abdf6a54b38876a96d8ec4a0368])
I20250903 20:05:08.333993 27669 tablet_loader.cc:96] loaded metadata for tablet af116a66fda74949836bda28300565f3 (table default.Table_cbebd771513c4bfaa06d0776b4a10303 [id=cbae62c266f24526affd029472f41b94])
I20250903 20:05:08.334699 27669 tablet_loader.cc:96] loaded metadata for tablet fff44282202c4dfcb74743e5e8d09fc5 (table default.Table_b616c8470952474b8bdef7d06f6da59c [id=8622aba89ccc48c5ac6a28e038fa9db5])
I20250903 20:05:08.334952 27669 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 20:05:08.335978 27669 catalog_manager.cc:1261] Loaded cluster ID: 07b8834eda114abdb992ba4c63cadc24
I20250903 20:05:08.336064 27669 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 20:05:08.337180 27646 raft_consensus.cc:491] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:08.337374 27646 raft_consensus.cc:513] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:08.337970 27669 catalog_manager.cc:1506] Loading token signing keys...
I20250903 20:05:08.339083 27669 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Loaded TSK: 0
I20250903 20:05:08.339474 27646 leader_election.cc:290] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:08.340095 27669 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 20:05:08.347121 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fff44282202c4dfcb74743e5e8d09fc5" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:08.347555 27527 raft_consensus.cc:2466] T fff44282202c4dfcb74743e5e8d09fc5 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
W20250903 20:05:08.347721 27184 leader_election.cc:343] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: fff44282202c4dfcb74743e5e8d09fc5
I20250903 20:05:08.348081 27183 leader_election.cc:304] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:08.348485 27646 raft_consensus.cc:2802] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:08.348619 27646 raft_consensus.cc:491] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:08.348708 27646 raft_consensus.cc:3058] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:08.350744 27646 raft_consensus.cc:513] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:08.351397 27646 leader_election.cc:290] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:08.351946 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fff44282202c4dfcb74743e5e8d09fc5" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:08.352156 27527 raft_consensus.cc:3058] T fff44282202c4dfcb74743e5e8d09fc5 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:08.352252 27184 leader_election.cc:343] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: fff44282202c4dfcb74743e5e8d09fc5
I20250903 20:05:08.354197 27527 raft_consensus.cc:2466] T fff44282202c4dfcb74743e5e8d09fc5 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:08.354648 27183 leader_election.cc:304] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:08.354964 27646 raft_consensus.cc:2802] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:08.355408 27646 raft_consensus.cc:695] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:08.356009 27646 consensus_queue.cc:237] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:08.361707 27066 catalog_manager.cc:5582] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a reported cstate change: term changed from 0 to 1, leader changed from <none> to 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65). New cstate: current_term: 1 leader_uuid: "89f8f0ee68b940db9ce250c0604f934a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:08.362511 27066 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet fff44282202c4dfcb74743e5e8d09fc5 (table default.Table_b616c8470952474b8bdef7d06f6da59c [id=8622aba89ccc48c5ac6a28e038fa9db5]). Expected version 1 got 0
I20250903 20:05:08.366534 27004 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 3 FOLLOWER]: Refusing update from remote peer 35bdb0090d8747059b5ee103e786e979: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
W20250903 20:05:08.366875 27053 consensus_peers.cc:489] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 -> Peer 8857b5d4ea1e4f45aafb8e029837cca2 (127.26.65.126:39015): Couldn't send request to peer 8857b5d4ea1e4f45aafb8e029837cca2. Status: Network error: Client connection negotiation failed: client connection to 127.26.65.126:39015: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:08.367210 27668 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250903 20:05:08.368070 27064 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b762f9202b4141ccb73d262c8b6314a4" } modify_external_catalogs: true
I20250903 20:05:08.368125 27065 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3e4560347ecf4f09a263e7d0d562105d" } modify_external_catalogs: true
I20250903 20:05:08.368423 27064 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b762f9202b4141ccb73d262c8b6314a4" } modify_external_catalogs: true
I20250903 20:05:08.368664 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_cbebd771513c4bfaa06d0776b4a10303" } new_table_name: "default.Table_aa89054dd40a407388ac79ee2effd5ef" modify_external_catalogs: true
I20250903 20:05:08.368582 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_8d1a53fac8c24b9c8b2b4549fa8d1dbc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:08.370492 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d1a53fac8c24b9c8b2b4549fa8d1dbc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:08.370572 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b616c8470952474b8bdef7d06f6da59c" } new_table_name: "default.Table_514f576d8f594b9b92981f4a2ad81f1b" modify_external_catalogs: true
I20250903 20:05:08.371176 27061 master_service.cc:948] ReplaceTablet: received request to replace tablet fff44282202c4dfcb74743e5e8d09fc5 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:08.368420 27065 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3e4560347ecf4f09a263e7d0d562105d" } modify_external_catalogs: true
I20250903 20:05:08.371702 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_59b61e83a1dd4c7386d0ffca89f82104"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:08.372498 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59b61e83a1dd4c7386d0ffca89f82104 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:08.373744 27058 master_service.cc:948] ReplaceTablet: received request to replace tablet 4efd72d607f34650b4bcc832770fa2f7 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:08.374074 27057 master_service.cc:948] ReplaceTablet: received request to replace tablet af116a66fda74949836bda28300565f3 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:08.377802 27662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35bdb0090d8747059b5ee103e786e979. Latest consensus state: current_term: 3 leader_uuid: "35bdb0090d8747059b5ee103e786e979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:08.378043 27662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 20:05:08.379295 27668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "35bdb0090d8747059b5ee103e786e979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:08.379498 27668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [sys.catalog]: This master's current role is: LEADER
I20250903 20:05:08.388355 27662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "35bdb0090d8747059b5ee103e786e979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:08.388582 27662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [sys.catalog]: This master's current role is: FOLLOWER
W20250903 20:05:08.390743 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet af116a66fda74949836bda28300565f3 consensus state has no leader
I20250903 20:05:08.391348 27063 master_service.cc:948] ReplaceTablet: received request to replace tablet fff44282202c4dfcb74743e5e8d09fc5 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:08.391731 27060 master_service.cc:948] ReplaceTablet: received request to replace tablet 01bef73ab62d4d10a88c3478d4bd3b0c from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:08.392429 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 01bef73ab62d4d10a88c3478d4bd3b0c
I20250903 20:05:08.402103 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 4efd72d607f34650b4bcc832770fa2f7
I20250903 20:05:08.407352 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet af116a66fda74949836bda28300565f3
I20250903 20:05:08.407965 27057 catalog_manager.cc:6652] ReplaceTablet: tablet af116a66fda74949836bda28300565f3 deleted and replaced by tablet 0bcfca89bc3b44d78bc1b875c796329b
I20250903 20:05:08.411989 27505 tablet_service.cc:1515] Processing DeleteTablet for tablet 01bef73ab62d4d10a88c3478d4bd3b0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:08.412631 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 4efd72d607f34650b4bcc832770fa2f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:08.412890 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet af116a66fda74949836bda28300565f3 with delete_type TABLET_DATA_DELETED (replaced by tablet 0bcfca89bc3b44d78bc1b875c796329b) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:08.417589 27228 tablet_service.cc:1515] Processing DeleteTablet for tablet 01bef73ab62d4d10a88c3478d4bd3b0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:08.417867 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet af116a66fda74949836bda28300565f3 with delete_type TABLET_DATA_DELETED (replaced by tablet 0bcfca89bc3b44d78bc1b875c796329b) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:08.418478 27684 tablet_replica.cc:331] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:08.418788 27685 tablet_replica.cc:331] T 01bef73ab62d4d10a88c3478d4bd3b0c P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:08.418993 27684 raft_consensus.cc:2241] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:08.419061 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet fff44282202c4dfcb74743e5e8d09fc5
I20250903 20:05:08.419171 27685 raft_consensus.cc:2241] T 01bef73ab62d4d10a88c3478d4bd3b0c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:08.419528 27685 raft_consensus.cc:2270] T 01bef73ab62d4d10a88c3478d4bd3b0c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:08.419580 27684 pending_rounds.cc:70] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f: Trying to abort 1 pending ops.
I20250903 20:05:08.419592 27061 catalog_manager.cc:6652] ReplaceTablet: tablet fff44282202c4dfcb74743e5e8d09fc5 deleted and replaced by tablet 6dc3e66c6446439abd3eba441f4c782f
I20250903 20:05:08.419726 27684 pending_rounds.cc:77] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384903292264448 op_type: NO_OP noop_request { }
I20250903 20:05:08.419932 27684 raft_consensus.cc:2887] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:08.420058 27684 raft_consensus.cc:2270] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:08.417678 27229 tablet_service.cc:1515] Processing DeleteTablet for tablet 4efd72d607f34650b4bcc832770fa2f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:08.420665 27505 tablet_service.cc:1515] Processing DeleteTablet for tablet fff44282202c4dfcb74743e5e8d09fc5 with delete_type TABLET_DATA_DELETED (replaced by tablet 6dc3e66c6446439abd3eba441f4c782f) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:08.421422 27685 ts_tablet_manager.cc:1905] T 01bef73ab62d4d10a88c3478d4bd3b0c P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:08.421478 27228 tablet_service.cc:1515] Processing DeleteTablet for tablet fff44282202c4dfcb74743e5e8d09fc5 with delete_type TABLET_DATA_DELETED (replaced by tablet 6dc3e66c6446439abd3eba441f4c782f) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:08.422600 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_a5683637bfe9461e8a8c7274069d042f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:08.422928 27684 ts_tablet_manager.cc:1905] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:08.422955 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b4235fb7128c4c8dac00f0e30456d8fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:08.421967 27366 tablet_service.cc:1515] Processing DeleteTablet for tablet 01bef73ab62d4d10a88c3478d4bd3b0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:08.423302 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5683637bfe9461e8a8c7274069d042f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:08.422591 27363 tablet_service.cc:1515] Processing DeleteTablet for tablet fff44282202c4dfcb74743e5e8d09fc5 with delete_type TABLET_DATA_DELETED (replaced by tablet 6dc3e66c6446439abd3eba441f4c782f) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:08.423570 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4235fb7128c4c8dac00f0e30456d8fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:08.422068 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 4efd72d607f34650b4bcc832770fa2f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:08.425101 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet af116a66fda74949836bda28300565f3 with delete_type TABLET_DATA_DELETED (replaced by tablet 0bcfca89bc3b44d78bc1b875c796329b) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:08.425557 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_6106b8f9be34419983b266581e83bed8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:08.425949 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_562dd57ebbc14070b3c772c08e93d7df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:08.426108 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6106b8f9be34419983b266581e83bed8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:08.426298 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_723d4b821bf24b91b5319e60778cf204"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:08.426487 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_562dd57ebbc14070b3c772c08e93d7df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:08.426798 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_723d4b821bf24b91b5319e60778cf204 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:08.428064 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_78d83e20caac4177a194a3c3434d4ffc" } new_table_name: "default.Table_12ae79d84647470fa2032d61a167b96c" modify_external_catalogs: true
I20250903 20:05:08.428340 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_aa89054dd40a407388ac79ee2effd5ef" } new_table_name: "default.Table_5546f9bd9a2944c3ac968de1de6dda20" modify_external_catalogs: true
I20250903 20:05:08.430148 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_d0d3dd3e485546579b73419a892afba3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:08.425578 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ba07d5582e744dc5a2eb94cbfec74a5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:08.431913 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba07d5582e744dc5a2eb94cbfec74a5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:08.432040 27684 ts_tablet_manager.cc:1918] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:08.432627 27684 log.cc:1199] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/01bef73ab62d4d10a88c3478d4bd3b0c
I20250903 20:05:08.433141 27684 ts_tablet_manager.cc:1939] T 01bef73ab62d4d10a88c3478d4bd3b0c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
W20250903 20:05:08.433482 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0d3dd3e485546579b73419a892afba3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:08.435055 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 01bef73ab62d4d10a88c3478d4bd3b0c (table default.Table_3e4560347ecf4f09a263e7d0d562105d [id=a85e4c476fbe422d8aaee54022e6dfde]) successfully deleted
I20250903 20:05:08.435949 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_8d1a53fac8c24b9c8b2b4549fa8d1dbc" } new_table_name: "default.Table_f7e6051d454149d685213bea3186fb79" modify_external_catalogs: true
I20250903 20:05:08.438769 27527 raft_consensus.cc:1273] T fff44282202c4dfcb74743e5e8d09fc5 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Refusing update from remote peer 89f8f0ee68b940db9ce250c0604f934a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 20:05:08.439376 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_99cc3106e9a94dfc85c7a9e145fc442b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:08.439934 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99cc3106e9a94dfc85c7a9e145fc442b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:08.440761 27184 consensus_peers.cc:489] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fff44282202c4dfcb74743e5e8d09fc5. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:08.441124 27685 ts_tablet_manager.cc:1918] T 01bef73ab62d4d10a88c3478d4bd3b0c P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:08.441120 27646 consensus_queue.cc:1035] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:08.441411 27684 tablet_replica.cc:331] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:08.441304 27685 log.cc:1199] T 01bef73ab62d4d10a88c3478d4bd3b0c P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/01bef73ab62d4d10a88c3478d4bd3b0c
I20250903 20:05:08.441612 27684 raft_consensus.cc:2241] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:08.441879 27684 pending_rounds.cc:70] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f: Trying to abort 1 pending ops.
I20250903 20:05:08.441937 27684 pending_rounds.cc:77] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384904224841728 op_type: NO_OP noop_request { }
I20250903 20:05:08.442132 27685 ts_tablet_manager.cc:1939] T 01bef73ab62d4d10a88c3478d4bd3b0c P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:08.442128 27684 raft_consensus.cc:2887] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:08.442301 27684 raft_consensus.cc:2270] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:08.443236 27684 ts_tablet_manager.cc:1905] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:08.443939 27685 tablet_replica.cc:331] T af116a66fda74949836bda28300565f3 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:08.444136 27685 raft_consensus.cc:2241] T af116a66fda74949836bda28300565f3 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:08.444285 27685 raft_consensus.cc:2270] T af116a66fda74949836bda28300565f3 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:08.444780 27685 ts_tablet_manager.cc:1905] T af116a66fda74949836bda28300565f3 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:08.445837 27050 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet af116a66fda74949836bda28300565f3 (table default.Table_aa89054dd40a407388ac79ee2effd5ef [id=cbae62c266f24526affd029472f41b94], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet af116a66fda74949836bda28300565f3 consensus state has no leader
W20250903 20:05:08.452787 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0bcfca89bc3b44d78bc1b875c796329b has no consensus state
I20250903 20:05:08.452730 27684 ts_tablet_manager.cc:1918] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:08.453073 27684 log.cc:1199] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/4efd72d607f34650b4bcc832770fa2f7
I20250903 20:05:08.453536 27684 ts_tablet_manager.cc:1939] T 4efd72d607f34650b4bcc832770fa2f7 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:08.454656 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 4efd72d607f34650b4bcc832770fa2f7 (table default.Table_b762f9202b4141ccb73d262c8b6314a4 [id=0ab13735ce35424286be08f4f966ddc4]) successfully deleted
I20250903 20:05:08.455013 27684 tablet_replica.cc:331] T af116a66fda74949836bda28300565f3 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:08.455206 27684 raft_consensus.cc:2241] T af116a66fda74949836bda28300565f3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:08.455408 27684 raft_consensus.cc:2270] T af116a66fda74949836bda28300565f3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:08.455958 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_514f576d8f594b9b92981f4a2ad81f1b" } modify_external_catalogs: true
I20250903 20:05:08.455919 27684 ts_tablet_manager.cc:1905] T af116a66fda74949836bda28300565f3 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:08.456387 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_514f576d8f594b9b92981f4a2ad81f1b" } modify_external_catalogs: true
I20250903 20:05:08.456775 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 01bef73ab62d4d10a88c3478d4bd3b0c (table default.Table_3e4560347ecf4f09a263e7d0d562105d [id=a85e4c476fbe422d8aaee54022e6dfde]) successfully deleted
I20250903 20:05:08.456898 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_59b61e83a1dd4c7386d0ffca89f82104" } modify_external_catalogs: true
I20250903 20:05:08.457124 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_59b61e83a1dd4c7386d0ffca89f82104" } modify_external_catalogs: true
W20250903 20:05:08.457834 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9d37a5d51a184c76bb8496679dfe3991 consensus state has no leader
I20250903 20:05:08.459365 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a5683637bfe9461e8a8c7274069d042f" } modify_external_catalogs: true
I20250903 20:05:08.459573 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a5683637bfe9461e8a8c7274069d042f" } modify_external_catalogs: true
I20250903 20:05:08.460088 27685 ts_tablet_manager.cc:1918] T af116a66fda74949836bda28300565f3 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:08.460227 27685 log.cc:1199] T af116a66fda74949836bda28300565f3 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/af116a66fda74949836bda28300565f3
I20250903 20:05:08.460621 27685 ts_tablet_manager.cc:1939] T af116a66fda74949836bda28300565f3 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:08.461727 27685 tablet_replica.cc:331] T 4efd72d607f34650b4bcc832770fa2f7 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:08.465056 27684 ts_tablet_manager.cc:1918] T af116a66fda74949836bda28300565f3 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:08.465207 27684 log.cc:1199] T af116a66fda74949836bda28300565f3 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/af116a66fda74949836bda28300565f3
I20250903 20:05:08.465580 27684 ts_tablet_manager.cc:1939] T af116a66fda74949836bda28300565f3 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:08.467454 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet af116a66fda74949836bda28300565f3 (table default.Table_5546f9bd9a2944c3ac968de1de6dda20 [id=cbae62c266f24526affd029472f41b94]) successfully deleted
I20250903 20:05:08.467648 27684 tablet_replica.cc:331] T fff44282202c4dfcb74743e5e8d09fc5 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:08.468261 27505 tablet_service.cc:1468] Processing CreateTablet for tablet ea10476553384e8fb9cf75fd1c6595da (DEFAULT_TABLE table=default.Table_59b61e83a1dd4c7386d0ffca89f82104 [id=f378777cae174d209abd3bd7f5766259]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.468451 27504 tablet_service.cc:1468] Processing CreateTablet for tablet b197747f8d134f978a2cd1fe3425a5d5 (DEFAULT_TABLE table=default.Table_8d1a53fac8c24b9c8b2b4549fa8d1dbc [id=88e0361a865a4b28a08e6c7f30dee9e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.468690 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea10476553384e8fb9cf75fd1c6595da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.468905 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 0bcfca89bc3b44d78bc1b875c796329b (DEFAULT_TABLE table=default.Table_5546f9bd9a2944c3ac968de1de6dda20 [id=cbae62c266f24526affd029472f41b94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.469233 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bcfca89bc3b44d78bc1b875c796329b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.469362 27364 tablet_service.cc:1468] Processing CreateTablet for tablet ea10476553384e8fb9cf75fd1c6595da (DEFAULT_TABLE table=default.Table_59b61e83a1dd4c7386d0ffca89f82104 [id=f378777cae174d209abd3bd7f5766259]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.470508 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 6dc3e66c6446439abd3eba441f4c782f (DEFAULT_TABLE table=default.Table_514f576d8f594b9b92981f4a2ad81f1b [id=8622aba89ccc48c5ac6a28e038fa9db5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.470868 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dc3e66c6446439abd3eba441f4c782f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.471187 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea10476553384e8fb9cf75fd1c6595da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.472975 27366 tablet_service.cc:1468] Processing CreateTablet for tablet b197747f8d134f978a2cd1fe3425a5d5 (DEFAULT_TABLE table=default.Table_8d1a53fac8c24b9c8b2b4549fa8d1dbc [id=88e0361a865a4b28a08e6c7f30dee9e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.473412 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b197747f8d134f978a2cd1fe3425a5d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.474653 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b197747f8d134f978a2cd1fe3425a5d5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:08.475965 27183 consensus_peers.cc:489] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:08.481575 27685 raft_consensus.cc:2241] T 4efd72d607f34650b4bcc832770fa2f7 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:08.481798 27685 raft_consensus.cc:2270] T 4efd72d607f34650b4bcc832770fa2f7 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:08.482409 27685 ts_tablet_manager.cc:1905] T 4efd72d607f34650b4bcc832770fa2f7 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:08.482491 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet af116a66fda74949836bda28300565f3 (table default.Table_5546f9bd9a2944c3ac968de1de6dda20 [id=cbae62c266f24526affd029472f41b94]) successfully deleted
I20250903 20:05:08.482945 27228 tablet_service.cc:1468] Processing CreateTablet for tablet b197747f8d134f978a2cd1fe3425a5d5 (DEFAULT_TABLE table=default.Table_8d1a53fac8c24b9c8b2b4549fa8d1dbc [id=88e0361a865a4b28a08e6c7f30dee9e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.483319 27228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b197747f8d134f978a2cd1fe3425a5d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.483268 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 0bcfca89bc3b44d78bc1b875c796329b (DEFAULT_TABLE table=default.Table_5546f9bd9a2944c3ac968de1de6dda20 [id=cbae62c266f24526affd029472f41b94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.483752 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bcfca89bc3b44d78bc1b875c796329b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.484059 27684 raft_consensus.cc:2241] T fff44282202c4dfcb74743e5e8d09fc5 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:08.484655 27684 pending_rounds.cc:70] T fff44282202c4dfcb74743e5e8d09fc5 P e72ef1dbcc5143cf9f04a808f101c80f: Trying to abort 2 pending ops.
I20250903 20:05:08.484732 27684 pending_rounds.cc:77] T fff44282202c4dfcb74743e5e8d09fc5 P e72ef1dbcc5143cf9f04a808f101c80f: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7196384904951836672 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "fff44282202c4dfcb74743e5e8d09fc5" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 2 new_table_name: "default.Table_514f576d8f594b9b92981f4a2ad81f1b" dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" new_extra_config { } }
I20250903 20:05:08.485370 27684 pending_rounds.cc:77] T fff44282202c4dfcb74743e5e8d09fc5 P e72ef1dbcc5143cf9f04a808f101c80f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384904630210560 op_type: NO_OP noop_request { }
I20250903 20:05:08.485491 27684 raft_consensus.cc:2887] T fff44282202c4dfcb74743e5e8d09fc5 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:08.485615 27684 raft_consensus.cc:2270] T fff44282202c4dfcb74743e5e8d09fc5 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:08.486439 27684 ts_tablet_manager.cc:1905] T fff44282202c4dfcb74743e5e8d09fc5 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:08.486490 27363 tablet_service.cc:1468] Processing CreateTablet for tablet 6dc3e66c6446439abd3eba441f4c782f (DEFAULT_TABLE table=default.Table_514f576d8f594b9b92981f4a2ad81f1b [id=8622aba89ccc48c5ac6a28e038fa9db5]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:08.486609 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 01bef73ab62d4d10a88c3478d4bd3b0c because the tablet was not found. No further retry: Not found: Tablet not found: 01bef73ab62d4d10a88c3478d4bd3b0c
W20250903 20:05:08.486877 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet fff44282202c4dfcb74743e5e8d09fc5 because the tablet was not found. No further retry: Not found: Tablet not found: fff44282202c4dfcb74743e5e8d09fc5
I20250903 20:05:08.486928 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dc3e66c6446439abd3eba441f4c782f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:08.487100 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 4efd72d607f34650b4bcc832770fa2f7 because the tablet was not found. No further retry: Not found: Tablet not found: 4efd72d607f34650b4bcc832770fa2f7
I20250903 20:05:08.487169 27229 tablet_service.cc:1468] Processing CreateTablet for tablet 0bcfca89bc3b44d78bc1b875c796329b (DEFAULT_TABLE table=default.Table_5546f9bd9a2944c3ac968de1de6dda20 [id=cbae62c266f24526affd029472f41b94]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:08.487310 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet af116a66fda74949836bda28300565f3 because the tablet was not found. No further retry: Not found: Tablet not found: af116a66fda74949836bda28300565f3
I20250903 20:05:08.487550 27229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bcfca89bc3b44d78bc1b875c796329b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.490487 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 6dc3e66c6446439abd3eba441f4c782f (DEFAULT_TABLE table=default.Table_514f576d8f594b9b92981f4a2ad81f1b [id=8622aba89ccc48c5ac6a28e038fa9db5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.490852 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dc3e66c6446439abd3eba441f4c782f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.491228 27647 raft_consensus.cc:491] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:08.491359 27647 raft_consensus.cc:513] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:08.492029 27647 leader_election.cc:290] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:08.493108 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d37a5d51a184c76bb8496679dfe3991" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:08.493359 27249 raft_consensus.cc:2466] T 9d37a5d51a184c76bb8496679dfe3991 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:08.494331 27462 leader_election.cc:304] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:08.494889 27227 tablet_service.cc:1468] Processing CreateTablet for tablet ea10476553384e8fb9cf75fd1c6595da (DEFAULT_TABLE table=default.Table_59b61e83a1dd4c7386d0ffca89f82104 [id=f378777cae174d209abd3bd7f5766259]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:08.494902 27462 leader_election.cc:343] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 9d37a5d51a184c76bb8496679dfe3991
I20250903 20:05:08.495218 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea10476553384e8fb9cf75fd1c6595da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.495298 27647 raft_consensus.cc:2802] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:08.495429 27647 raft_consensus.cc:491] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:08.495551 27647 raft_consensus.cc:3058] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:08.496153 27684 ts_tablet_manager.cc:1918] T fff44282202c4dfcb74743e5e8d09fc5 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 20:05:08.496304 27684 log.cc:1199] T fff44282202c4dfcb74743e5e8d09fc5 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/fff44282202c4dfcb74743e5e8d09fc5
I20250903 20:05:08.496726 27684 ts_tablet_manager.cc:1939] T fff44282202c4dfcb74743e5e8d09fc5 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:08.498677 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet fff44282202c4dfcb74743e5e8d09fc5 (table default.Table_514f576d8f594b9b92981f4a2ad81f1b [id=8622aba89ccc48c5ac6a28e038fa9db5]) successfully deleted
I20250903 20:05:08.499346 27647 raft_consensus.cc:513] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:08.500587 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d37a5d51a184c76bb8496679dfe3991" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:08.500804 27249 raft_consensus.cc:3058] T 9d37a5d51a184c76bb8496679dfe3991 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:08.500950 27685 ts_tablet_manager.cc:1918] T 4efd72d607f34650b4bcc832770fa2f7 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:08.501070 27685 log.cc:1199] T 4efd72d607f34650b4bcc832770fa2f7 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/4efd72d607f34650b4bcc832770fa2f7
W20250903 20:05:08.501135 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b197747f8d134f978a2cd1fe3425a5d5 consensus state has no leader
I20250903 20:05:08.501510 27685 ts_tablet_manager.cc:1939] T 4efd72d607f34650b4bcc832770fa2f7 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:08.502511 27685 tablet_replica.cc:331] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:08.502552 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 4efd72d607f34650b4bcc832770fa2f7 (table default.Table_b762f9202b4141ccb73d262c8b6314a4 [id=0ab13735ce35424286be08f4f966ddc4]) successfully deleted
I20250903 20:05:08.504287 27249 raft_consensus.cc:2466] T 9d37a5d51a184c76bb8496679dfe3991 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:08.504859 27462 leader_election.cc:304] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:08.505213 27657 raft_consensus.cc:2802] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
W20250903 20:05:08.515244 27462 leader_election.cc:343] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 9d37a5d51a184c76bb8496679dfe3991
I20250903 20:05:08.515764 27647 leader_election.cc:290] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:08.521589 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 6dc3e66c6446439abd3eba441f4c782f
I20250903 20:05:08.521589 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet ea10476553384e8fb9cf75fd1c6595da
I20250903 20:05:08.522881 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 6dc3e66c6446439abd3eba441f4c782f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:08.524024 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet ea10476553384e8fb9cf75fd1c6595da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:08.524655 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet 6dc3e66c6446439abd3eba441f4c782f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:08.524896 27224 tablet_service.cc:1515] Processing DeleteTablet for tablet ea10476553384e8fb9cf75fd1c6595da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:08.525213 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 6dc3e66c6446439abd3eba441f4c782f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6dc3e66c6446439abd3eba441f4c782f already in progress: creating tablet
W20250903 20:05:08.525599 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet ea10476553384e8fb9cf75fd1c6595da because the tablet was not found. No further retry: Not found: Tablet not found: ea10476553384e8fb9cf75fd1c6595da
I20250903 20:05:08.526633 27061 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 00540b3b97dc4f0fa461f3d5ac12cc13
I20250903 20:05:08.536152 27572 heartbeater.cc:499] Master 127.26.65.124:40911 was elected leader, sending a full tablet report...
I20250903 20:05:08.536429 27503 tablet_service.cc:1515] Processing DeleteTablet for tablet ea10476553384e8fb9cf75fd1c6595da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:08.536643 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet fff44282202c4dfcb74743e5e8d09fc5 with delete_type TABLET_DATA_DELETED (replaced by tablet 6dc3e66c6446439abd3eba441f4c782f) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:08.537457 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet ea10476553384e8fb9cf75fd1c6595da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea10476553384e8fb9cf75fd1c6595da already in progress: creating tablet
W20250903 20:05:08.537809 27051 catalog_manager.cc:4895] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet fff44282202c4dfcb74743e5e8d09fc5 because the tablet was not found. No further retry: Not found: Tablet not found: fff44282202c4dfcb74743e5e8d09fc5
I20250903 20:05:08.544353 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 6dc3e66c6446439abd3eba441f4c782f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:08.545151 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 6dc3e66c6446439abd3eba441f4c782f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6dc3e66c6446439abd3eba441f4c782f already in progress: creating tablet
I20250903 20:05:08.546389 27691 tablet.cc:1722] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:08.548494 27685 raft_consensus.cc:2241] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:08.549239 27294 heartbeater.cc:499] Master 127.26.65.124:40911 was elected leader, sending a full tablet report...
I20250903 20:05:08.549818 27692 tablet_bootstrap.cc:492] T ea10476553384e8fb9cf75fd1c6595da P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:08.552481 27692 tablet_bootstrap.cc:654] T ea10476553384e8fb9cf75fd1c6595da P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:08.556190 27695 tablet_bootstrap.cc:492] T ea10476553384e8fb9cf75fd1c6595da P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:08.557703 27692 tablet_bootstrap.cc:492] T ea10476553384e8fb9cf75fd1c6595da P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:08.557956 27692 ts_tablet_manager.cc:1397] T ea10476553384e8fb9cf75fd1c6595da P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20250903 20:05:08.558086 27695 tablet_bootstrap.cc:654] T ea10476553384e8fb9cf75fd1c6595da P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:08.558969 27692 raft_consensus.cc:357] T ea10476553384e8fb9cf75fd1c6595da P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:08.559269 27692 raft_consensus.cc:383] T ea10476553384e8fb9cf75fd1c6595da P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:08.559800 27692 raft_consensus.cc:738] T ea10476553384e8fb9cf75fd1c6595da P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:08.560428 27692 consensus_queue.cc:260] T ea10476553384e8fb9cf75fd1c6595da P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:08.561264 27692 ts_tablet_manager.cc:1428] T ea10476553384e8fb9cf75fd1c6595da P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250903 20:05:08.561764 27692 tablet_bootstrap.cc:492] T 6dc3e66c6446439abd3eba441f4c782f P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:08.564334 27692 tablet_bootstrap.cc:654] T 6dc3e66c6446439abd3eba441f4c782f P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:08.566828 27695 tablet_bootstrap.cc:492] T ea10476553384e8fb9cf75fd1c6595da P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:08.567059 27695 ts_tablet_manager.cc:1397] T ea10476553384e8fb9cf75fd1c6595da P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20250903 20:05:08.567907 27695 raft_consensus.cc:357] T ea10476553384e8fb9cf75fd1c6595da P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:08.568177 27695 raft_consensus.cc:383] T ea10476553384e8fb9cf75fd1c6595da P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:08.568310 27695 raft_consensus.cc:738] T ea10476553384e8fb9cf75fd1c6595da P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:08.568675 27695 consensus_queue.cc:260] T ea10476553384e8fb9cf75fd1c6595da P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:08.569260 27695 ts_tablet_manager.cc:1428] T ea10476553384e8fb9cf75fd1c6595da P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250903 20:05:08.569617 27695 tablet_bootstrap.cc:492] T 6dc3e66c6446439abd3eba441f4c782f P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:08.571424 27695 tablet_bootstrap.cc:654] T 6dc3e66c6446439abd3eba441f4c782f P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:08.585304 27692 tablet_bootstrap.cc:492] T 6dc3e66c6446439abd3eba441f4c782f P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:08.585510 27692 ts_tablet_manager.cc:1397] T 6dc3e66c6446439abd3eba441f4c782f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.001s
I20250903 20:05:08.585589 27695 tablet_bootstrap.cc:492] T 6dc3e66c6446439abd3eba441f4c782f P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:08.585814 27695 ts_tablet_manager.cc:1397] T 6dc3e66c6446439abd3eba441f4c782f P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.008s
I20250903 20:05:08.586329 27692 raft_consensus.cc:357] T 6dc3e66c6446439abd3eba441f4c782f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:08.586553 27692 raft_consensus.cc:383] T 6dc3e66c6446439abd3eba441f4c782f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:08.586648 27692 raft_consensus.cc:738] T 6dc3e66c6446439abd3eba441f4c782f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:08.586897 27692 consensus_queue.cc:260] T 6dc3e66c6446439abd3eba441f4c782f P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:08.586889 27695 raft_consensus.cc:357] T 6dc3e66c6446439abd3eba441f4c782f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:08.587153 27695 raft_consensus.cc:383] T 6dc3e66c6446439abd3eba441f4c782f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:08.587272 27695 raft_consensus.cc:738] T 6dc3e66c6446439abd3eba441f4c782f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:08.587373 27692 ts_tablet_manager.cc:1428] T 6dc3e66c6446439abd3eba441f4c782f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:08.587723 27692 tablet_bootstrap.cc:492] T b197747f8d134f978a2cd1fe3425a5d5 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:08.589591 27692 tablet_bootstrap.cc:654] T b197747f8d134f978a2cd1fe3425a5d5 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:08.589654 27695 consensus_queue.cc:260] T 6dc3e66c6446439abd3eba441f4c782f P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:08.593747 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5546f9bd9a2944c3ac968de1de6dda20" } new_table_name: "default.Table_c106f52e6b6b458391a12b3043e7eedf" modify_external_catalogs: true
I20250903 20:05:08.595551 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_0bf705fb6eb5452385b532032694acfc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:08.596299 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0bf705fb6eb5452385b532032694acfc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:08.599316 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_562dd57ebbc14070b3c772c08e93d7df" } modify_external_catalogs: true
I20250903 20:05:08.599498 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_562dd57ebbc14070b3c772c08e93d7df" } modify_external_catalogs: true
I20250903 20:05:08.599748 27695 ts_tablet_manager.cc:1428] T 6dc3e66c6446439abd3eba441f4c782f P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20250903 20:05:08.604563 27695 tablet_bootstrap.cc:492] T 0bcfca89bc3b44d78bc1b875c796329b P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:08.605831 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d0d3dd3e485546579b73419a892afba3" } modify_external_catalogs: true
I20250903 20:05:08.606115 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d0d3dd3e485546579b73419a892afba3" } modify_external_catalogs: true
I20250903 20:05:08.606844 27061 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 9d37a5d51a184c76bb8496679dfe3991 (table default.Table_12ae79d84647470fa2032d61a167b96c [id=374b9abdf6a54b38876a96d8ec4a0368]). Expected version 2 got 0
I20250903 20:05:08.607052 27061 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 0bcfca89bc3b44d78bc1b875c796329b (table default.Table_5546f9bd9a2944c3ac968de1de6dda20 [id=cbae62c266f24526affd029472f41b94]). Expected version 3 got 0
I20250903 20:05:08.607312 27061 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet b197747f8d134f978a2cd1fe3425a5d5 (table default.Table_f7e6051d454149d685213bea3186fb79 [id=88e0361a865a4b28a08e6c7f30dee9e3]). Expected version 1 got 0
I20250903 20:05:08.607710 27058 master_service.cc:948] ReplaceTablet: received request to replace tablet b197747f8d134f978a2cd1fe3425a5d5 from {username='slave'} at 127.0.0.1:45936
W20250903 20:05:08.607810 27059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0bcfca89bc3b44d78bc1b875c796329b consensus state has no leader
I20250903 20:05:08.610121 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 218f51a5391141b8941a2b40980147fd (DEFAULT_TABLE table=default.Table_d0d3dd3e485546579b73419a892afba3 [id=86d4245a96f94a789a163b51ae32906e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.610597 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 218f51a5391141b8941a2b40980147fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.610713 27227 tablet_service.cc:1468] Processing CreateTablet for tablet 218f51a5391141b8941a2b40980147fd (DEFAULT_TABLE table=default.Table_d0d3dd3e485546579b73419a892afba3 [id=86d4245a96f94a789a163b51ae32906e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.611155 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 218f51a5391141b8941a2b40980147fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.612125 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 218f51a5391141b8941a2b40980147fd (DEFAULT_TABLE table=default.Table_d0d3dd3e485546579b73419a892afba3 [id=86d4245a96f94a789a163b51ae32906e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.612550 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 218f51a5391141b8941a2b40980147fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.613692 27695 tablet_bootstrap.cc:654] T 0bcfca89bc3b44d78bc1b875c796329b P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:08.619134 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_7d476d0d69e246e28664c7c5b84f3ba2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:08.619979 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d476d0d69e246e28664c7c5b84f3ba2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:08.621112 27685 raft_consensus.cc:2270] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:08.621353 27064 master_service.cc:948] ReplaceTablet: received request to replace tablet ea10476553384e8fb9cf75fd1c6595da from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:08.622153 27685 ts_tablet_manager.cc:1905] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:08.619154 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_723d4b821bf24b91b5319e60778cf204" } new_table_name: "default.Table_1a5c8c3256c24c7f8ef8597efa53dde5" modify_external_catalogs: true
I20250903 20:05:08.626011 27057 master_service.cc:948] ReplaceTablet: received request to replace tablet 218f51a5391141b8941a2b40980147fd from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:08.628962 27685 ts_tablet_manager.cc:1918] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 20:05:08.629143 27685 log.cc:1199] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/fff44282202c4dfcb74743e5e8d09fc5
I20250903 20:05:08.629588 27685 ts_tablet_manager.cc:1939] T fff44282202c4dfcb74743e5e8d09fc5 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:08.630577 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet d08c9e1e7d564886a90f6c056794becc
I20250903 20:05:08.631186 27695 tablet_bootstrap.cc:492] T 0bcfca89bc3b44d78bc1b875c796329b P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:08.631525 27695 ts_tablet_manager.cc:1397] T 0bcfca89bc3b44d78bc1b875c796329b P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.001s
I20250903 20:05:08.632022 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet d08c9e1e7d564886a90f6c056794becc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:08.632716 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet d08c9e1e7d564886a90f6c056794becc because the tablet was not found. No further retry: Not found: Tablet not found: d08c9e1e7d564886a90f6c056794becc
I20250903 20:05:08.632905 27695 raft_consensus.cc:357] T 0bcfca89bc3b44d78bc1b875c796329b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:08.633160 27695 raft_consensus.cc:383] T 0bcfca89bc3b44d78bc1b875c796329b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:08.633267 27695 raft_consensus.cc:738] T 0bcfca89bc3b44d78bc1b875c796329b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:08.633415 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet d08c9e1e7d564886a90f6c056794becc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:08.633572 27695 consensus_queue.cc:260] T 0bcfca89bc3b44d78bc1b875c796329b P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:08.634318 27695 ts_tablet_manager.cc:1428] T 0bcfca89bc3b44d78bc1b875c796329b P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20250903 20:05:08.634354 27051 catalog_manager.cc:4895] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet d08c9e1e7d564886a90f6c056794becc because the tablet was not found. No further retry: Not found: Tablet not found: d08c9e1e7d564886a90f6c056794becc
I20250903 20:05:08.635260 27685 tablet_replica.cc:331] T 6dc3e66c6446439abd3eba441f4c782f P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:08.635435 27685 raft_consensus.cc:2241] T 6dc3e66c6446439abd3eba441f4c782f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:08.635624 27685 raft_consensus.cc:2270] T 6dc3e66c6446439abd3eba441f4c782f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:08.636315 27066 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 0bcfca89bc3b44d78bc1b875c796329b (table default.Table_c106f52e6b6b458391a12b3043e7eedf [id=cbae62c266f24526affd029472f41b94]). Expected version 4 got 0
I20250903 20:05:08.637460 27695 tablet_bootstrap.cc:492] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:08.639241 27064 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 0bcfca89bc3b44d78bc1b875c796329b (table default.Table_c106f52e6b6b458391a12b3043e7eedf [id=cbae62c266f24526affd029472f41b94]). Expected version 4 got 0
I20250903 20:05:08.639557 27685 ts_tablet_manager.cc:1905] T 6dc3e66c6446439abd3eba441f4c782f P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:08.641294 27227 tablet_service.cc:1515] Processing DeleteTablet for tablet d08c9e1e7d564886a90f6c056794becc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:08.641635 27060 master_service.cc:948] ReplaceTablet: received request to replace tablet 9d37a5d51a184c76bb8496679dfe3991 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:08.641745 27066 master_service.cc:948] ReplaceTablet: received request to replace tablet 0bcfca89bc3b44d78bc1b875c796329b from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:08.642695 27695 tablet_bootstrap.cc:654] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:08.642756 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_4ace9616711f4778a94a0827ff79cc07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:08.643453 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ace9616711f4778a94a0827ff79cc07 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:08.644016 27657 raft_consensus.cc:695] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:08.644433 27657 consensus_queue.cc:237] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:08.645385 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 218f51a5391141b8941a2b40980147fd
I20250903 20:05:08.646333 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 218f51a5391141b8941a2b40980147fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:08.646541 27227 tablet_service.cc:1515] Processing DeleteTablet for tablet 218f51a5391141b8941a2b40980147fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:08.647078 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 218f51a5391141b8941a2b40980147fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 218f51a5391141b8941a2b40980147fd already in progress: creating tablet
I20250903 20:05:08.647387 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 218f51a5391141b8941a2b40980147fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:08.647428 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 943146f84c4a4adf89c5c2e91f9f04d4 (DEFAULT_TABLE table=default.Table_ba07d5582e744dc5a2eb94cbfec74a5b [id=322e6a14fef14f5c8623684144339103]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.647945 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 943146f84c4a4adf89c5c2e91f9f04d4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:08.648159 27051 catalog_manager.cc:4895] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 218f51a5391141b8941a2b40980147fd because the tablet was not found. No further retry: Not found: Tablet not found: 218f51a5391141b8941a2b40980147fd
I20250903 20:05:08.649138 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 943146f84c4a4adf89c5c2e91f9f04d4 (DEFAULT_TABLE table=default.Table_ba07d5582e744dc5a2eb94cbfec74a5b [id=322e6a14fef14f5c8623684144339103]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.649549 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 943146f84c4a4adf89c5c2e91f9f04d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.655455 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_82f29a4a80194976bc7c8f864ec877a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:08.656068 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82f29a4a80194976bc7c8f864ec877a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:08.656437 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet b197747f8d134f978a2cd1fe3425a5d5
I20250903 20:05:08.656886 27058 catalog_manager.cc:6652] ReplaceTablet: tablet b197747f8d134f978a2cd1fe3425a5d5 deleted and replaced by tablet 03f5657ca6e44313b1541bc93eedf621
I20250903 20:05:08.657285 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_1ba72813b9b347e1b6ef05748418aa84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:08.657770 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ba72813b9b347e1b6ef05748418aa84 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:08.658244 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet b197747f8d134f978a2cd1fe3425a5d5 with delete_type TABLET_DATA_DELETED (replaced by tablet 03f5657ca6e44313b1541bc93eedf621) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:08.658491 27695 tablet_bootstrap.cc:492] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:08.658540 27227 tablet_service.cc:1515] Processing DeleteTablet for tablet b197747f8d134f978a2cd1fe3425a5d5 with delete_type TABLET_DATA_DELETED (replaced by tablet 03f5657ca6e44313b1541bc93eedf621) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:08.658671 27695 ts_tablet_manager.cc:1397] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.002s
I20250903 20:05:08.659227 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b4235fb7128c4c8dac00f0e30456d8fa" } new_table_name: "default.Table_260dadc90c8e415c8dee534464e23000" modify_external_catalogs: true
I20250903 20:05:08.659260 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_a7abfdf9caa14a85b12e07546a8518ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:08.659502 27695 raft_consensus.cc:357] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:08.659698 27695 raft_consensus.cc:383] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250903 20:05:08.659724 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7abfdf9caa14a85b12e07546a8518ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:08.659770 27695 raft_consensus.cc:738] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:08.659758 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_d6ca992193f94f4aa3fe1ec509c04ac5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:08.660018 27695 consensus_queue.cc:260] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:08.660264 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6ca992193f94f4aa3fe1ec509c04ac5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:08.660519 27695 ts_tablet_manager.cc:1428] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20250903 20:05:08.661592 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet b197747f8d134f978a2cd1fe3425a5d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b197747f8d134f978a2cd1fe3425a5d5 already in progress: creating tablet
I20250903 20:05:08.662899 27505 tablet_service.cc:1515] Processing DeleteTablet for tablet b197747f8d134f978a2cd1fe3425a5d5 with delete_type TABLET_DATA_DELETED (replaced by tablet 03f5657ca6e44313b1541bc93eedf621) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:08.663594 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_99cc3106e9a94dfc85c7a9e145fc442b" } new_table_name: "default.Table_60698087f8f34bbe86a9994a117f2b37" modify_external_catalogs: true
I20250903 20:05:08.665448 27227 tablet_service.cc:1468] Processing CreateTablet for tablet 943146f84c4a4adf89c5c2e91f9f04d4 (DEFAULT_TABLE table=default.Table_ba07d5582e744dc5a2eb94cbfec74a5b [id=322e6a14fef14f5c8623684144339103]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.665895 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 943146f84c4a4adf89c5c2e91f9f04d4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:08.668406 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet b197747f8d134f978a2cd1fe3425a5d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b197747f8d134f978a2cd1fe3425a5d5 already in progress: creating tablet
W20250903 20:05:08.672070 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 161f2c2afc2f4c82b2025f7daf460bd6 consensus state has no leader
I20250903 20:05:08.672178 27695 tablet_bootstrap.cc:492] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:08.673025 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 9d37a5d51a184c76bb8496679dfe3991
I20250903 20:05:08.673655 27060 catalog_manager.cc:6652] ReplaceTablet: tablet 9d37a5d51a184c76bb8496679dfe3991 deleted and replaced by tablet fd499b81344c4b58bf77f3622422763c
I20250903 20:05:08.673938 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 0bcfca89bc3b44d78bc1b875c796329b
I20250903 20:05:08.673996 27695 tablet_bootstrap.cc:654] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:08.674932 27066 catalog_manager.cc:6652] ReplaceTablet: tablet 0bcfca89bc3b44d78bc1b875c796329b deleted and replaced by tablet 1d9a2508437b41f08821a29f58163159
I20250903 20:05:08.677770 27228 tablet_service.cc:1515] Processing DeleteTablet for tablet 9d37a5d51a184c76bb8496679dfe3991 with delete_type TABLET_DATA_DELETED (replaced by tablet fd499b81344c4b58bf77f3622422763c) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:08.678117 27229 tablet_service.cc:1515] Processing DeleteTablet for tablet 0bcfca89bc3b44d78bc1b875c796329b with delete_type TABLET_DATA_DELETED (replaced by tablet 1d9a2508437b41f08821a29f58163159) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:08.678476 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 0bcfca89bc3b44d78bc1b875c796329b with delete_type TABLET_DATA_DELETED (replaced by tablet 1d9a2508437b41f08821a29f58163159) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:08.679374 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 0bcfca89bc3b44d78bc1b875c796329b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bcfca89bc3b44d78bc1b875c796329b already in progress: creating tablet
I20250903 20:05:08.680333 27505 tablet_service.cc:1515] Processing DeleteTablet for tablet 9d37a5d51a184c76bb8496679dfe3991 with delete_type TABLET_DATA_DELETED (replaced by tablet fd499b81344c4b58bf77f3622422763c) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:08.680671 27684 tablet_replica.cc:331] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:08.682864 27692 tablet_bootstrap.cc:492] T b197747f8d134f978a2cd1fe3425a5d5 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:08.683089 27692 ts_tablet_manager.cc:1397] T b197747f8d134f978a2cd1fe3425a5d5 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.095s user 0.007s sys 0.000s
I20250903 20:05:08.683537 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 161f2c2afc2f4c82b2025f7daf460bd6 (DEFAULT_TABLE table=default.Table_1a5c8c3256c24c7f8ef8597efa53dde5 [id=7caef592f80a4d6cb6df4e2c3971de49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.684023 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 161f2c2afc2f4c82b2025f7daf460bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.684142 27692 raft_consensus.cc:357] T b197747f8d134f978a2cd1fe3425a5d5 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:08.684549 27692 raft_consensus.cc:383] T b197747f8d134f978a2cd1fe3425a5d5 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:08.684688 27692 raft_consensus.cc:738] T b197747f8d134f978a2cd1fe3425a5d5 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:08.684998 27692 consensus_queue.cc:260] T b197747f8d134f978a2cd1fe3425a5d5 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:08.685598 27692 ts_tablet_manager.cc:1428] T b197747f8d134f978a2cd1fe3425a5d5 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:08.687502 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_db311e4928f54b0d8fce9e417635b974"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:08.688233 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db311e4928f54b0d8fce9e417635b974 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:08.692222 27505 tablet_service.cc:1515] Processing DeleteTablet for tablet 0bcfca89bc3b44d78bc1b875c796329b with delete_type TABLET_DATA_DELETED (replaced by tablet 1d9a2508437b41f08821a29f58163159) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:08.692776 27695 tablet_bootstrap.cc:492] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:08.692991 27695 ts_tablet_manager.cc:1397] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.006s
I20250903 20:05:08.693866 27692 tablet_bootstrap.cc:492] T 0bcfca89bc3b44d78bc1b875c796329b P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:08.694316 27695 raft_consensus.cc:357] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:08.694610 27695 raft_consensus.cc:383] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:08.694742 27695 raft_consensus.cc:738] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:08.695245 27695 consensus_queue.cc:260] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:08.696015 27695 ts_tablet_manager.cc:1428] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250903 20:05:08.700138 27695 tablet_bootstrap.cc:492] T 943146f84c4a4adf89c5c2e91f9f04d4 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:08.702534 27695 tablet_bootstrap.cc:654] T 943146f84c4a4adf89c5c2e91f9f04d4 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:08.702607 27227 tablet_service.cc:1468] Processing CreateTablet for tablet 161f2c2afc2f4c82b2025f7daf460bd6 (DEFAULT_TABLE table=default.Table_1a5c8c3256c24c7f8ef8597efa53dde5 [id=7caef592f80a4d6cb6df4e2c3971de49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.704552 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 161f2c2afc2f4c82b2025f7daf460bd6 (DEFAULT_TABLE table=default.Table_1a5c8c3256c24c7f8ef8597efa53dde5 [id=7caef592f80a4d6cb6df4e2c3971de49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.704571 27229 tablet_service.cc:1468] Processing CreateTablet for tablet d08c9e1e7d564886a90f6c056794becc (DEFAULT_TABLE table=default.Table_562dd57ebbc14070b3c772c08e93d7df [id=02a9013803934e17884b377e69dd6cac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.705138 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 161f2c2afc2f4c82b2025f7daf460bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.705758 27363 tablet_service.cc:1468] Processing CreateTablet for tablet 07361048416c43b6bc43d8737906db99 (DEFAULT_TABLE table=default.Table_b4235fb7128c4c8dac00f0e30456d8fa [id=9cea308979eb4c7f9deaa02e1c8feee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.706192 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07361048416c43b6bc43d8737906db99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.710873 27506 tablet_service.cc:1468] Processing CreateTablet for tablet d08c9e1e7d564886a90f6c056794becc (DEFAULT_TABLE table=default.Table_562dd57ebbc14070b3c772c08e93d7df [id=02a9013803934e17884b377e69dd6cac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.711493 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d08c9e1e7d564886a90f6c056794becc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.713662 27692 tablet_bootstrap.cc:654] T 0bcfca89bc3b44d78bc1b875c796329b P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:08.719183 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_3b8908bfafc443798402fc7b449f8c4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:08.719208 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_3ca111aa05124c5b91822761043b2d94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:08.719560 27695 tablet_bootstrap.cc:492] T 943146f84c4a4adf89c5c2e91f9f04d4 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:08.719741 27695 ts_tablet_manager.cc:1397] T 943146f84c4a4adf89c5c2e91f9f04d4 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.003s
W20250903 20:05:08.719907 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b8908bfafc443798402fc7b449f8c4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:08.719911 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ca111aa05124c5b91822761043b2d94 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:08.720647 27695 raft_consensus.cc:357] T 943146f84c4a4adf89c5c2e91f9f04d4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:08.720860 27695 raft_consensus.cc:383] T 943146f84c4a4adf89c5c2e91f9f04d4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:08.721545 27229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d08c9e1e7d564886a90f6c056794becc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.721546 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ba07d5582e744dc5a2eb94cbfec74a5b" } new_table_name: "default.Table_fac0948b3046453395be5ca6cf3a70b6" modify_external_catalogs: true
I20250903 20:05:08.721905 27695 raft_consensus.cc:738] T 943146f84c4a4adf89c5c2e91f9f04d4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:08.722260 27695 consensus_queue.cc:260] T 943146f84c4a4adf89c5c2e91f9f04d4 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:08.722903 27695 ts_tablet_manager.cc:1428] T 943146f84c4a4adf89c5c2e91f9f04d4 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
W20250903 20:05:08.724020 27059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 07361048416c43b6bc43d8737906db99 consensus state has no leader
I20250903 20:05:08.730998 27365 tablet_service.cc:1468] Processing CreateTablet for tablet d08c9e1e7d564886a90f6c056794becc (DEFAULT_TABLE table=default.Table_562dd57ebbc14070b3c772c08e93d7df [id=02a9013803934e17884b377e69dd6cac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.731909 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d08c9e1e7d564886a90f6c056794becc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.732729 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 161f2c2afc2f4c82b2025f7daf460bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.735989 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet 9d37a5d51a184c76bb8496679dfe3991 with delete_type TABLET_DATA_DELETED (replaced by tablet fd499b81344c4b58bf77f3622422763c) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:08.736756 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 9d37a5d51a184c76bb8496679dfe3991 because the tablet was not found. No further retry: Not found: Tablet not found: 9d37a5d51a184c76bb8496679dfe3991
I20250903 20:05:08.737231 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0bf705fb6eb5452385b532032694acfc" } modify_external_catalogs: true
I20250903 20:05:08.737329 27059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f7e6051d454149d685213bea3186fb79" } modify_external_catalogs: true
I20250903 20:05:08.737450 27059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f7e6051d454149d685213bea3186fb79" } modify_external_catalogs: true
I20250903 20:05:08.737454 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0bf705fb6eb5452385b532032694acfc" } modify_external_catalogs: true
I20250903 20:05:08.743661 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6106b8f9be34419983b266581e83bed8" } modify_external_catalogs: true
I20250903 20:05:08.743882 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6106b8f9be34419983b266581e83bed8" } modify_external_catalogs: true
I20250903 20:05:08.746039 27695 tablet_bootstrap.cc:492] T d08c9e1e7d564886a90f6c056794becc P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:08.747529 27228 tablet_service.cc:1468] Processing CreateTablet for tablet cd3092c89c74475a999f5b3a6d77e652 (DEFAULT_TABLE table=default.Table_99cc3106e9a94dfc85c7a9e145fc442b [id=0523254ee07b4c5695b05a486945cb66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.748021 27228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd3092c89c74475a999f5b3a6d77e652. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.750372 27695 tablet_bootstrap.cc:654] T d08c9e1e7d564886a90f6c056794becc P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:08.751045 27507 tablet_service.cc:1468] Processing CreateTablet for tablet cd3092c89c74475a999f5b3a6d77e652 (DEFAULT_TABLE table=default.Table_99cc3106e9a94dfc85c7a9e145fc442b [id=0523254ee07b4c5695b05a486945cb66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.751498 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd3092c89c74475a999f5b3a6d77e652. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:08.754304 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cd3092c89c74475a999f5b3a6d77e652 consensus state has no leader
I20250903 20:05:08.756031 27695 tablet_bootstrap.cc:492] T d08c9e1e7d564886a90f6c056794becc P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:08.756207 27695 ts_tablet_manager.cc:1397] T d08c9e1e7d564886a90f6c056794becc P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.003s
I20250903 20:05:08.756798 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 07361048416c43b6bc43d8737906db99 (DEFAULT_TABLE table=default.Table_260dadc90c8e415c8dee534464e23000 [id=9cea308979eb4c7f9deaa02e1c8feee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.757123 27695 raft_consensus.cc:357] T d08c9e1e7d564886a90f6c056794becc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:08.757361 27695 raft_consensus.cc:383] T d08c9e1e7d564886a90f6c056794becc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:08.757480 27695 raft_consensus.cc:738] T d08c9e1e7d564886a90f6c056794becc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:08.757726 27695 consensus_queue.cc:260] T d08c9e1e7d564886a90f6c056794becc P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:08.758656 27695 ts_tablet_manager.cc:1428] T d08c9e1e7d564886a90f6c056794becc P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250903 20:05:08.758781 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07361048416c43b6bc43d8737906db99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.759864 27695 tablet_bootstrap.cc:492] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:08.761584 27695 tablet_bootstrap.cc:654] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:08.768429 27229 tablet_service.cc:1468] Processing CreateTablet for tablet 07361048416c43b6bc43d8737906db99 (DEFAULT_TABLE table=default.Table_260dadc90c8e415c8dee534464e23000 [id=9cea308979eb4c7f9deaa02e1c8feee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.768959 27229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07361048416c43b6bc43d8737906db99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.777511 27364 tablet_service.cc:1468] Processing CreateTablet for tablet cd3092c89c74475a999f5b3a6d77e652 (DEFAULT_TABLE table=default.Table_99cc3106e9a94dfc85c7a9e145fc442b [id=0523254ee07b4c5695b05a486945cb66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.777621 27695 tablet_bootstrap.cc:492] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:08.777803 27695 ts_tablet_manager.cc:1397] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.004s
I20250903 20:05:08.778074 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd3092c89c74475a999f5b3a6d77e652. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.778761 27695 raft_consensus.cc:357] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:08.778991 27695 raft_consensus.cc:383] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:08.779083 27695 raft_consensus.cc:738] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:08.779352 27695 consensus_queue.cc:260] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:08.780062 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 943146f84c4a4adf89c5c2e91f9f04d4 consensus state has no leader
I20250903 20:05:08.781428 27059 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 03f5657ca6e44313b1541bc93eedf621
I20250903 20:05:08.781749 27063 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 390c86536f474ab89e6003b67ffc2d8a
I20250903 20:05:08.782692 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 806d4fa868224194a0cf05e81b0ff377
I20250903 20:05:08.783304 27695 ts_tablet_manager.cc:1428] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250903 20:05:08.783660 27695 tablet_bootstrap.cc:492] T cd3092c89c74475a999f5b3a6d77e652 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:08.783758 27063 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 943146f84c4a4adf89c5c2e91f9f04d4 (table default.Table_fac0948b3046453395be5ca6cf3a70b6 [id=322e6a14fef14f5c8623684144339103]). Expected version 1 got 0
I20250903 20:05:08.785472 27228 tablet_service.cc:1515] Processing DeleteTablet for tablet 806d4fa868224194a0cf05e81b0ff377 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:08.785650 27695 tablet_bootstrap.cc:654] T cd3092c89c74475a999f5b3a6d77e652 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:08.785599 27363 tablet_service.cc:1515] Processing DeleteTablet for tablet 806d4fa868224194a0cf05e81b0ff377 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:08.786412 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 806d4fa868224194a0cf05e81b0ff377 because the tablet was not found. No further retry: Not found: Tablet not found: 806d4fa868224194a0cf05e81b0ff377
I20250903 20:05:08.788384 27684 raft_consensus.cc:2241] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:08.788911 27684 pending_rounds.cc:70] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f: Trying to abort 1 pending ops.
I20250903 20:05:08.788982 27684 pending_rounds.cc:77] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384906000658432 op_type: NO_OP noop_request { }
I20250903 20:05:08.789141 27684 raft_consensus.cc:2887] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:08.789258 27684 raft_consensus.cc:2270] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:08.795431 27684 ts_tablet_manager.cc:1905] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:08.800261 27057 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 161f2c2afc2f4c82b2025f7daf460bd6 (table default.Table_1a5c8c3256c24c7f8ef8597efa53dde5 [id=7caef592f80a4d6cb6df4e2c3971de49]). Expected version 1 got 0
I20250903 20:05:08.807549 27646 raft_consensus.cc:491] T 0bcfca89bc3b44d78bc1b875c796329b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:08.807783 27646 raft_consensus.cc:513] T 0bcfca89bc3b44d78bc1b875c796329b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:08.808710 27646 leader_election.cc:290] T 0bcfca89bc3b44d78bc1b875c796329b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:08.812050 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0bcfca89bc3b44d78bc1b875c796329b" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:08.813030 27183 leader_election.cc:343] T 0bcfca89bc3b44d78bc1b875c796329b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:08.809731 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0bcfca89bc3b44d78bc1b875c796329b" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:08.814450 27184 leader_election.cc:343] T 0bcfca89bc3b44d78bc1b875c796329b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:08.814633 27184 leader_election.cc:304] T 0bcfca89bc3b44d78bc1b875c796329b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:08.815516 27646 raft_consensus.cc:2747] T 0bcfca89bc3b44d78bc1b875c796329b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:08.819892 27057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_fac0948b3046453395be5ca6cf3a70b6" } new_table_name: "default.Table_dbf9fa7169a64141bbe6afa3f74254a8" modify_external_catalogs: true
I20250903 20:05:08.821563 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_6d1b2694cd1a4963ac658f25026e9ef5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:08.822036 27684 ts_tablet_manager.cc:1918] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:08.822249 27684 log.cc:1199] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/9d37a5d51a184c76bb8496679dfe3991
W20250903 20:05:08.822353 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d1b2694cd1a4963ac658f25026e9ef5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:08.822732 27684 ts_tablet_manager.cc:1939] T 9d37a5d51a184c76bb8496679dfe3991 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:08.823732 27646 raft_consensus.cc:491] T d08c9e1e7d564886a90f6c056794becc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:08.823941 27646 raft_consensus.cc:513] T d08c9e1e7d564886a90f6c056794becc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:08.824169 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet ea10476553384e8fb9cf75fd1c6595da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:08.824803 27646 leader_election.cc:290] T d08c9e1e7d564886a90f6c056794becc P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:08.825842 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d08c9e1e7d564886a90f6c056794becc" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:08.826148 27184 leader_election.cc:343] T d08c9e1e7d564886a90f6c056794becc P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: d08c9e1e7d564886a90f6c056794becc
W20250903 20:05:08.826624 27183 leader_election.cc:343] T d08c9e1e7d564886a90f6c056794becc P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:08.826668 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 9d37a5d51a184c76bb8496679dfe3991 (table default.Table_12ae79d84647470fa2032d61a167b96c [id=374b9abdf6a54b38876a96d8ec4a0368]) successfully deleted
I20250903 20:05:08.826771 27183 leader_election.cc:304] T d08c9e1e7d564886a90f6c056794becc P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:08.827137 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 0bcfca89bc3b44d78bc1b875c796329b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bcfca89bc3b44d78bc1b875c796329b already in progress: creating tablet
I20250903 20:05:08.827198 27684 tablet_replica.cc:331] T ea10476553384e8fb9cf75fd1c6595da P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:08.827382 27684 raft_consensus.cc:2241] T ea10476553384e8fb9cf75fd1c6595da P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:08.827584 27684 raft_consensus.cc:2270] T ea10476553384e8fb9cf75fd1c6595da P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:08.831116 27685 ts_tablet_manager.cc:1918] T 6dc3e66c6446439abd3eba441f4c782f P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:08.831315 27685 log.cc:1199] T 6dc3e66c6446439abd3eba441f4c782f P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/6dc3e66c6446439abd3eba441f4c782f
I20250903 20:05:08.831463 27646 raft_consensus.cc:2747] T d08c9e1e7d564886a90f6c056794becc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:08.831940 27685 ts_tablet_manager.cc:1939] T 6dc3e66c6446439abd3eba441f4c782f P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:08.833292 27685 tablet_replica.cc:331] T ea10476553384e8fb9cf75fd1c6595da P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:08.833629 27685 raft_consensus.cc:2241] T ea10476553384e8fb9cf75fd1c6595da P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:08.833884 27685 raft_consensus.cc:2270] T ea10476553384e8fb9cf75fd1c6595da P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:08.834425 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 6dc3e66c6446439abd3eba441f4c782f (table default.Table_514f576d8f594b9b92981f4a2ad81f1b [id=8622aba89ccc48c5ac6a28e038fa9db5]) successfully deleted
I20250903 20:05:08.835981 27363 tablet_service.cc:1468] Processing CreateTablet for tablet 806d4fa868224194a0cf05e81b0ff377 (DEFAULT_TABLE table=default.Table_6106b8f9be34419983b266581e83bed8 [id=de7efd68262248f38a3d8d4a8d6c87db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.836069 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 806d4fa868224194a0cf05e81b0ff377 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:08.836478 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 806d4fa868224194a0cf05e81b0ff377. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.837042 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet b197747f8d134f978a2cd1fe3425a5d5 with delete_type TABLET_DATA_DELETED (replaced by tablet 03f5657ca6e44313b1541bc93eedf621) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:08.837718 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 6dc3e66c6446439abd3eba441f4c782f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:08.843245 27063 master_service.cc:948] ReplaceTablet: received request to replace tablet 6dc3e66c6446439abd3eba441f4c782f from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:08.844533 27692 tablet_bootstrap.cc:492] T 0bcfca89bc3b44d78bc1b875c796329b P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:08.844730 27692 ts_tablet_manager.cc:1397] T 0bcfca89bc3b44d78bc1b875c796329b P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.151s user 0.006s sys 0.008s
I20250903 20:05:08.844924 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c106f52e6b6b458391a12b3043e7eedf" } modify_external_catalogs: true
I20250903 20:05:08.845270 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c106f52e6b6b458391a12b3043e7eedf" } modify_external_catalogs: true
I20250903 20:05:08.845633 27692 raft_consensus.cc:357] T 0bcfca89bc3b44d78bc1b875c796329b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:08.845850 27692 raft_consensus.cc:383] T 0bcfca89bc3b44d78bc1b875c796329b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:08.845933 27692 raft_consensus.cc:738] T 0bcfca89bc3b44d78bc1b875c796329b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:08.846241 27692 consensus_queue.cc:260] T 0bcfca89bc3b44d78bc1b875c796329b P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:08.847947 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 806d4fa868224194a0cf05e81b0ff377 (DEFAULT_TABLE table=default.Table_6106b8f9be34419983b266581e83bed8 [id=de7efd68262248f38a3d8d4a8d6c87db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.848690 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 806d4fa868224194a0cf05e81b0ff377. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.851959 27692 ts_tablet_manager.cc:1428] T 0bcfca89bc3b44d78bc1b875c796329b P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20250903 20:05:08.853314 27692 tablet_bootstrap.cc:492] T 218f51a5391141b8941a2b40980147fd P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:08.855162 27692 tablet_bootstrap.cc:654] T 218f51a5391141b8941a2b40980147fd P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:08.857102 27057 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 943146f84c4a4adf89c5c2e91f9f04d4 consensus state has no leader
I20250903 20:05:08.857093 27693 tablet_bootstrap.cc:492] T b197747f8d134f978a2cd1fe3425a5d5 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:08.843709 27228 tablet_service.cc:1468] Processing CreateTablet for tablet 806d4fa868224194a0cf05e81b0ff377 (DEFAULT_TABLE table=default.Table_6106b8f9be34419983b266581e83bed8 [id=de7efd68262248f38a3d8d4a8d6c87db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.861745 27228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 806d4fa868224194a0cf05e81b0ff377. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.865531 27693 tablet_bootstrap.cc:654] T b197747f8d134f978a2cd1fe3425a5d5 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:08.867034 27693 log.cc:826] T b197747f8d134f978a2cd1fe3425a5d5 P 51578fb7bceb4e119b3ab1e58c4f5528: Log is configured to *not* fsync() on all Append() calls
I20250903 20:05:08.867746 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7d476d0d69e246e28664c7c5b84f3ba2" } modify_external_catalogs: true
I20250903 20:05:08.868196 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7d476d0d69e246e28664c7c5b84f3ba2" } modify_external_catalogs: true
I20250903 20:05:08.868448 27058 master_service.cc:948] ReplaceTablet: received request to replace tablet ea10476553384e8fb9cf75fd1c6595da from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:08.868847 27684 ts_tablet_manager.cc:1905] T ea10476553384e8fb9cf75fd1c6595da P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:08.869225 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_37c2b48693594bc38683d256efb83828"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:08.870005 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37c2b48693594bc38683d256efb83828 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:08.874857 27685 ts_tablet_manager.cc:1905] T ea10476553384e8fb9cf75fd1c6595da P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:08.875185 27058 master_service.cc:948] ReplaceTablet: received request to replace tablet 9d37a5d51a184c76bb8496679dfe3991 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:08.876477 27695 tablet_bootstrap.cc:492] T cd3092c89c74475a999f5b3a6d77e652 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:08.876637 27695 ts_tablet_manager.cc:1397] T cd3092c89c74475a999f5b3a6d77e652 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.093s user 0.018s sys 0.042s
I20250903 20:05:08.877496 27695 raft_consensus.cc:357] T cd3092c89c74475a999f5b3a6d77e652 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:08.877705 27695 raft_consensus.cc:383] T cd3092c89c74475a999f5b3a6d77e652 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:08.877801 27695 raft_consensus.cc:738] T cd3092c89c74475a999f5b3a6d77e652 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:08.880157 27695 consensus_queue.cc:260] T cd3092c89c74475a999f5b3a6d77e652 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:08.881134 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_12ae79d84647470fa2032d61a167b96c" } new_table_name: "default.Table_326de6d0004e4f6eb09b66bf736c6fd4" modify_external_catalogs: true
I20250903 20:05:08.881834 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_c180dbba07434ca797370519dbfd20b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:08.882555 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c180dbba07434ca797370519dbfd20b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:08.885324 27695 ts_tablet_manager.cc:1428] T cd3092c89c74475a999f5b3a6d77e652 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.009s user 0.001s sys 0.002s
I20250903 20:05:08.885645 27695 tablet_bootstrap.cc:492] T 07361048416c43b6bc43d8737906db99 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:08.887573 27695 tablet_bootstrap.cc:654] T 07361048416c43b6bc43d8737906db99 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:08.890965 27685 ts_tablet_manager.cc:1918] T ea10476553384e8fb9cf75fd1c6595da P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:08.891166 27685 log.cc:1199] T ea10476553384e8fb9cf75fd1c6595da P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/ea10476553384e8fb9cf75fd1c6595da
I20250903 20:05:08.891626 27685 ts_tablet_manager.cc:1939] T ea10476553384e8fb9cf75fd1c6595da P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:08.892208 27684 ts_tablet_manager.cc:1918] T ea10476553384e8fb9cf75fd1c6595da P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:08.892439 27684 log.cc:1199] T ea10476553384e8fb9cf75fd1c6595da P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/ea10476553384e8fb9cf75fd1c6595da
I20250903 20:05:08.893034 27684 ts_tablet_manager.cc:1939] T ea10476553384e8fb9cf75fd1c6595da P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:08.894418 27066 master_service.cc:948] ReplaceTablet: received request to replace tablet 0bcfca89bc3b44d78bc1b875c796329b from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:08.894981 27684 tablet_replica.cc:331] T b197747f8d134f978a2cd1fe3425a5d5 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:08.895321 27684 raft_consensus.cc:2241] T b197747f8d134f978a2cd1fe3425a5d5 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:08.895447 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet ea10476553384e8fb9cf75fd1c6595da (table default.Table_59b61e83a1dd4c7386d0ffca89f82104 [id=f378777cae174d209abd3bd7f5766259]) successfully deleted
I20250903 20:05:08.895671 27684 raft_consensus.cc:2270] T b197747f8d134f978a2cd1fe3425a5d5 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:08.895718 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 806d4fa868224194a0cf05e81b0ff377 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 806d4fa868224194a0cf05e81b0ff377 already in progress: creating tablet
I20250903 20:05:08.896756 27684 ts_tablet_manager.cc:1905] T b197747f8d134f978a2cd1fe3425a5d5 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:08.897929 27692 tablet_bootstrap.cc:492] T 218f51a5391141b8941a2b40980147fd P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:08.898085 27692 ts_tablet_manager.cc:1397] T 218f51a5391141b8941a2b40980147fd P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.045s user 0.000s sys 0.003s
I20250903 20:05:08.898972 27692 raft_consensus.cc:357] T 218f51a5391141b8941a2b40980147fd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:08.899171 27692 raft_consensus.cc:383] T 218f51a5391141b8941a2b40980147fd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:08.899292 27692 raft_consensus.cc:738] T 218f51a5391141b8941a2b40980147fd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:08.899520 27692 consensus_queue.cc:260] T 218f51a5391141b8941a2b40980147fd P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:08.899806 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet ea10476553384e8fb9cf75fd1c6595da (table default.Table_59b61e83a1dd4c7386d0ffca89f82104 [id=f378777cae174d209abd3bd7f5766259]) successfully deleted
I20250903 20:05:08.900945 27066 master_service.cc:948] ReplaceTablet: received request to replace tablet b197747f8d134f978a2cd1fe3425a5d5 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:08.901312 27685 tablet_replica.cc:331] T d08c9e1e7d564886a90f6c056794becc P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:08.901631 27685 raft_consensus.cc:2241] T d08c9e1e7d564886a90f6c056794becc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:08.901916 27685 raft_consensus.cc:2270] T d08c9e1e7d564886a90f6c056794becc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:08.904282 27692 ts_tablet_manager.cc:1428] T 218f51a5391141b8941a2b40980147fd P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250903 20:05:08.905838 27692 tablet_bootstrap.cc:492] T 943146f84c4a4adf89c5c2e91f9f04d4 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:08.907620 27693 tablet_bootstrap.cc:492] T b197747f8d134f978a2cd1fe3425a5d5 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:08.908008 27693 ts_tablet_manager.cc:1397] T b197747f8d134f978a2cd1fe3425a5d5 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.051s user 0.010s sys 0.000s
W20250903 20:05:08.910953 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fd499b81344c4b58bf77f3622422763c has no consensus state
I20250903 20:05:08.912384 27692 tablet_bootstrap.cc:654] T 943146f84c4a4adf89c5c2e91f9f04d4 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:08.914595 27062 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet cd3092c89c74475a999f5b3a6d77e652 (table default.Table_60698087f8f34bbe86a9994a117f2b37 [id=0523254ee07b4c5695b05a486945cb66]). Expected version 1 got 0
I20250903 20:05:08.915501 27507 tablet_service.cc:1468] Processing CreateTablet for tablet c8a293c6fa034e9fb29682439cb3bcc2 (DEFAULT_TABLE table=default.Table_d6ca992193f94f4aa3fe1ec509c04ac5 [id=5cfc8827e4ca4f4a94424c48d6f71eb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.915689 27506 tablet_service.cc:1468] Processing CreateTablet for tablet e0d1b29d7f1e422e8bb5b8bbc33423e8 (DEFAULT_TABLE table=default.Table_82f29a4a80194976bc7c8f864ec877a3 [id=49551de884e7488fb6db7249f834e12f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.915897 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8a293c6fa034e9fb29682439cb3bcc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.916074 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0d1b29d7f1e422e8bb5b8bbc33423e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.918632 27366 tablet_service.cc:1468] Processing CreateTablet for tablet 3906285457f449e6824d1f044bffca24 (DEFAULT_TABLE table=default.Table_1ba72813b9b347e1b6ef05748418aa84 [id=b35d1fca6bcd4bb6a1581c2c81cac712]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.919064 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3906285457f449e6824d1f044bffca24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.919212 27364 tablet_service.cc:1468] Processing CreateTablet for tablet e0d1b29d7f1e422e8bb5b8bbc33423e8 (DEFAULT_TABLE table=default.Table_82f29a4a80194976bc7c8f864ec877a3 [id=49551de884e7488fb6db7249f834e12f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.919687 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0d1b29d7f1e422e8bb5b8bbc33423e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.919060 27362 tablet_service.cc:1468] Processing CreateTablet for tablet fd499b81344c4b58bf77f3622422763c (DEFAULT_TABLE table=default.Table_326de6d0004e4f6eb09b66bf736c6fd4 [id=374b9abdf6a54b38876a96d8ec4a0368]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.922574 27365 tablet_service.cc:1468] Processing CreateTablet for tablet b6b4713936134681aaef1732d46c6ab2 (DEFAULT_TABLE table=default.Table_db311e4928f54b0d8fce9e417635b974 [id=d0a75545b91c48bba47fb581bb0d1a96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.922756 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd499b81344c4b58bf77f3622422763c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.923033 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6b4713936134681aaef1732d46c6ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.927244 26885 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskH7eH49/build/asan/bin/kudu
/tmp/dist-test-taskH7eH49/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.65.126:39015
--webserver_interface=127.26.65.126
--webserver_port=33509
--builtin_ntp_servers=127.26.65.84:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.65.126:39015,127.26.65.125:44059,127.26.65.124:40911
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250903 20:05:08.930135 27507 tablet_service.cc:1468] Processing CreateTablet for tablet b6b4713936134681aaef1732d46c6ab2 (DEFAULT_TABLE table=default.Table_db311e4928f54b0d8fce9e417635b974 [id=d0a75545b91c48bba47fb581bb0d1a96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.930368 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 3906285457f449e6824d1f044bffca24 (DEFAULT_TABLE table=default.Table_1ba72813b9b347e1b6ef05748418aa84 [id=b35d1fca6bcd4bb6a1581c2c81cac712]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.930570 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6b4713936134681aaef1732d46c6ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.930761 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3906285457f449e6824d1f044bffca24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.931087 27229 tablet_service.cc:1468] Processing CreateTablet for tablet c8a293c6fa034e9fb29682439cb3bcc2 (DEFAULT_TABLE table=default.Table_d6ca992193f94f4aa3fe1ec509c04ac5 [id=5cfc8827e4ca4f4a94424c48d6f71eb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.931727 27228 tablet_service.cc:1468] Processing CreateTablet for tablet e0d1b29d7f1e422e8bb5b8bbc33423e8 (DEFAULT_TABLE table=default.Table_82f29a4a80194976bc7c8f864ec877a3 [id=49551de884e7488fb6db7249f834e12f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.931752 27363 tablet_service.cc:1468] Processing CreateTablet for tablet c8a293c6fa034e9fb29682439cb3bcc2 (DEFAULT_TABLE table=default.Table_d6ca992193f94f4aa3fe1ec509c04ac5 [id=5cfc8827e4ca4f4a94424c48d6f71eb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.932231 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8a293c6fa034e9fb29682439cb3bcc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.932204 27224 tablet_service.cc:1468] Processing CreateTablet for tablet 3906285457f449e6824d1f044bffca24 (DEFAULT_TABLE table=default.Table_1ba72813b9b347e1b6ef05748418aa84 [id=b35d1fca6bcd4bb6a1581c2c81cac712]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.932518 27229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8a293c6fa034e9fb29682439cb3bcc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.937867 27062 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet cd3092c89c74475a999f5b3a6d77e652 (table default.Table_60698087f8f34bbe86a9994a117f2b37 [id=0523254ee07b4c5695b05a486945cb66]). Expected version 1 got 0
I20250903 20:05:08.938297 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3906285457f449e6824d1f044bffca24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.938920 27646 raft_consensus.cc:491] T 943146f84c4a4adf89c5c2e91f9f04d4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:08.939085 27646 raft_consensus.cc:513] T 943146f84c4a4adf89c5c2e91f9f04d4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:08.939251 27695 tablet_bootstrap.cc:492] T 07361048416c43b6bc43d8737906db99 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:08.942561 27366 tablet_service.cc:1468] Processing CreateTablet for tablet 43f88abc8f3640a48598c07c3d76755d (DEFAULT_TABLE table=default.Table_3ca111aa05124c5b91822761043b2d94 [id=71d850a0cea14b0ba337a822ad99d513]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.943050 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43f88abc8f3640a48598c07c3d76755d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.944967 27362 tablet_service.cc:1468] Processing CreateTablet for tablet 0a266b1bbf1540ac86d49c56df2dd4ad (DEFAULT_TABLE table=default.Table_a7abfdf9caa14a85b12e07546a8518ff [id=75e5cb0bab9b46a6b5caec7c99efc7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.945961 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a266b1bbf1540ac86d49c56df2dd4ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.947175 27361 tablet_service.cc:1468] Processing CreateTablet for tablet d1e1add08013459ba0dab3b7e66f0136 (DEFAULT_TABLE table=default.Table_4ace9616711f4778a94a0827ff79cc07 [id=e20fa37b0e0343ed8ab3a7e814358ae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.947633 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1e1add08013459ba0dab3b7e66f0136. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.948014 27693 raft_consensus.cc:357] T b197747f8d134f978a2cd1fe3425a5d5 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:08.948419 27693 raft_consensus.cc:383] T b197747f8d134f978a2cd1fe3425a5d5 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:08.948532 27693 raft_consensus.cc:738] T b197747f8d134f978a2cd1fe3425a5d5 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:08.950680 27360 tablet_service.cc:1468] Processing CreateTablet for tablet c12c957a36f643d9bfb60b755619cbbc (DEFAULT_TABLE table=default.Table_7d476d0d69e246e28664c7c5b84f3ba2 [id=7c51b3299c2948189100608937aa1ef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.951203 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c12c957a36f643d9bfb60b755619cbbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.951922 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_92a25ef74d6648d0a0052fe858b10b77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:08.952729 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_92a25ef74d6648d0a0052fe858b10b77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:08.953271 27693 consensus_queue.cc:260] T b197747f8d134f978a2cd1fe3425a5d5 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:08.955248 27507 tablet_service.cc:1468] Processing CreateTablet for tablet fd499b81344c4b58bf77f3622422763c (DEFAULT_TABLE table=default.Table_326de6d0004e4f6eb09b66bf736c6fd4 [id=374b9abdf6a54b38876a96d8ec4a0368]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.955382 27359 tablet_service.cc:1468] Processing CreateTablet for tablet 1d9a2508437b41f08821a29f58163159 (DEFAULT_TABLE table=default.Table_c106f52e6b6b458391a12b3043e7eedf [id=cbae62c266f24526affd029472f41b94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.955726 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd499b81344c4b58bf77f3622422763c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.955878 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d9a2508437b41f08821a29f58163159. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.956010 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 43f88abc8f3640a48598c07c3d76755d (DEFAULT_TABLE table=default.Table_3ca111aa05124c5b91822761043b2d94 [id=71d850a0cea14b0ba337a822ad99d513]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.956367 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43f88abc8f3640a48598c07c3d76755d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.957029 27503 tablet_service.cc:1468] Processing CreateTablet for tablet c12c957a36f643d9bfb60b755619cbbc (DEFAULT_TABLE table=default.Table_7d476d0d69e246e28664c7c5b84f3ba2 [id=7c51b3299c2948189100608937aa1ef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.957434 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c12c957a36f643d9bfb60b755619cbbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.958592 27358 tablet_service.cc:1468] Processing CreateTablet for tablet f99c05e72b9a4ff9a1e7c269ca029b6a (DEFAULT_TABLE table=default.Table_3b8908bfafc443798402fc7b449f8c4c [id=0b57a09164704002b7fcd0b8f08da1e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.959033 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f99c05e72b9a4ff9a1e7c269ca029b6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.962539 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 0a266b1bbf1540ac86d49c56df2dd4ad (DEFAULT_TABLE table=default.Table_a7abfdf9caa14a85b12e07546a8518ff [id=75e5cb0bab9b46a6b5caec7c99efc7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.962901 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a266b1bbf1540ac86d49c56df2dd4ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.965154 27505 tablet_service.cc:1468] Processing CreateTablet for tablet f99c05e72b9a4ff9a1e7c269ca029b6a (DEFAULT_TABLE table=default.Table_3b8908bfafc443798402fc7b449f8c4c [id=0b57a09164704002b7fcd0b8f08da1e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.965518 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f99c05e72b9a4ff9a1e7c269ca029b6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.966475 27502 tablet_service.cc:1468] Processing CreateTablet for tablet d1e1add08013459ba0dab3b7e66f0136 (DEFAULT_TABLE table=default.Table_4ace9616711f4778a94a0827ff79cc07 [id=e20fa37b0e0343ed8ab3a7e814358ae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.966930 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1e1add08013459ba0dab3b7e66f0136. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.970463 27503 tablet_service.cc:1468] Processing CreateTablet for tablet 1d9a2508437b41f08821a29f58163159 (DEFAULT_TABLE table=default.Table_c106f52e6b6b458391a12b3043e7eedf [id=cbae62c266f24526affd029472f41b94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.970856 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d9a2508437b41f08821a29f58163159. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.970814 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_a249558ab60e466ca7734ffeb24ac5fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:08.971423 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a249558ab60e466ca7734ffeb24ac5fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:08.974920 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "943146f84c4a4adf89c5c2e91f9f04d4" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:08.939644 27227 tablet_service.cc:1468] Processing CreateTablet for tablet b6b4713936134681aaef1732d46c6ab2 (DEFAULT_TABLE table=default.Table_db311e4928f54b0d8fce9e417635b974 [id=d0a75545b91c48bba47fb581bb0d1a96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:08.976198 27228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0d1b29d7f1e422e8bb5b8bbc33423e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:08.980067 27684 ts_tablet_manager.cc:1918] T b197747f8d134f978a2cd1fe3425a5d5 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:08.994046 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1ba72813b9b347e1b6ef05748418aa84" } new_table_name: "default.Table_264581b9b25640c58ed0dd863bf6bb8e" modify_external_catalogs: true
I20250903 20:05:08.998051 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_26d34ebce42f44858e5961a8baaff5a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:08.998880 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26d34ebce42f44858e5961a8baaff5a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:09.004357 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_db7d825a12ef40739b70f11044170c72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:09.004998 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db7d825a12ef40739b70f11044170c72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:09.008950 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d6ca992193f94f4aa3fe1ec509c04ac5" } new_table_name: "default.Table_0d37ecfeb62240568297039867aa4c20" modify_external_catalogs: true
I20250903 20:05:09.010205 27685 ts_tablet_manager.cc:1905] T d08c9e1e7d564886a90f6c056794becc P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:09.010385 27431 heartbeater.cc:499] Master 127.26.65.124:40911 was elected leader, sending a full tablet report...
I20250903 20:05:09.011839 27693 ts_tablet_manager.cc:1428] T b197747f8d134f978a2cd1fe3425a5d5 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.104s user 0.015s sys 0.003s
I20250903 20:05:09.012135 27646 leader_election.cc:290] T 943146f84c4a4adf89c5c2e91f9f04d4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
W20250903 20:05:09.013401 27183 leader_election.cc:343] T 943146f84c4a4adf89c5c2e91f9f04d4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:09.013634 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 43f88abc8f3640a48598c07c3d76755d (DEFAULT_TABLE table=default.Table_3ca111aa05124c5b91822761043b2d94 [id=71d850a0cea14b0ba337a822ad99d513]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.014412 27693 tablet_bootstrap.cc:492] T 6dc3e66c6446439abd3eba441f4c782f P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:09.014634 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6b4713936134681aaef1732d46c6ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.016244 27693 tablet_bootstrap.cc:654] T 6dc3e66c6446439abd3eba441f4c782f P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:09.016366 27695 ts_tablet_manager.cc:1397] T 07361048416c43b6bc43d8737906db99 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.054s user 0.019s sys 0.032s
I20250903 20:05:09.018913 27226 tablet_service.cc:1468] Processing CreateTablet for tablet fd499b81344c4b58bf77f3622422763c (DEFAULT_TABLE table=default.Table_326de6d0004e4f6eb09b66bf736c6fd4 [id=374b9abdf6a54b38876a96d8ec4a0368]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.020556 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_857612f25ca54f8b8d0ea31485d96a5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:09.020793 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43f88abc8f3640a48598c07c3d76755d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:09.021188 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_857612f25ca54f8b8d0ea31485d96a5d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:09.023145 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "943146f84c4a4adf89c5c2e91f9f04d4" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:09.024212 27693 tablet_bootstrap.cc:492] T 6dc3e66c6446439abd3eba441f4c782f P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:09.024410 27693 ts_tablet_manager.cc:1397] T 6dc3e66c6446439abd3eba441f4c782f P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20250903 20:05:09.020540 27222 tablet_service.cc:1468] Processing CreateTablet for tablet d1e1add08013459ba0dab3b7e66f0136 (DEFAULT_TABLE table=default.Table_4ace9616711f4778a94a0827ff79cc07 [id=e20fa37b0e0343ed8ab3a7e814358ae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.025898 27684 log.cc:1199] T b197747f8d134f978a2cd1fe3425a5d5 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/b197747f8d134f978a2cd1fe3425a5d5
I20250903 20:05:09.018500 27223 tablet_service.cc:1468] Processing CreateTablet for tablet 0a266b1bbf1540ac86d49c56df2dd4ad (DEFAULT_TABLE table=default.Table_a7abfdf9caa14a85b12e07546a8518ff [id=75e5cb0bab9b46a6b5caec7c99efc7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.026981 27646 raft_consensus.cc:491] T cd3092c89c74475a999f5b3a6d77e652 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:09.027308 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd499b81344c4b58bf77f3622422763c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.027464 27693 raft_consensus.cc:357] T 6dc3e66c6446439abd3eba441f4c782f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:09.027647 27693 raft_consensus.cc:383] T 6dc3e66c6446439abd3eba441f4c782f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:09.027778 27693 raft_consensus.cc:738] T 6dc3e66c6446439abd3eba441f4c782f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:09.028225 27693 consensus_queue.cc:260] T 6dc3e66c6446439abd3eba441f4c782f P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:09.028734 27693 ts_tablet_manager.cc:1428] T 6dc3e66c6446439abd3eba441f4c782f P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:09.028713 27672 raft_consensus.cc:491] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:09.028893 27672 raft_consensus.cc:513] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:09.031210 27184 leader_election.cc:343] T 943146f84c4a4adf89c5c2e91f9f04d4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:09.031405 27184 leader_election.cc:304] T 943146f84c4a4adf89c5c2e91f9f04d4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:09.032410 27220 tablet_service.cc:1468] Processing CreateTablet for tablet f99c05e72b9a4ff9a1e7c269ca029b6a (DEFAULT_TABLE table=default.Table_3b8908bfafc443798402fc7b449f8c4c [id=0b57a09164704002b7fcd0b8f08da1e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.033280 27229 tablet_service.cc:1468] Processing CreateTablet for tablet 1d9a2508437b41f08821a29f58163159 (DEFAULT_TABLE table=default.Table_c106f52e6b6b458391a12b3043e7eedf [id=cbae62c266f24526affd029472f41b94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.031920 27221 tablet_service.cc:1468] Processing CreateTablet for tablet c12c957a36f643d9bfb60b755619cbbc (DEFAULT_TABLE table=default.Table_7d476d0d69e246e28664c7c5b84f3ba2 [id=7c51b3299c2948189100608937aa1ef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.034016 27693 tablet_bootstrap.cc:492] T 0bcfca89bc3b44d78bc1b875c796329b P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:09.034279 27646 raft_consensus.cc:513] T cd3092c89c74475a999f5b3a6d77e652 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.035087 27646 leader_election.cc:290] T cd3092c89c74475a999f5b3a6d77e652 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:09.036556 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 1d9a2508437b41f08821a29f58163159
I20250903 20:05:09.037364 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_2dd5b5956b44471da9d0b89e782d469e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:09.037468 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_326de6d0004e4f6eb09b66bf736c6fd4" } new_table_name: "default.Table_e914faf335d44d019f91bda71758c473" modify_external_catalogs: true
I20250903 20:05:09.037647 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet 1d9a2508437b41f08821a29f58163159 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:09.037982 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2dd5b5956b44471da9d0b89e782d469e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:09.038326 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 1d9a2508437b41f08821a29f58163159 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d9a2508437b41f08821a29f58163159 already in progress: creating tablet
I20250903 20:05:09.039054 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1e1add08013459ba0dab3b7e66f0136. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.040709 27693 tablet_bootstrap.cc:654] T 0bcfca89bc3b44d78bc1b875c796329b P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:09.046236 27695 raft_consensus.cc:357] T 07361048416c43b6bc43d8737906db99 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.048310 27646 raft_consensus.cc:2747] T 943146f84c4a4adf89c5c2e91f9f04d4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:09.051903 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_5f1e055d858e4d31bce98ac57741cc30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:09.052102 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_82f29a4a80194976bc7c8f864ec877a3" } modify_external_catalogs: true
I20250903 20:05:09.052282 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_82f29a4a80194976bc7c8f864ec877a3" } modify_external_catalogs: true
I20250903 20:05:09.052588 27716 raft_consensus.cc:491] T 6dc3e66c6446439abd3eba441f4c782f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250903 20:05:09.052719 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f1e055d858e4d31bce98ac57741cc30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:09.052830 27716 raft_consensus.cc:513] T 6dc3e66c6446439abd3eba441f4c782f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:09.054132 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_37c2b48693594bc38683d256efb83828" } modify_external_catalogs: true
I20250903 20:05:09.054415 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_37c2b48693594bc38683d256efb83828" } modify_external_catalogs: true
I20250903 20:05:09.055254 27719 raft_consensus.cc:491] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:09.056087 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_60698087f8f34bbe86a9994a117f2b37" } modify_external_catalogs: true
I20250903 20:05:09.056254 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_60698087f8f34bbe86a9994a117f2b37" } modify_external_catalogs: true
I20250903 20:05:09.058712 27229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d9a2508437b41f08821a29f58163159. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.055425 27719 raft_consensus.cc:513] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.063364 27716 leader_election.cc:290] T 6dc3e66c6446439abd3eba441f4c782f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:09.064137 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd3092c89c74475a999f5b3a6d77e652" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:09.064237 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b197747f8d134f978a2cd1fe3425a5d5" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:09.064502 27385 raft_consensus.cc:2466] T b197747f8d134f978a2cd1fe3425a5d5 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:09.068997 27223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a266b1bbf1540ac86d49c56df2dd4ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.078775 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a7abfdf9caa14a85b12e07546a8518ff" } new_table_name: "default.Table_5de0a37e50d44850a1486650571595d4" modify_external_catalogs: true
I20250903 20:05:09.079888 27693 tablet_bootstrap.cc:492] T 0bcfca89bc3b44d78bc1b875c796329b P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:09.080207 27693 ts_tablet_manager.cc:1397] T 0bcfca89bc3b44d78bc1b875c796329b P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.001s
I20250903 20:05:09.081624 27693 raft_consensus.cc:357] T 0bcfca89bc3b44d78bc1b875c796329b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:09.081888 27693 raft_consensus.cc:383] T 0bcfca89bc3b44d78bc1b875c796329b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:09.082036 27693 raft_consensus.cc:738] T 0bcfca89bc3b44d78bc1b875c796329b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:09.082655 27693 consensus_queue.cc:260] T 0bcfca89bc3b44d78bc1b875c796329b P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:09.087178 27693 ts_tablet_manager.cc:1428] T 0bcfca89bc3b44d78bc1b875c796329b P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250903 20:05:09.087707 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_a1c7e65e2fd446f6a895e8e6cfc2e27d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:09.093937 27434 tablet.cc:2378] T 0bcfca89bc3b44d78bc1b875c796329b P 51578fb7bceb4e119b3ab1e58c4f5528: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250903 20:05:09.095352 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1c7e65e2fd446f6a895e8e6cfc2e27d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:09.098121 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b197747f8d134f978a2cd1fe3425a5d5" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:09.098709 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "218f51a5391141b8941a2b40980147fd" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:09.099073 27525 raft_consensus.cc:2466] T 218f51a5391141b8941a2b40980147fd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:09.100847 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd3092c89c74475a999f5b3a6d77e652" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:09.103056 27693 tablet_bootstrap.cc:492] T 218f51a5391141b8941a2b40980147fd P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:09.105562 27693 tablet_bootstrap.cc:654] T 218f51a5391141b8941a2b40980147fd P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:09.106385 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 1d9a2508437b41f08821a29f58163159 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:09.109903 27684 ts_tablet_manager.cc:1939] T b197747f8d134f978a2cd1fe3425a5d5 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:09.109869 27503 tablet_service.cc:1515] Processing DeleteTablet for tablet 0bcfca89bc3b44d78bc1b875c796329b with delete_type TABLET_DATA_DELETED (replaced by tablet 1d9a2508437b41f08821a29f58163159) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:09.111649 27684 tablet_replica.cc:331] T 6dc3e66c6446439abd3eba441f4c782f P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:09.111896 27684 raft_consensus.cc:2241] T 6dc3e66c6446439abd3eba441f4c782f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:09.112406 27061 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet d898d94fa19443d995ba9c3e198e378c
I20250903 20:05:09.112843 27684 raft_consensus.cc:2270] T 6dc3e66c6446439abd3eba441f4c782f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:09.113504 27684 ts_tablet_manager.cc:1905] T 6dc3e66c6446439abd3eba441f4c782f P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:09.114686 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_f9e41274ca534e46b7906480d7fc32fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:09.118649 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9e41274ca534e46b7906480d7fc32fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:09.121649 27505 tablet_service.cc:1515] Processing DeleteTablet for tablet 218f51a5391141b8941a2b40980147fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:09.130681 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c12c957a36f643d9bfb60b755619cbbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.132431 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "218f51a5391141b8941a2b40980147fd" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:09.133483 27684 ts_tablet_manager.cc:1918] T 6dc3e66c6446439abd3eba441f4c782f P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:09.133634 27684 log.cc:1199] T 6dc3e66c6446439abd3eba441f4c782f P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/6dc3e66c6446439abd3eba441f4c782f
I20250903 20:05:09.133940 27672 leader_election.cc:290] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:09.134052 27684 ts_tablet_manager.cc:1939] T 6dc3e66c6446439abd3eba441f4c782f P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
W20250903 20:05:09.134438 27183 leader_election.cc:343] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: Tablet not RUNNING: STOPPED
I20250903 20:05:09.135301 27684 tablet_replica.cc:331] T 0bcfca89bc3b44d78bc1b875c796329b P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:09.135521 27684 raft_consensus.cc:2241] T 0bcfca89bc3b44d78bc1b875c796329b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:09.074765 27719 leader_election.cc:290] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:09.135716 27684 raft_consensus.cc:2270] T 0bcfca89bc3b44d78bc1b875c796329b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:09.136875 27183 leader_election.cc:304] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:09.138051 27695 raft_consensus.cc:383] T 07361048416c43b6bc43d8737906db99 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:09.138698 27692 tablet_bootstrap.cc:492] T 943146f84c4a4adf89c5c2e91f9f04d4 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:09.138887 27692 ts_tablet_manager.cc:1397] T 943146f84c4a4adf89c5c2e91f9f04d4 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.233s user 0.066s sys 0.095s
I20250903 20:05:09.139183 27219 tablet_service.cc:1515] Processing DeleteTablet for tablet 1d9a2508437b41f08821a29f58163159 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:09.139798 27692 raft_consensus.cc:357] T 943146f84c4a4adf89c5c2e91f9f04d4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.140003 27692 raft_consensus.cc:383] T 943146f84c4a4adf89c5c2e91f9f04d4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:09.140079 27692 raft_consensus.cc:738] T 943146f84c4a4adf89c5c2e91f9f04d4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:09.139902 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet b197747f8d134f978a2cd1fe3425a5d5 (table default.Table_f7e6051d454149d685213bea3186fb79 [id=88e0361a865a4b28a08e6c7f30dee9e3]) successfully deleted
I20250903 20:05:09.140332 27692 consensus_queue.cc:260] T 943146f84c4a4adf89c5c2e91f9f04d4 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.140630 27719 raft_consensus.cc:2802] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:09.140848 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 6dc3e66c6446439abd3eba441f4c782f (table default.Table_514f576d8f594b9b92981f4a2ad81f1b [id=8622aba89ccc48c5ac6a28e038fa9db5]) successfully deleted
W20250903 20:05:09.141266 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 1d9a2508437b41f08821a29f58163159 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d9a2508437b41f08821a29f58163159 already in progress: creating tablet
I20250903 20:05:09.142266 27692 ts_tablet_manager.cc:1428] T 943146f84c4a4adf89c5c2e91f9f04d4 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20250903 20:05:09.143154 27692 tablet_bootstrap.cc:492] T d08c9e1e7d564886a90f6c056794becc P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:09.144182 27684 ts_tablet_manager.cc:1905] T 0bcfca89bc3b44d78bc1b875c796329b P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:09.145395 27695 raft_consensus.cc:738] T 07361048416c43b6bc43d8737906db99 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:09.146670 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f99c05e72b9a4ff9a1e7c269ca029b6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.147441 27719 raft_consensus.cc:491] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:09.147600 27719 raft_consensus.cc:3058] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:09.148416 27692 tablet_bootstrap.cc:654] T d08c9e1e7d564886a90f6c056794becc P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:09.148492 27320 leader_election.cc:343] T 6dc3e66c6446439abd3eba441f4c782f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 6dc3e66c6446439abd3eba441f4c782f
W20250903 20:05:09.149329 27184 leader_election.cc:343] T cd3092c89c74475a999f5b3a6d77e652 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:09.150264 27183 leader_election.cc:343] T cd3092c89c74475a999f5b3a6d77e652 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:09.152155 27183 leader_election.cc:304] T cd3092c89c74475a999f5b3a6d77e652 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:09.156123 27684 ts_tablet_manager.cc:1918] T 0bcfca89bc3b44d78bc1b875c796329b P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:09.156368 27684 log.cc:1199] T 0bcfca89bc3b44d78bc1b875c796329b P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/0bcfca89bc3b44d78bc1b875c796329b
I20250903 20:05:09.151288 27695 consensus_queue.cc:260] T 07361048416c43b6bc43d8737906db99 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.156931 27684 ts_tablet_manager.cc:1939] T 0bcfca89bc3b44d78bc1b875c796329b P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:09.157792 27672 raft_consensus.cc:2747] T cd3092c89c74475a999f5b3a6d77e652 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:09.158031 27684 tablet_replica.cc:331] T 218f51a5391141b8941a2b40980147fd P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:09.158303 27684 raft_consensus.cc:2241] T 218f51a5391141b8941a2b40980147fd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:09.158555 27684 raft_consensus.cc:2270] T 218f51a5391141b8941a2b40980147fd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:09.159507 27684 ts_tablet_manager.cc:1905] T 218f51a5391141b8941a2b40980147fd P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:09.164266 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0a266b1bbf1540ac86d49c56df2dd4ad consensus state has no leader
I20250903 20:05:09.165808 27684 ts_tablet_manager.cc:1918] T 218f51a5391141b8941a2b40980147fd P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:09.166052 27684 log.cc:1199] T 218f51a5391141b8941a2b40980147fd P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/218f51a5391141b8941a2b40980147fd
I20250903 20:05:09.166556 27684 ts_tablet_manager.cc:1939] T 218f51a5391141b8941a2b40980147fd P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:09.168645 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3ca111aa05124c5b91822761043b2d94" } new_table_name: "default.Table_3e3261a8bf49425b8203ec49357abd67" modify_external_catalogs: true
I20250903 20:05:09.170444 27364 tablet_service.cc:1468] Processing CreateTablet for tablet 0062307c6ee54aada74a08cf0d087f17 (DEFAULT_TABLE table=default.Table_6d1b2694cd1a4963ac658f25026e9ef5 [id=f4bc244b05e840b9843b4d448d4f28aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.170977 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0062307c6ee54aada74a08cf0d087f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.171774 27184 leader_election.cc:304] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:09.176257 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_2facdec2620546d5bf3dcdbaaf76a56e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:09.176985 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2facdec2620546d5bf3dcdbaaf76a56e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:09.178113 27672 raft_consensus.cc:2802] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:09.178907 27362 tablet_service.cc:1468] Processing CreateTablet for tablet 12a7be47c9c2406596c9e00800074a7c (DEFAULT_TABLE table=default.Table_c180dbba07434ca797370519dbfd20b7 [id=fdc7899c8b334d309b834ae1daa14fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.179462 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12a7be47c9c2406596c9e00800074a7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.179495 27672 raft_consensus.cc:491] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:09.180948 27672 raft_consensus.cc:3058] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:09.182893 27184 leader_election.cc:343] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:09.185668 27219 tablet_service.cc:1468] Processing CreateTablet for tablet 12a7be47c9c2406596c9e00800074a7c (DEFAULT_TABLE table=default.Table_c180dbba07434ca797370519dbfd20b7 [id=fdc7899c8b334d309b834ae1daa14fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.186985 27695 ts_tablet_manager.cc:1428] T 07361048416c43b6bc43d8737906db99 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.160s user 0.000s sys 0.003s
I20250903 20:05:09.196482 27693 tablet_bootstrap.cc:492] T 218f51a5391141b8941a2b40980147fd P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:09.201530 27693 ts_tablet_manager.cc:1397] T 218f51a5391141b8941a2b40980147fd P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.099s user 0.005s sys 0.000s
I20250903 20:05:09.202440 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 0bcfca89bc3b44d78bc1b875c796329b (table default.Table_c106f52e6b6b458391a12b3043e7eedf [id=cbae62c266f24526affd029472f41b94]) successfully deleted
I20250903 20:05:09.203064 27693 raft_consensus.cc:357] T 218f51a5391141b8941a2b40980147fd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.203367 27693 raft_consensus.cc:383] T 218f51a5391141b8941a2b40980147fd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:09.203531 27693 raft_consensus.cc:738] T 218f51a5391141b8941a2b40980147fd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:09.203778 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 218f51a5391141b8941a2b40980147fd (table default.Table_d0d3dd3e485546579b73419a892afba3 [id=86d4245a96f94a789a163b51ae32906e]) successfully deleted
I20250903 20:05:09.203841 27693 consensus_queue.cc:260] T 218f51a5391141b8941a2b40980147fd P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.205101 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 0062307c6ee54aada74a08cf0d087f17 (DEFAULT_TABLE table=default.Table_6d1b2694cd1a4963ac658f25026e9ef5 [id=f4bc244b05e840b9843b4d448d4f28aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.205518 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0062307c6ee54aada74a08cf0d087f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.205956 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 12a7be47c9c2406596c9e00800074a7c (DEFAULT_TABLE table=default.Table_c180dbba07434ca797370519dbfd20b7 [id=fdc7899c8b334d309b834ae1daa14fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.206295 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12a7be47c9c2406596c9e00800074a7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.212097 27219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12a7be47c9c2406596c9e00800074a7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.213179 27695 tablet_bootstrap.cc:492] T 806d4fa868224194a0cf05e81b0ff377 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:09.205417 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 0062307c6ee54aada74a08cf0d087f17 (DEFAULT_TABLE table=default.Table_6d1b2694cd1a4963ac658f25026e9ef5 [id=f4bc244b05e840b9843b4d448d4f28aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.214686 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0062307c6ee54aada74a08cf0d087f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.219511 27064 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 07361048416c43b6bc43d8737906db99 (table default.Table_260dadc90c8e415c8dee534464e23000 [id=9cea308979eb4c7f9deaa02e1c8feee9]). Expected version 1 got 0
I20250903 20:05:09.219995 27064 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet fd499b81344c4b58bf77f3622422763c (table default.Table_326de6d0004e4f6eb09b66bf736c6fd4 [id=374b9abdf6a54b38876a96d8ec4a0368]). Expected version 3 got 0
I20250903 20:05:09.220729 27064 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 161f2c2afc2f4c82b2025f7daf460bd6 (table default.Table_1a5c8c3256c24c7f8ef8597efa53dde5 [id=7caef592f80a4d6cb6df4e2c3971de49]). Expected version 1 got 0
I20250903 20:05:09.221208 27064 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 943146f84c4a4adf89c5c2e91f9f04d4 (table default.Table_dbf9fa7169a64141bbe6afa3f74254a8 [id=322e6a14fef14f5c8623684144339103]). Expected version 2 got 0
I20250903 20:05:09.221488 27693 ts_tablet_manager.cc:1428] T 218f51a5391141b8941a2b40980147fd P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20250903 20:05:09.221625 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet cd3092c89c74475a999f5b3a6d77e652
I20250903 20:05:09.223453 27505 tablet_service.cc:1515] Processing DeleteTablet for tablet cd3092c89c74475a999f5b3a6d77e652 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:09.223479 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet e0d1b29d7f1e422e8bb5b8bbc33423e8
W20250903 20:05:09.223913 27059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fd499b81344c4b58bf77f3622422763c consensus state has no leader
W20250903 20:05:09.225054 27066 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 43f88abc8f3640a48598c07c3d76755d consensus state has no leader
W20250903 20:05:09.225173 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c8a293c6fa034e9fb29682439cb3bcc2 consensus state has no leader
W20250903 20:05:09.225373 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet cd3092c89c74475a999f5b3a6d77e652 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd3092c89c74475a999f5b3a6d77e652 already in progress: creating tablet
I20250903 20:05:09.225518 27505 tablet_service.cc:1515] Processing DeleteTablet for tablet e0d1b29d7f1e422e8bb5b8bbc33423e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:09.225569 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet c12c957a36f643d9bfb60b755619cbbc
W20250903 20:05:09.225762 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3906285457f449e6824d1f044bffca24 consensus state has no leader
I20250903 20:05:09.226764 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet c12c957a36f643d9bfb60b755619cbbc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:09.226794 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet cd3092c89c74475a999f5b3a6d77e652 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:09.227571 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet c12c957a36f643d9bfb60b755619cbbc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c12c957a36f643d9bfb60b755619cbbc already in progress: creating tablet
W20250903 20:05:09.227898 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet cd3092c89c74475a999f5b3a6d77e652 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd3092c89c74475a999f5b3a6d77e652 already in progress: creating tablet
I20250903 20:05:09.228026 27505 tablet_service.cc:1515] Processing DeleteTablet for tablet c12c957a36f643d9bfb60b755619cbbc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:09.229310 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet e0d1b29d7f1e422e8bb5b8bbc33423e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0d1b29d7f1e422e8bb5b8bbc33423e8 already in progress: creating tablet
I20250903 20:05:09.229480 27063 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 943146f84c4a4adf89c5c2e91f9f04d4 (table default.Table_dbf9fa7169a64141bbe6afa3f74254a8 [id=322e6a14fef14f5c8623684144339103]). Expected version 2 got 0
W20250903 20:05:09.229923 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet c12c957a36f643d9bfb60b755619cbbc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c12c957a36f643d9bfb60b755619cbbc already in progress: creating tablet
I20250903 20:05:09.232561 27693 tablet_bootstrap.cc:492] T 943146f84c4a4adf89c5c2e91f9f04d4 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:09.234555 27693 tablet_bootstrap.cc:654] T 943146f84c4a4adf89c5c2e91f9f04d4 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:09.237859 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet e0d1b29d7f1e422e8bb5b8bbc33423e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:09.238662 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet e0d1b29d7f1e422e8bb5b8bbc33423e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0d1b29d7f1e422e8bb5b8bbc33423e8 already in progress: creating tablet
I20250903 20:05:09.240471 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet e0d1b29d7f1e422e8bb5b8bbc33423e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:09.245704 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6d1b2694cd1a4963ac658f25026e9ef5" } modify_external_catalogs: true
I20250903 20:05:09.245921 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6d1b2694cd1a4963ac658f25026e9ef5" } modify_external_catalogs: true
I20250903 20:05:09.250200 27061 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 07361048416c43b6bc43d8737906db99 (table default.Table_260dadc90c8e415c8dee534464e23000 [id=9cea308979eb4c7f9deaa02e1c8feee9]). Expected version 1 got 0
I20250903 20:05:09.253554 27061 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 07361048416c43b6bc43d8737906db99 (table default.Table_260dadc90c8e415c8dee534464e23000 [id=9cea308979eb4c7f9deaa02e1c8feee9]). Expected version 1 got 0
I20250903 20:05:09.254096 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet c12c957a36f643d9bfb60b755619cbbc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:09.255215 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 0062307c6ee54aada74a08cf0d087f17
I20250903 20:05:09.256101 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 0062307c6ee54aada74a08cf0d087f17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:09.256608 27505 tablet_service.cc:1515] Processing DeleteTablet for tablet 0062307c6ee54aada74a08cf0d087f17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:09.257655 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet 0062307c6ee54aada74a08cf0d087f17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:09.258548 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 0062307c6ee54aada74a08cf0d087f17 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0062307c6ee54aada74a08cf0d087f17 already in progress: creating tablet
W20250903 20:05:09.258967 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 0062307c6ee54aada74a08cf0d087f17 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0062307c6ee54aada74a08cf0d087f17 already in progress: creating tablet
I20250903 20:05:09.262009 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_a3f1ce69cd8943b88954a6ded73a8713"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:09.262804 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3f1ce69cd8943b88954a6ded73a8713 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:09.264998 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_f20a070ead204fcab2d81f7025c09666"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:09.265066 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_7e75298df3b54e3794ab39c8b0fbffc7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:09.265588 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e75298df3b54e3794ab39c8b0fbffc7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:09.265682 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f20a070ead204fcab2d81f7025c09666 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:09.266986 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_6f5d3b1c0701490fb110f36cf852a96d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:09.267470 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f5d3b1c0701490fb110f36cf852a96d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:09.267802 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_c1ea9c78f7e54c64be3dcad3fe7314d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:09.268558 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1ea9c78f7e54c64be3dcad3fe7314d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:09.264657 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet 1d9a2508437b41f08821a29f58163159 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:09.270375 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 1d9a2508437b41f08821a29f58163159 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d9a2508437b41f08821a29f58163159 already in progress: creating tablet
I20250903 20:05:09.272372 27672 raft_consensus.cc:513] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:09.273875 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b197747f8d134f978a2cd1fe3425a5d5" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:09.274096 27385 raft_consensus.cc:3058] T b197747f8d134f978a2cd1fe3425a5d5 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:09.274683 27183 leader_election.cc:343] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: b197747f8d134f978a2cd1fe3425a5d5
I20250903 20:05:09.275069 27672 leader_election.cc:290] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:09.276885 27719 raft_consensus.cc:513] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.278219 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "218f51a5391141b8941a2b40980147fd" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:09.278447 27386 raft_consensus.cc:3058] T 218f51a5391141b8941a2b40980147fd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:09.281095 27386 raft_consensus.cc:2466] T 218f51a5391141b8941a2b40980147fd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:09.278429 27385 raft_consensus.cc:2466] T b197747f8d134f978a2cd1fe3425a5d5 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:09.282394 27184 leader_election.cc:304] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:09.282784 27184 leader_election.cc:304] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:09.283726 27646 raft_consensus.cc:2802] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:09.285239 27063 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 943146f84c4a4adf89c5c2e91f9f04d4 (table default.Table_dbf9fa7169a64141bbe6afa3f74254a8 [id=322e6a14fef14f5c8623684144339103]). Expected version 2 got 0
I20250903 20:05:09.287153 27364 tablet_service.cc:1468] Processing CreateTablet for tablet eaa11d17303142ff935705c16f0b9690 (DEFAULT_TABLE table=default.Table_857612f25ca54f8b8d0ea31485d96a5d [id=ddebc54a9f9849478f0d1606dfe3bc96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.287647 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaa11d17303142ff935705c16f0b9690. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:09.288520 27183 leader_election.cc:343] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 218f51a5391141b8941a2b40980147fd
I20250903 20:05:09.288842 27719 leader_election.cc:290] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:09.289270 27719 raft_consensus.cc:2802] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:09.289429 27719 raft_consensus.cc:695] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:09.294103 27719 consensus_queue.cc:237] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:09.295661 27217 tablet_service.cc:1468] Processing CreateTablet for tablet eaa11d17303142ff935705c16f0b9690 (DEFAULT_TABLE table=default.Table_857612f25ca54f8b8d0ea31485d96a5d [id=ddebc54a9f9849478f0d1606dfe3bc96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.296559 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaa11d17303142ff935705c16f0b9690. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.296732 27216 tablet_service.cc:1468] Processing CreateTablet for tablet 7e0810c96eb24b8ab4278038801be245 (DEFAULT_TABLE table=default.Table_db7d825a12ef40739b70f11044170c72 [id=790f57164c7d4183a3df17ba7f230d24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.297389 27216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e0810c96eb24b8ab4278038801be245. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.299247 27362 tablet_service.cc:1468] Processing CreateTablet for tablet 7e0810c96eb24b8ab4278038801be245 (DEFAULT_TABLE table=default.Table_db7d825a12ef40739b70f11044170c72 [id=790f57164c7d4183a3df17ba7f230d24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.299922 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e0810c96eb24b8ab4278038801be245. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.302675 27365 tablet_service.cc:1468] Processing CreateTablet for tablet e722212b281a405fb08908df0a42d864 (DEFAULT_TABLE table=default.Table_5f1e055d858e4d31bce98ac57741cc30 [id=7fd6a8bf57a846f78b1bf70fa3ea5175]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.303045 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e722212b281a405fb08908df0a42d864. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.303306 27646 raft_consensus.cc:695] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:09.303797 27646 consensus_queue.cc:237] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.309005 27505 tablet_service.cc:1468] Processing CreateTablet for tablet eaa11d17303142ff935705c16f0b9690 (DEFAULT_TABLE table=default.Table_857612f25ca54f8b8d0ea31485d96a5d [id=ddebc54a9f9849478f0d1606dfe3bc96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.309406 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaa11d17303142ff935705c16f0b9690. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.318382 27503 tablet_service.cc:1468] Processing CreateTablet for tablet 7e0810c96eb24b8ab4278038801be245 (DEFAULT_TABLE table=default.Table_db7d825a12ef40739b70f11044170c72 [id=790f57164c7d4183a3df17ba7f230d24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.318892 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e0810c96eb24b8ab4278038801be245. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.331100 27693 tablet_bootstrap.cc:492] T 943146f84c4a4adf89c5c2e91f9f04d4 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:09.331388 27693 ts_tablet_manager.cc:1397] T 943146f84c4a4adf89c5c2e91f9f04d4 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.099s user 0.004s sys 0.000s
I20250903 20:05:09.332466 27693 raft_consensus.cc:357] T 943146f84c4a4adf89c5c2e91f9f04d4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.332811 27693 raft_consensus.cc:383] T 943146f84c4a4adf89c5c2e91f9f04d4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:09.332934 27693 raft_consensus.cc:738] T 943146f84c4a4adf89c5c2e91f9f04d4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:09.333273 27693 consensus_queue.cc:260] T 943146f84c4a4adf89c5c2e91f9f04d4 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.335412 27363 tablet_service.cc:1468] Processing CreateTablet for tablet fb76ba1d7cd54d6aa69d06a9a5618cd3 (DEFAULT_TABLE table=default.Table_a249558ab60e466ca7734ffeb24ac5fd [id=d1ed6b104f9541fbaf34c6b03b297f0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.335837 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb76ba1d7cd54d6aa69d06a9a5618cd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.336938 27215 tablet_service.cc:1468] Processing CreateTablet for tablet e722212b281a405fb08908df0a42d864 (DEFAULT_TABLE table=default.Table_5f1e055d858e4d31bce98ac57741cc30 [id=7fd6a8bf57a846f78b1bf70fa3ea5175]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.337656 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e722212b281a405fb08908df0a42d864. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.338582 27366 tablet_service.cc:1468] Processing CreateTablet for tablet a45604ad611e418e84dab793e04101a0 (DEFAULT_TABLE table=default.Table_92a25ef74d6648d0a0052fe858b10b77 [id=5383eb1038e6480893e6eab6c7f7dd39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.338958 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a45604ad611e418e84dab793e04101a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.342581 27358 tablet_service.cc:1468] Processing CreateTablet for tablet 0f4ef613b0ab4bca98294682b934f6ed (DEFAULT_TABLE table=default.Table_26d34ebce42f44858e5961a8baaff5a9 [id=d20347a8196e44459448223a3af2dead]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.342967 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f4ef613b0ab4bca98294682b934f6ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.345788 27213 tablet_service.cc:1468] Processing CreateTablet for tablet fb76ba1d7cd54d6aa69d06a9a5618cd3 (DEFAULT_TABLE table=default.Table_a249558ab60e466ca7734ffeb24ac5fd [id=d1ed6b104f9541fbaf34c6b03b297f0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.346721 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb76ba1d7cd54d6aa69d06a9a5618cd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.349007 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 59e120e5e39e4de8a239f13545e4b1af (DEFAULT_TABLE table=default.Table_2dd5b5956b44471da9d0b89e782d469e [id=0bfcd1b8b71d45db8d9c370bd1231a38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.349442 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59e120e5e39e4de8a239f13545e4b1af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.365182 27693 ts_tablet_manager.cc:1428] T 943146f84c4a4adf89c5c2e91f9f04d4 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.034s user 0.002s sys 0.000s
I20250903 20:05:09.369221 27693 tablet_bootstrap.cc:492] T ea10476553384e8fb9cf75fd1c6595da P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:09.371059 27693 tablet_bootstrap.cc:654] T ea10476553384e8fb9cf75fd1c6595da P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:09.385722 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 59e120e5e39e4de8a239f13545e4b1af (DEFAULT_TABLE table=default.Table_2dd5b5956b44471da9d0b89e782d469e [id=0bfcd1b8b71d45db8d9c370bd1231a38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.386545 27061 master_service.cc:948] ReplaceTablet: received request to replace tablet 806d4fa868224194a0cf05e81b0ff377 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:09.387573 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_35a3167eda1a45949f8cd8a281a88733"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:09.388419 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35a3167eda1a45949f8cd8a281a88733 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:09.389335 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_a9aa14a6383343bc83014581ac775319"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:09.389899 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a9aa14a6383343bc83014581ac775319 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:09.390511 27066 master_service.cc:948] ReplaceTablet: received request to replace tablet cd3092c89c74475a999f5b3a6d77e652 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:09.391422 27058 master_service.cc:948] ReplaceTablet: received request to replace tablet 9d37a5d51a184c76bb8496679dfe3991 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:09.392042 27061 master_service.cc:948] ReplaceTablet: received request to replace tablet 943146f84c4a4adf89c5c2e91f9f04d4 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:09.394459 27227 tablet_service.cc:1515] Processing DeleteTablet for tablet cd3092c89c74475a999f5b3a6d77e652 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:09.395008 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_260dadc90c8e415c8dee534464e23000" } modify_external_catalogs: true
I20250903 20:05:09.395262 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_260dadc90c8e415c8dee534464e23000" } modify_external_catalogs: true
I20250903 20:05:09.395995 27693 tablet_bootstrap.cc:492] T ea10476553384e8fb9cf75fd1c6595da P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:09.396271 27693 ts_tablet_manager.cc:1397] T ea10476553384e8fb9cf75fd1c6595da P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.004s
I20250903 20:05:09.396895 27503 tablet_service.cc:1468] Processing CreateTablet for tablet e722212b281a405fb08908df0a42d864 (DEFAULT_TABLE table=default.Table_5f1e055d858e4d31bce98ac57741cc30 [id=7fd6a8bf57a846f78b1bf70fa3ea5175]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.397383 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e722212b281a405fb08908df0a42d864. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.398702 27693 raft_consensus.cc:357] T ea10476553384e8fb9cf75fd1c6595da P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:09.399070 27693 raft_consensus.cc:383] T ea10476553384e8fb9cf75fd1c6595da P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:09.399230 27693 raft_consensus.cc:738] T ea10476553384e8fb9cf75fd1c6595da P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:09.399677 27693 consensus_queue.cc:260] T ea10476553384e8fb9cf75fd1c6595da P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:09.402072 27693 ts_tablet_manager.cc:1428] T ea10476553384e8fb9cf75fd1c6595da P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250903 20:05:09.403847 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 59e120e5e39e4de8a239f13545e4b1af (DEFAULT_TABLE table=default.Table_2dd5b5956b44471da9d0b89e782d469e [id=0bfcd1b8b71d45db8d9c370bd1231a38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.404350 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59e120e5e39e4de8a239f13545e4b1af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.408296 27693 tablet_bootstrap.cc:492] T 161f2c2afc2f4c82b2025f7daf460bd6 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:09.411193 27693 tablet_bootstrap.cc:654] T 161f2c2afc2f4c82b2025f7daf460bd6 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:09.405580 27226 tablet_service.cc:1468] Processing CreateTablet for tablet a45604ad611e418e84dab793e04101a0 (DEFAULT_TABLE table=default.Table_92a25ef74d6648d0a0052fe858b10b77 [id=5383eb1038e6480893e6eab6c7f7dd39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.417510 27505 tablet_service.cc:1468] Processing CreateTablet for tablet fb76ba1d7cd54d6aa69d06a9a5618cd3 (DEFAULT_TABLE table=default.Table_a249558ab60e466ca7734ffeb24ac5fd [id=d1ed6b104f9541fbaf34c6b03b297f0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.418037 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb76ba1d7cd54d6aa69d06a9a5618cd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.420405 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 0f4ef613b0ab4bca98294682b934f6ed (DEFAULT_TABLE table=default.Table_26d34ebce42f44858e5961a8baaff5a9 [id=d20347a8196e44459448223a3af2dead]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.422832 27504 tablet_service.cc:1468] Processing CreateTablet for tablet a45604ad611e418e84dab793e04101a0 (DEFAULT_TABLE table=default.Table_92a25ef74d6648d0a0052fe858b10b77 [id=5383eb1038e6480893e6eab6c7f7dd39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.423293 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a45604ad611e418e84dab793e04101a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.425599 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59e120e5e39e4de8a239f13545e4b1af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.426571 27502 tablet_service.cc:1468] Processing CreateTablet for tablet 0f4ef613b0ab4bca98294682b934f6ed (DEFAULT_TABLE table=default.Table_26d34ebce42f44858e5961a8baaff5a9 [id=d20347a8196e44459448223a3af2dead]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.427011 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f4ef613b0ab4bca98294682b934f6ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.430158 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a45604ad611e418e84dab793e04101a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.439304 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f4ef613b0ab4bca98294682b934f6ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.443203 27363 tablet_service.cc:1515] Processing DeleteTablet for tablet e0d1b29d7f1e422e8bb5b8bbc33423e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:09.445518 27363 tablet_service.cc:1515] Processing DeleteTablet for tablet c12c957a36f643d9bfb60b755619cbbc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:09.446580 27051 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet fd499b81344c4b58bf77f3622422763c (table default.Table_e914faf335d44d019f91bda71758c473 [id=374b9abdf6a54b38876a96d8ec4a0368], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet fd499b81344c4b58bf77f3622422763c consensus state has no leader
I20250903 20:05:09.446974 27363 tablet_service.cc:1515] Processing DeleteTablet for tablet 0bcfca89bc3b44d78bc1b875c796329b with delete_type TABLET_DATA_DELETED (replaced by tablet 1d9a2508437b41f08821a29f58163159) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:09.447454 27687 tablet_replica.cc:331] T 0bcfca89bc3b44d78bc1b875c796329b P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:09.448081 27687 raft_consensus.cc:2241] T 0bcfca89bc3b44d78bc1b875c796329b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:09.448625 27687 raft_consensus.cc:2270] T 0bcfca89bc3b44d78bc1b875c796329b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:09.449380 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3b8908bfafc443798402fc7b449f8c4c" } new_table_name: "default.Table_0c3cf60b884e47eaaa60b3e0e80d56a3" modify_external_catalogs: true
I20250903 20:05:09.450626 27687 ts_tablet_manager.cc:1905] T 0bcfca89bc3b44d78bc1b875c796329b P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:09.451315 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b4bd57e0d5384edaa027cfcdf1b1db23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:09.451916 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4bd57e0d5384edaa027cfcdf1b1db23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:09.453083 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_79300b77de8c404a9e85783ce977861c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:09.453704 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79300b77de8c404a9e85783ce977861c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:09.453648 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_2ace21da933742e69b8ec85acbd919c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:09.454160 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ace21da933742e69b8ec85acbd919c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:09.455736 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_92a25ef74d6648d0a0052fe858b10b77" } new_table_name: "default.Table_7d8b7496fbc04f35bab6d6da573d4868" modify_external_catalogs: true
I20250903 20:05:09.456571 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a1c7e65e2fd446f6a895e8e6cfc2e27d" } new_table_name: "default.Table_f6203681100e4faeb016f0e6afc283f7" modify_external_catalogs: true
I20250903 20:05:09.460565 27363 tablet_service.cc:1515] Processing DeleteTablet for tablet d08c9e1e7d564886a90f6c056794becc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:09.461036 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet ea10476553384e8fb9cf75fd1c6595da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:09.461814 27687 ts_tablet_manager.cc:1918] T 0bcfca89bc3b44d78bc1b875c796329b P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:09.462002 27687 log.cc:1199] T 0bcfca89bc3b44d78bc1b875c796329b P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/0bcfca89bc3b44d78bc1b875c796329b
I20250903 20:05:09.462508 27687 ts_tablet_manager.cc:1939] T 0bcfca89bc3b44d78bc1b875c796329b P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:09.462638 27363 tablet_service.cc:1515] Processing DeleteTablet for tablet 806d4fa868224194a0cf05e81b0ff377 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:09.462988 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet 218f51a5391141b8941a2b40980147fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:09.463886 27363 tablet_service.cc:1515] Processing DeleteTablet for tablet b197747f8d134f978a2cd1fe3425a5d5 with delete_type TABLET_DATA_DELETED (replaced by tablet 03f5657ca6e44313b1541bc93eedf621) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:09.464190 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet 6dc3e66c6446439abd3eba441f4c782f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:09.465044 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet e0d1b29d7f1e422e8bb5b8bbc33423e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0d1b29d7f1e422e8bb5b8bbc33423e8 already in progress: creating tablet
I20250903 20:05:09.468659 27687 tablet_replica.cc:331] T ea10476553384e8fb9cf75fd1c6595da P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:09.469003 27687 raft_consensus.cc:2241] T ea10476553384e8fb9cf75fd1c6595da P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:09.469310 27687 raft_consensus.cc:2270] T ea10476553384e8fb9cf75fd1c6595da P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:09.470232 27687 ts_tablet_manager.cc:1905] T ea10476553384e8fb9cf75fd1c6595da P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:09.471781 27695 tablet_bootstrap.cc:654] T 806d4fa868224194a0cf05e81b0ff377 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:09.476693 27687 ts_tablet_manager.cc:1918] T ea10476553384e8fb9cf75fd1c6595da P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:09.476871 27687 log.cc:1199] T ea10476553384e8fb9cf75fd1c6595da P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/ea10476553384e8fb9cf75fd1c6595da
I20250903 20:05:09.477298 27687 ts_tablet_manager.cc:1939] T ea10476553384e8fb9cf75fd1c6595da P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:09.484946 27687 tablet_replica.cc:331] T 218f51a5391141b8941a2b40980147fd P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:09.485196 27687 raft_consensus.cc:2241] T 218f51a5391141b8941a2b40980147fd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:09.485409 27687 raft_consensus.cc:2270] T 218f51a5391141b8941a2b40980147fd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:09.486816 27692 tablet_bootstrap.cc:492] T d08c9e1e7d564886a90f6c056794becc P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:09.486996 27692 ts_tablet_manager.cc:1397] T d08c9e1e7d564886a90f6c056794becc P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.344s user 0.103s sys 0.180s
I20250903 20:05:09.487874 27692 raft_consensus.cc:357] T d08c9e1e7d564886a90f6c056794becc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.488086 27692 raft_consensus.cc:383] T d08c9e1e7d564886a90f6c056794becc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:09.488170 27692 raft_consensus.cc:738] T d08c9e1e7d564886a90f6c056794becc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:09.488440 27692 consensus_queue.cc:260] T d08c9e1e7d564886a90f6c056794becc P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.488974 27687 ts_tablet_manager.cc:1905] T 218f51a5391141b8941a2b40980147fd P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:09.489213 27692 ts_tablet_manager.cc:1428] T d08c9e1e7d564886a90f6c056794becc P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250903 20:05:09.489523 27692 tablet_bootstrap.cc:492] T 161f2c2afc2f4c82b2025f7daf460bd6 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:09.491416 27692 tablet_bootstrap.cc:654] T 161f2c2afc2f4c82b2025f7daf460bd6 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:09.496793 27687 ts_tablet_manager.cc:1918] T 218f51a5391141b8941a2b40980147fd P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:09.497015 27687 log.cc:1199] T 218f51a5391141b8941a2b40980147fd P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/218f51a5391141b8941a2b40980147fd
I20250903 20:05:09.497443 27687 ts_tablet_manager.cc:1939] T 218f51a5391141b8941a2b40980147fd P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:09.498605 27687 tablet_replica.cc:331] T b197747f8d134f978a2cd1fe3425a5d5 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:09.498857 27687 raft_consensus.cc:2241] T b197747f8d134f978a2cd1fe3425a5d5 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:09.499073 27687 raft_consensus.cc:2270] T b197747f8d134f978a2cd1fe3425a5d5 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:09.499670 27687 ts_tablet_manager.cc:1905] T b197747f8d134f978a2cd1fe3425a5d5 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:09.506754 27685 ts_tablet_manager.cc:1918] T d08c9e1e7d564886a90f6c056794becc P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:09.507180 27685 log.cc:1199] T d08c9e1e7d564886a90f6c056794becc P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/d08c9e1e7d564886a90f6c056794becc
I20250903 20:05:09.507468 27687 ts_tablet_manager.cc:1918] T b197747f8d134f978a2cd1fe3425a5d5 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:09.507660 27687 log.cc:1199] T b197747f8d134f978a2cd1fe3425a5d5 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/b197747f8d134f978a2cd1fe3425a5d5
I20250903 20:05:09.507819 27685 ts_tablet_manager.cc:1939] T d08c9e1e7d564886a90f6c056794becc P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:09.508057 27687 ts_tablet_manager.cc:1939] T b197747f8d134f978a2cd1fe3425a5d5 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:09.509088 27687 tablet_replica.cc:331] T 6dc3e66c6446439abd3eba441f4c782f P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:09.508975 27685 tablet_replica.cc:331] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:09.509347 27687 raft_consensus.cc:2241] T 6dc3e66c6446439abd3eba441f4c782f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:09.509454 27685 raft_consensus.cc:2241] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:09.509631 27687 raft_consensus.cc:2270] T 6dc3e66c6446439abd3eba441f4c782f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:09.509791 27685 pending_rounds.cc:70] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a: Trying to abort 1 pending ops.
I20250903 20:05:09.510077 27685 pending_rounds.cc:77] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384908512112640 op_type: NO_OP noop_request { }
I20250903 20:05:09.510286 27685 raft_consensus.cc:2887] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:09.510465 27685 raft_consensus.cc:2270] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:09.519683 27687 ts_tablet_manager.cc:1905] T 6dc3e66c6446439abd3eba441f4c782f P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:09.520720 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet c12c957a36f643d9bfb60b755619cbbc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c12c957a36f643d9bfb60b755619cbbc already in progress: creating tablet
I20250903 20:05:09.521908 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet cd3092c89c74475a999f5b3a6d77e652 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:09.524382 27693 tablet_bootstrap.cc:492] T 161f2c2afc2f4c82b2025f7daf460bd6 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:09.524679 27693 ts_tablet_manager.cc:1397] T 161f2c2afc2f4c82b2025f7daf460bd6 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.117s user 0.000s sys 0.005s
I20250903 20:05:09.525610 27693 raft_consensus.cc:357] T 161f2c2afc2f4c82b2025f7daf460bd6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.525898 27693 raft_consensus.cc:383] T 161f2c2afc2f4c82b2025f7daf460bd6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:09.526023 27693 raft_consensus.cc:738] T 161f2c2afc2f4c82b2025f7daf460bd6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:09.526481 27693 consensus_queue.cc:260] T 161f2c2afc2f4c82b2025f7daf460bd6 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.527068 27687 ts_tablet_manager.cc:1918] T 6dc3e66c6446439abd3eba441f4c782f P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:09.527196 27687 log.cc:1199] T 6dc3e66c6446439abd3eba441f4c782f P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/6dc3e66c6446439abd3eba441f4c782f
I20250903 20:05:09.527668 27687 ts_tablet_manager.cc:1939] T 6dc3e66c6446439abd3eba441f4c782f P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:09.529104 27693 ts_tablet_manager.cc:1428] T 161f2c2afc2f4c82b2025f7daf460bd6 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:09.529454 27692 tablet_bootstrap.cc:492] T 161f2c2afc2f4c82b2025f7daf460bd6 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:09.529496 27693 tablet_bootstrap.cc:492] T 07361048416c43b6bc43d8737906db99 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:09.529711 27692 ts_tablet_manager.cc:1397] T 161f2c2afc2f4c82b2025f7daf460bd6 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.040s user 0.023s sys 0.015s
I20250903 20:05:09.530630 27692 raft_consensus.cc:357] T 161f2c2afc2f4c82b2025f7daf460bd6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.530860 27692 raft_consensus.cc:383] T 161f2c2afc2f4c82b2025f7daf460bd6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:09.530951 27692 raft_consensus.cc:738] T 161f2c2afc2f4c82b2025f7daf460bd6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:09.531231 27692 consensus_queue.cc:260] T 161f2c2afc2f4c82b2025f7daf460bd6 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.531726 27692 ts_tablet_manager.cc:1428] T 161f2c2afc2f4c82b2025f7daf460bd6 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250903 20:05:09.532044 27692 tablet_bootstrap.cc:492] T cd3092c89c74475a999f5b3a6d77e652 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:09.532958 27693 tablet_bootstrap.cc:654] T 07361048416c43b6bc43d8737906db99 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:09.533844 27692 tablet_bootstrap.cc:654] T cd3092c89c74475a999f5b3a6d77e652 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:09.538239 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f3a33e90dc1f4540aee457de96f5b95b has no consensus state
W20250903 20:05:09.543094 27052 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 54957us
W20250903 20:05:09.544193 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a45604ad611e418e84dab793e04101a0 consensus state has no leader
I20250903 20:05:09.544443 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_dbf9fa7169a64141bbe6afa3f74254a8" } modify_external_catalogs: true
I20250903 20:05:09.547497 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 0bcfca89bc3b44d78bc1b875c796329b (table default.Table_c106f52e6b6b458391a12b3043e7eedf [id=cbae62c266f24526affd029472f41b94]) successfully deleted
W20250903 20:05:09.547739 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet d08c9e1e7d564886a90f6c056794becc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d08c9e1e7d564886a90f6c056794becc already in progress: creating tablet
I20250903 20:05:09.548679 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet ea10476553384e8fb9cf75fd1c6595da (table default.Table_59b61e83a1dd4c7386d0ffca89f82104 [id=f378777cae174d209abd3bd7f5766259]) successfully deleted
W20250903 20:05:09.548965 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 806d4fa868224194a0cf05e81b0ff377 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 806d4fa868224194a0cf05e81b0ff377 already in progress: creating tablet
I20250903 20:05:09.549667 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 218f51a5391141b8941a2b40980147fd (table default.Table_d0d3dd3e485546579b73419a892afba3 [id=86d4245a96f94a789a163b51ae32906e]) successfully deleted
I20250903 20:05:09.549847 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet b197747f8d134f978a2cd1fe3425a5d5 (table default.Table_f7e6051d454149d685213bea3186fb79 [id=88e0361a865a4b28a08e6c7f30dee9e3]) successfully deleted
I20250903 20:05:09.550009 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 6dc3e66c6446439abd3eba441f4c782f (table default.Table_514f576d8f594b9b92981f4a2ad81f1b [id=8622aba89ccc48c5ac6a28e038fa9db5]) successfully deleted
I20250903 20:05:09.549405 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0d37ecfeb62240568297039867aa4c20" } new_table_name: "default.Table_37e1d397cf99443d87211c3a8e1b9b82" modify_external_catalogs: true
I20250903 20:05:09.552238 27059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_4ace9616711f4778a94a0827ff79cc07" } modify_external_catalogs: true
I20250903 20:05:09.552462 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_db311e4928f54b0d8fce9e417635b974" } new_table_name: "default.Table_b6f25e1f7a484577a9cbff11e0deb857" modify_external_catalogs: true
I20250903 20:05:09.552913 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_dbf9fa7169a64141bbe6afa3f74254a8" } modify_external_catalogs: true
W20250903 20:05:09.558681 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet cd3092c89c74475a999f5b3a6d77e652 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd3092c89c74475a999f5b3a6d77e652 already in progress: creating tablet
I20250903 20:05:09.560107 27362 tablet_service.cc:1468] Processing CreateTablet for tablet be278233c7c34061988b267a44ee674f (DEFAULT_TABLE table=default.Table_2facdec2620546d5bf3dcdbaaf76a56e [id=3c69612aaf164dc899dcc252d85636f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.560725 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be278233c7c34061988b267a44ee674f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.561473 27059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_4ace9616711f4778a94a0827ff79cc07" } modify_external_catalogs: true
W20250903 20:05:09.563899 27066 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f99c05e72b9a4ff9a1e7c269ca029b6a consensus state has no leader
I20250903 20:05:09.565800 27366 tablet_service.cc:1468] Processing CreateTablet for tablet 99a21a3e49934a79a66700c0b0d86307 (DEFAULT_TABLE table=default.Table_a3f1ce69cd8943b88954a6ded73a8713 [id=428657b402d748c0a9e3942d13342b4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.566195 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99a21a3e49934a79a66700c0b0d86307. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.567395 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2dd5b5956b44471da9d0b89e782d469e" } modify_external_catalogs: true
I20250903 20:05:09.567552 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2dd5b5956b44471da9d0b89e782d469e" } modify_external_catalogs: true
I20250903 20:05:09.569633 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet d08c9e1e7d564886a90f6c056794becc (table default.Table_562dd57ebbc14070b3c772c08e93d7df [id=02a9013803934e17884b377e69dd6cac]) successfully deleted
W20250903 20:05:09.573549 27730 log.cc:927] Time spent T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
W20250903 20:05:09.574254 27730 consensus_queue.cc:1162] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "89f8f0ee68b940db9ce250c0604f934a" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20250903 20:05:09.574628 27505 tablet_service.cc:1468] Processing CreateTablet for tablet be278233c7c34061988b267a44ee674f (DEFAULT_TABLE table=default.Table_2facdec2620546d5bf3dcdbaaf76a56e [id=3c69612aaf164dc899dcc252d85636f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.575047 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be278233c7c34061988b267a44ee674f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.575060 27225 tablet_service.cc:1468] Processing CreateTablet for tablet be278233c7c34061988b267a44ee674f (DEFAULT_TABLE table=default.Table_2facdec2620546d5bf3dcdbaaf76a56e [id=3c69612aaf164dc899dcc252d85636f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.575683 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be278233c7c34061988b267a44ee674f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.577934 27503 tablet_service.cc:1468] Processing CreateTablet for tablet f3a33e90dc1f4540aee457de96f5b95b (DEFAULT_TABLE table=default.Table_f6203681100e4faeb016f0e6afc283f7 [id=91268ab201a4423faedd970b9a296152]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.578509 27502 tablet_service.cc:1468] Processing CreateTablet for tablet e1a49d0599654552bf1c72857031699b (DEFAULT_TABLE table=default.Table_f9e41274ca534e46b7906480d7fc32fe [id=febabbd421d241948dd40f5480c052bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.579023 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1a49d0599654552bf1c72857031699b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.580271 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a249558ab60e466ca7734ffeb24ac5fd" } modify_external_catalogs: true
I20250903 20:05:09.581277 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 99a21a3e49934a79a66700c0b0d86307 (DEFAULT_TABLE table=default.Table_a3f1ce69cd8943b88954a6ded73a8713 [id=428657b402d748c0a9e3942d13342b4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.583827 27217 tablet_service.cc:1468] Processing CreateTablet for tablet f3a33e90dc1f4540aee457de96f5b95b (DEFAULT_TABLE table=default.Table_f6203681100e4faeb016f0e6afc283f7 [id=91268ab201a4423faedd970b9a296152]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.584335 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3a33e90dc1f4540aee457de96f5b95b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.584354 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99a21a3e49934a79a66700c0b0d86307. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.586997 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a249558ab60e466ca7734ffeb24ac5fd" } modify_external_catalogs: true
W20250903 20:05:09.587071 27321 leader_election.cc:343] T 6dc3e66c6446439abd3eba441f4c782f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 6dc3e66c6446439abd3eba441f4c782f
I20250903 20:05:09.587291 27321 leader_election.cc:304] T 6dc3e66c6446439abd3eba441f4c782f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:09.587720 27321 raft_consensus.cc:2684] T 6dc3e66c6446439abd3eba441f4c782f P 51578fb7bceb4e119b3ab1e58c4f5528: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250903 20:05:09.589108 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 07361048416c43b6bc43d8737906db99
I20250903 20:05:09.597589 27363 tablet_service.cc:1468] Processing CreateTablet for tablet e1a49d0599654552bf1c72857031699b (DEFAULT_TABLE table=default.Table_f9e41274ca534e46b7906480d7fc32fe [id=febabbd421d241948dd40f5480c052bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.598243 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1a49d0599654552bf1c72857031699b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.602088 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 943146f84c4a4adf89c5c2e91f9f04d4
I20250903 20:05:09.602622 27061 catalog_manager.cc:6652] ReplaceTablet: tablet 943146f84c4a4adf89c5c2e91f9f04d4 deleted and replaced by tablet 8a1edae5527f4862bcb2fcca31f24b76
I20250903 20:05:09.602990 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_835ced1f8b194192809c3b6fbf4d1d96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:09.603590 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_835ced1f8b194192809c3b6fbf4d1d96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:09.605183 27719 raft_consensus.cc:491] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:09.605468 27719 raft_consensus.cc:513] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:09.606899 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b6b4713936134681aaef1732d46c6ab2 consensus state has no leader
I20250903 20:05:09.607002 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "161f2c2afc2f4c82b2025f7daf460bd6" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:09.607213 27064 master_service.cc:948] ReplaceTablet: received request to replace tablet 161f2c2afc2f4c82b2025f7daf460bd6 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:09.607297 27385 raft_consensus.cc:2466] T 161f2c2afc2f4c82b2025f7daf460bd6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:09.607957 27719 leader_election.cc:290] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:09.608182 27225 tablet_service.cc:1468] Processing CreateTablet for tablet e1a49d0599654552bf1c72857031699b (DEFAULT_TABLE table=default.Table_f9e41274ca534e46b7906480d7fc32fe [id=febabbd421d241948dd40f5480c052bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.608652 27184 leader_election.cc:304] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:09.609061 27719 raft_consensus.cc:2802] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:09.609182 27719 raft_consensus.cc:491] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:09.609323 27719 raft_consensus.cc:3058] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:09.609570 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet d1e1add08013459ba0dab3b7e66f0136
W20250903 20:05:09.613857 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c8a293c6fa034e9fb29682439cb3bcc2 consensus state has no leader
I20250903 20:05:09.618182 27358 tablet_service.cc:1515] Processing DeleteTablet for tablet d1e1add08013459ba0dab3b7e66f0136 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:09.618335 27366 tablet_service.cc:1515] Processing DeleteTablet for tablet 943146f84c4a4adf89c5c2e91f9f04d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 8a1edae5527f4862bcb2fcca31f24b76) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:09.618950 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 07361048416c43b6bc43d8737906db99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:09.619011 27687 tablet_replica.cc:331] T 943146f84c4a4adf89c5c2e91f9f04d4 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:09.619284 27687 raft_consensus.cc:2241] T 943146f84c4a4adf89c5c2e91f9f04d4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250903 20:05:09.619313 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet d1e1add08013459ba0dab3b7e66f0136 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1e1add08013459ba0dab3b7e66f0136 already in progress: creating tablet
I20250903 20:05:09.619575 27687 raft_consensus.cc:2270] T 943146f84c4a4adf89c5c2e91f9f04d4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:09.620486 27213 tablet_service.cc:1515] Processing DeleteTablet for tablet 07361048416c43b6bc43d8737906db99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:09.620515 27687 ts_tablet_manager.cc:1905] T 943146f84c4a4adf89c5c2e91f9f04d4 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:09.620757 27223 tablet_service.cc:1515] Processing DeleteTablet for tablet 943146f84c4a4adf89c5c2e91f9f04d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 8a1edae5527f4862bcb2fcca31f24b76) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:09.621038 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet d1e1add08013459ba0dab3b7e66f0136 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:09.621423 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3a33e90dc1f4540aee457de96f5b95b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.622885 27719 raft_consensus.cc:513] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.623966 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "161f2c2afc2f4c82b2025f7daf460bd6" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:09.624189 27385 raft_consensus.cc:3058] T 161f2c2afc2f4c82b2025f7daf460bd6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:09.625666 27719 leader_election.cc:290] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:09.626170 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 99a21a3e49934a79a66700c0b0d86307 (DEFAULT_TABLE table=default.Table_a3f1ce69cd8943b88954a6ded73a8713 [id=428657b402d748c0a9e3942d13342b4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.608675 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1a49d0599654552bf1c72857031699b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.627987 27063 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 8a1edae5527f4862bcb2fcca31f24b76
I20250903 20:05:09.628576 27385 raft_consensus.cc:2466] T 161f2c2afc2f4c82b2025f7daf460bd6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:09.628782 27687 ts_tablet_manager.cc:1918] T 943146f84c4a4adf89c5c2e91f9f04d4 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:09.628929 27687 log.cc:1199] T 943146f84c4a4adf89c5c2e91f9f04d4 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/943146f84c4a4adf89c5c2e91f9f04d4
I20250903 20:05:09.629247 27184 leader_election.cc:304] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:09.629305 27687 ts_tablet_manager.cc:1939] T 943146f84c4a4adf89c5c2e91f9f04d4 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:09.631163 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 0bcfca89bc3b44d78bc1b875c796329b with delete_type TABLET_DATA_DELETED (replaced by tablet 1d9a2508437b41f08821a29f58163159) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:09.631448 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet ea10476553384e8fb9cf75fd1c6595da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:09.632076 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_03ce0e151fcc4259ba52504c9c5ddfcd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:09.632685 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03ce0e151fcc4259ba52504c9c5ddfcd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:09.632784 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 07361048416c43b6bc43d8737906db99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07361048416c43b6bc43d8737906db99 already in progress: creating tablet
I20250903 20:05:09.633142 27366 tablet_service.cc:1515] Processing DeleteTablet for tablet 218f51a5391141b8941a2b40980147fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:09.633347 27358 tablet_service.cc:1515] Processing DeleteTablet for tablet b197747f8d134f978a2cd1fe3425a5d5 with delete_type TABLET_DATA_DELETED (replaced by tablet 03f5657ca6e44313b1541bc93eedf621) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:09.633399 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 0bcfca89bc3b44d78bc1b875c796329b because the tablet was not found. No further retry: Not found: Tablet not found: 0bcfca89bc3b44d78bc1b875c796329b
W20250903 20:05:09.633963 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet ea10476553384e8fb9cf75fd1c6595da because the tablet was not found. No further retry: Not found: Tablet not found: ea10476553384e8fb9cf75fd1c6595da
W20250903 20:05:09.634516 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 218f51a5391141b8941a2b40980147fd because the tablet was not found. No further retry: Not found: Tablet not found: 218f51a5391141b8941a2b40980147fd
I20250903 20:05:09.635036 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 6dc3e66c6446439abd3eba441f4c782f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:09.635057 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet b197747f8d134f978a2cd1fe3425a5d5 because the tablet was not found. No further retry: Not found: Tablet not found: b197747f8d134f978a2cd1fe3425a5d5
I20250903 20:05:09.635417 27719 raft_consensus.cc:2802] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
W20250903 20:05:09.635582 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 6dc3e66c6446439abd3eba441f4c782f because the tablet was not found. No further retry: Not found: Tablet not found: 6dc3e66c6446439abd3eba441f4c782f
I20250903 20:05:09.637207 27719 raft_consensus.cc:695] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:09.637511 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 59e120e5e39e4de8a239f13545e4b1af
I20250903 20:05:09.637693 27719 consensus_queue.cc:237] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.638270 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 59e120e5e39e4de8a239f13545e4b1af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:09.638499 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_c93f9a9bfe224141b9ff4492dfc131e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:09.638931 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet 59e120e5e39e4de8a239f13545e4b1af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:09.639050 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 59e120e5e39e4de8a239f13545e4b1af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59e120e5e39e4de8a239f13545e4b1af already in progress: creating tablet
W20250903 20:05:09.639204 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c93f9a9bfe224141b9ff4492dfc131e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:09.639585 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99a21a3e49934a79a66700c0b0d86307. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.642639 27695 tablet_bootstrap.cc:492] T 806d4fa868224194a0cf05e81b0ff377 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:09.642575 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_3055928e679e4dd8b8559b363b580e13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:09.642798 27695 ts_tablet_manager.cc:1397] T 806d4fa868224194a0cf05e81b0ff377 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.445s user 0.010s sys 0.005s
W20250903 20:05:09.643172 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3055928e679e4dd8b8559b363b580e13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:09.643731 27695 raft_consensus.cc:357] T 806d4fa868224194a0cf05e81b0ff377 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.644047 27695 raft_consensus.cc:383] T 806d4fa868224194a0cf05e81b0ff377 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:09.644150 27695 raft_consensus.cc:738] T 806d4fa868224194a0cf05e81b0ff377 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:09.644719 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet fb76ba1d7cd54d6aa69d06a9a5618cd3
I20250903 20:05:09.645269 27695 consensus_queue.cc:260] T 806d4fa868224194a0cf05e81b0ff377 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.645969 27695 ts_tablet_manager.cc:1428] T 806d4fa868224194a0cf05e81b0ff377 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:09.646400 27695 tablet_bootstrap.cc:492] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:09.646672 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet fb76ba1d7cd54d6aa69d06a9a5618cd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:09.647389 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet fb76ba1d7cd54d6aa69d06a9a5618cd3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb76ba1d7cd54d6aa69d06a9a5618cd3 already in progress: creating tablet
I20250903 20:05:09.650172 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "161f2c2afc2f4c82b2025f7daf460bd6" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:09.650511 27527 raft_consensus.cc:2466] T 161f2c2afc2f4c82b2025f7daf460bd6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:09.651466 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet fb76ba1d7cd54d6aa69d06a9a5618cd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:09.651899 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 07361048416c43b6bc43d8737906db99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:09.653231 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "161f2c2afc2f4c82b2025f7daf460bd6" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:09.653467 27525 raft_consensus.cc:3058] T 161f2c2afc2f4c82b2025f7daf460bd6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:09.656662 27695 tablet_bootstrap.cc:654] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:09.663584 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet d1e1add08013459ba0dab3b7e66f0136 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:09.663758 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_fdb8322c3c6341448be3a3c1fc482ead"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:09.663903 27501 tablet_service.cc:1515] Processing DeleteTablet for tablet d08c9e1e7d564886a90f6c056794becc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:09.664427 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fdb8322c3c6341448be3a3c1fc482ead in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:09.665370 27684 tablet_replica.cc:331] T d08c9e1e7d564886a90f6c056794becc P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:09.665910 27364 tablet_service.cc:1468] Processing CreateTablet for tablet f3a33e90dc1f4540aee457de96f5b95b (DEFAULT_TABLE table=default.Table_f6203681100e4faeb016f0e6afc283f7 [id=91268ab201a4423faedd970b9a296152]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.666404 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3a33e90dc1f4540aee457de96f5b95b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.667196 27684 raft_consensus.cc:2241] T d08c9e1e7d564886a90f6c056794becc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:09.667397 27684 raft_consensus.cc:2270] T d08c9e1e7d564886a90f6c056794becc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:09.667572 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 943146f84c4a4adf89c5c2e91f9f04d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 8a1edae5527f4862bcb2fcca31f24b76) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:09.670331 27684 ts_tablet_manager.cc:1905] T d08c9e1e7d564886a90f6c056794becc P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:09.670704 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 59e120e5e39e4de8a239f13545e4b1af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:09.671764 27501 tablet_service.cc:1515] Processing DeleteTablet for tablet fb76ba1d7cd54d6aa69d06a9a5618cd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:09.672055 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 07361048416c43b6bc43d8737906db99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07361048416c43b6bc43d8737906db99 already in progress: creating tablet
W20250903 20:05:09.672400 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet d1e1add08013459ba0dab3b7e66f0136 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1e1add08013459ba0dab3b7e66f0136 already in progress: creating tablet
I20250903 20:05:09.680418 27525 raft_consensus.cc:2466] T 161f2c2afc2f4c82b2025f7daf460bd6 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:09.688448 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_2a55e0f3a78d422ab61df9f9ddb6515b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:09.689364 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a55e0f3a78d422ab61df9f9ddb6515b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:09.690979 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e1991ea3a3d14b85a4ce9160fa00bc35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:09.691680 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1991ea3a3d14b85a4ce9160fa00bc35 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:09.693130 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_c9766617998c49b199a79052643d496a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:09.693791 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9766617998c49b199a79052643d496a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:09.698868 27684 ts_tablet_manager.cc:1918] T d08c9e1e7d564886a90f6c056794becc P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:09.699702 27684 log.cc:1199] T d08c9e1e7d564886a90f6c056794becc P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/d08c9e1e7d564886a90f6c056794becc
I20250903 20:05:09.700462 27684 ts_tablet_manager.cc:1939] T d08c9e1e7d564886a90f6c056794becc P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:09.701752 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet d08c9e1e7d564886a90f6c056794becc (table default.Table_562dd57ebbc14070b3c772c08e93d7df [id=02a9013803934e17884b377e69dd6cac]) successfully deleted
I20250903 20:05:09.707384 27684 tablet_replica.cc:331] T 943146f84c4a4adf89c5c2e91f9f04d4 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:09.707597 27684 raft_consensus.cc:2241] T 943146f84c4a4adf89c5c2e91f9f04d4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:09.711977 27684 raft_consensus.cc:2270] T 943146f84c4a4adf89c5c2e91f9f04d4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:09.712672 27684 ts_tablet_manager.cc:1905] T 943146f84c4a4adf89c5c2e91f9f04d4 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:09.712723 27685 ts_tablet_manager.cc:1905] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:09.722448 27695 tablet_bootstrap.cc:492] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:09.722662 27695 ts_tablet_manager.cc:1397] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.076s user 0.004s sys 0.000s
I20250903 20:05:09.723529 27695 raft_consensus.cc:357] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.723764 27695 raft_consensus.cc:383] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:09.723863 27695 raft_consensus.cc:738] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:09.724124 27695 consensus_queue.cc:260] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.724624 27063 master_service.cc:948] ReplaceTablet: received request to replace tablet 9d37a5d51a184c76bb8496679dfe3991 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:09.724858 27695 ts_tablet_manager.cc:1428] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:09.725263 27695 tablet_bootstrap.cc:492] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:09.727108 27695 tablet_bootstrap.cc:654] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:09.732004 27062 master_service.cc:948] ReplaceTablet: received request to replace tablet 43f88abc8f3640a48598c07c3d76755d from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:09.732460 27065 master_service.cc:948] ReplaceTablet: received request to replace tablet 7e0810c96eb24b8ab4278038801be245 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:09.747902 27684 ts_tablet_manager.cc:1918] T 943146f84c4a4adf89c5c2e91f9f04d4 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:09.748077 27684 log.cc:1199] T 943146f84c4a4adf89c5c2e91f9f04d4 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/943146f84c4a4adf89c5c2e91f9f04d4
I20250903 20:05:09.748528 27684 ts_tablet_manager.cc:1939] T 943146f84c4a4adf89c5c2e91f9f04d4 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:09.756906 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 7ba18e3f8cb148d484010cb4e72510b2 (DEFAULT_TABLE table=default.Table_a9aa14a6383343bc83014581ac775319 [id=83d0f963c409436faa6c9a78bc0167cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.759665 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ba18e3f8cb148d484010cb4e72510b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.763967 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 7ba18e3f8cb148d484010cb4e72510b2 (DEFAULT_TABLE table=default.Table_a9aa14a6383343bc83014581ac775319 [id=83d0f963c409436faa6c9a78bc0167cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.764084 27363 tablet_service.cc:1468] Processing CreateTablet for tablet b5bdc27275cd4156afe767a1fb33a8bd (DEFAULT_TABLE table=default.Table_f20a070ead204fcab2d81f7025c09666 [id=31f68383b3734a8691dfe0317fc60345]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.764415 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ba18e3f8cb148d484010cb4e72510b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.764464 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5bdc27275cd4156afe767a1fb33a8bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.771090 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 161f2c2afc2f4c82b2025f7daf460bd6
I20250903 20:05:09.771669 27064 catalog_manager.cc:6652] ReplaceTablet: tablet 161f2c2afc2f4c82b2025f7daf460bd6 deleted and replaced by tablet 925c0ab4873041da82702cd9377ec7bc
I20250903 20:05:09.772635 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet 161f2c2afc2f4c82b2025f7daf460bd6 with delete_type TABLET_DATA_DELETED (replaced by tablet 925c0ab4873041da82702cd9377ec7bc) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:09.774679 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 161f2c2afc2f4c82b2025f7daf460bd6 with delete_type TABLET_DATA_DELETED (replaced by tablet 925c0ab4873041da82702cd9377ec7bc) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:09.775413 27687 tablet_replica.cc:331] T 161f2c2afc2f4c82b2025f7daf460bd6 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:09.775599 27687 raft_consensus.cc:2241] T 161f2c2afc2f4c82b2025f7daf460bd6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:09.775769 27687 raft_consensus.cc:2270] T 161f2c2afc2f4c82b2025f7daf460bd6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:09.776607 27687 ts_tablet_manager.cc:1905] T 161f2c2afc2f4c82b2025f7daf460bd6 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:09.778925 27061 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet c8a293c6fa034e9fb29682439cb3bcc2 (table default.Table_37e1d397cf99443d87211c3a8e1b9b82 [id=5cfc8827e4ca4f4a94424c48d6f71eb5]). Expected version 2 got 0
I20250903 20:05:09.780689 27695 tablet_bootstrap.cc:492] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:09.780833 27695 ts_tablet_manager.cc:1397] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.003s
I20250903 20:05:09.781127 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 7e0810c96eb24b8ab4278038801be245
I20250903 20:05:09.781719 27695 raft_consensus.cc:357] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.781859 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e0810c96eb24b8ab4278038801be245 with delete_type TABLET_DATA_DELETED (replaced by tablet e5136d5388bc4c2ba71e09be40e8f412) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:09.781910 27695 raft_consensus.cc:383] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:09.781994 27695 raft_consensus.cc:738] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:09.782305 27065 catalog_manager.cc:6652] ReplaceTablet: tablet 7e0810c96eb24b8ab4278038801be245 deleted and replaced by tablet e5136d5388bc4c2ba71e09be40e8f412
I20250903 20:05:09.782433 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e0810c96eb24b8ab4278038801be245 with delete_type TABLET_DATA_DELETED (replaced by tablet e5136d5388bc4c2ba71e09be40e8f412) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:09.782759 27695 consensus_queue.cc:260] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.783669 27687 ts_tablet_manager.cc:1918] T 161f2c2afc2f4c82b2025f7daf460bd6 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:09.783874 27687 log.cc:1199] T 161f2c2afc2f4c82b2025f7daf460bd6 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/161f2c2afc2f4c82b2025f7daf460bd6
I20250903 20:05:09.784163 27362 tablet_service.cc:1468] Processing CreateTablet for tablet 4bd2650e4d9e40f5ba5c09e367918224 (DEFAULT_TABLE table=default.Table_2ace21da933742e69b8ec85acbd919c2 [id=dcc845689a884dc69c7217043de59d32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.784582 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bd2650e4d9e40f5ba5c09e367918224. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.785594 27719 raft_consensus.cc:491] T 806d4fa868224194a0cf05e81b0ff377 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:09.785732 27719 raft_consensus.cc:513] T 806d4fa868224194a0cf05e81b0ff377 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.786453 27719 leader_election.cc:290] T 806d4fa868224194a0cf05e81b0ff377 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:09.787513 27687 ts_tablet_manager.cc:1939] T 161f2c2afc2f4c82b2025f7daf460bd6 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:09.787492 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "806d4fa868224194a0cf05e81b0ff377" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:09.788120 27183 leader_election.cc:343] T 806d4fa868224194a0cf05e81b0ff377 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:09.788177 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "806d4fa868224194a0cf05e81b0ff377" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:09.788852 27225 tablet_service.cc:1468] Processing CreateTablet for tablet b5bdc27275cd4156afe767a1fb33a8bd (DEFAULT_TABLE table=default.Table_f20a070ead204fcab2d81f7025c09666 [id=31f68383b3734a8691dfe0317fc60345]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:09.789369 27184 leader_election.cc:343] T 806d4fa868224194a0cf05e81b0ff377 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:09.789567 27184 leader_election.cc:304] T 806d4fa868224194a0cf05e81b0ff377 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:09.789856 27695 ts_tablet_manager.cc:1428] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250903 20:05:09.790068 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 161f2c2afc2f4c82b2025f7daf460bd6 (table default.Table_1a5c8c3256c24c7f8ef8597efa53dde5 [id=7caef592f80a4d6cb6df4e2c3971de49]) successfully deleted
W20250903 20:05:09.790426 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 7e0810c96eb24b8ab4278038801be245 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e0810c96eb24b8ab4278038801be245 already in progress: creating tablet
I20250903 20:05:09.790757 27719 raft_consensus.cc:2747] T 806d4fa868224194a0cf05e81b0ff377 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:09.791345 27695 tablet_bootstrap.cc:492] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:09.789366 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5bdc27275cd4156afe767a1fb33a8bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.795496 27695 tablet_bootstrap.cc:654] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:09.808341 27065 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet b6b4713936134681aaef1732d46c6ab2 (table default.Table_b6f25e1f7a484577a9cbff11e0deb857 [id=d0a75545b91c48bba47fb581bb0d1a96]). Expected version 1 got 0
I20250903 20:05:09.809528 27695 tablet_bootstrap.cc:492] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:09.809729 27695 ts_tablet_manager.cc:1397] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.002s
I20250903 20:05:09.810742 27695 raft_consensus.cc:357] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.810966 27695 raft_consensus.cc:383] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:09.811100 27695 raft_consensus.cc:738] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:09.811335 27695 consensus_queue.cc:260] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.811866 27695 ts_tablet_manager.cc:1428] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:09.814389 27695 tablet_bootstrap.cc:492] T 12a7be47c9c2406596c9e00800074a7c P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:09.816128 27695 tablet_bootstrap.cc:654] T 12a7be47c9c2406596c9e00800074a7c P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:09.821967 27058 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet b6b4713936134681aaef1732d46c6ab2 (table default.Table_b6f25e1f7a484577a9cbff11e0deb857 [id=d0a75545b91c48bba47fb581bb0d1a96]). Expected version 1 got 0
I20250903 20:05:09.822381 27058 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet fd499b81344c4b58bf77f3622422763c (table default.Table_e914faf335d44d019f91bda71758c473 [id=374b9abdf6a54b38876a96d8ec4a0368]). Expected version 4 got 0
W20250903 20:05:09.824267 27051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.125:44059, user_credentials={real_user=slave}} blocked reactor thread for 54987.6us
I20250903 20:05:09.825066 27505 tablet_service.cc:1515] Processing DeleteTablet for tablet 161f2c2afc2f4c82b2025f7daf460bd6 with delete_type TABLET_DATA_DELETED (replaced by tablet 925c0ab4873041da82702cd9377ec7bc) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:09.825403 27684 tablet_replica.cc:331] T 161f2c2afc2f4c82b2025f7daf460bd6 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:09.825588 27684 raft_consensus.cc:2241] T 161f2c2afc2f4c82b2025f7daf460bd6 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:09.825789 27684 raft_consensus.cc:2270] T 161f2c2afc2f4c82b2025f7daf460bd6 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:09.826478 27505 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e0810c96eb24b8ab4278038801be245 with delete_type TABLET_DATA_DELETED (replaced by tablet e5136d5388bc4c2ba71e09be40e8f412) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:09.827059 27684 ts_tablet_manager.cc:1905] T 161f2c2afc2f4c82b2025f7daf460bd6 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:09.827262 27505 tablet_service.cc:1468] Processing CreateTablet for tablet b5bdc27275cd4156afe767a1fb33a8bd (DEFAULT_TABLE table=default.Table_f20a070ead204fcab2d81f7025c09666 [id=31f68383b3734a8691dfe0317fc60345]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.827275 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_53cc54536434460ba6637a083afaddb2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:09.827674 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5bdc27275cd4156afe767a1fb33a8bd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:09.827910 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53cc54536434460ba6637a083afaddb2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:09.829442 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b2a82a2c310140caaeac5d0dcb4bb61a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:09.830071 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2a82a2c310140caaeac5d0dcb4bb61a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:09.831678 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_26d34ebce42f44858e5961a8baaff5a9" } new_table_name: "default.Table_9b700cf2d51b4fca81d158e22d72a3f3" modify_external_catalogs: true
I20250903 20:05:09.835253 27226 tablet_service.cc:1468] Processing CreateTablet for tablet 4bd2650e4d9e40f5ba5c09e367918224 (DEFAULT_TABLE table=default.Table_2ace21da933742e69b8ec85acbd919c2 [id=dcc845689a884dc69c7217043de59d32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.835729 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bd2650e4d9e40f5ba5c09e367918224. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.837570 27695 tablet_bootstrap.cc:492] T 12a7be47c9c2406596c9e00800074a7c P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:09.837787 27695 ts_tablet_manager.cc:1397] T 12a7be47c9c2406596c9e00800074a7c P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.005s
I20250903 20:05:09.839139 27695 raft_consensus.cc:357] T 12a7be47c9c2406596c9e00800074a7c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.839417 27695 raft_consensus.cc:383] T 12a7be47c9c2406596c9e00800074a7c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:09.839565 27695 raft_consensus.cc:738] T 12a7be47c9c2406596c9e00800074a7c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:09.839970 27695 consensus_queue.cc:260] T 12a7be47c9c2406596c9e00800074a7c P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.841730 27695 ts_tablet_manager.cc:1428] T 12a7be47c9c2406596c9e00800074a7c P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250903 20:05:09.842034 27695 tablet_bootstrap.cc:492] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:09.843782 27684 ts_tablet_manager.cc:1918] T 161f2c2afc2f4c82b2025f7daf460bd6 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:09.844244 27695 tablet_bootstrap.cc:654] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:09.844513 27684 log.cc:1199] T 161f2c2afc2f4c82b2025f7daf460bd6 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/161f2c2afc2f4c82b2025f7daf460bd6
I20250903 20:05:09.844954 27684 ts_tablet_manager.cc:1939] T 161f2c2afc2f4c82b2025f7daf460bd6 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:09.856863 27364 tablet_service.cc:1468] Processing CreateTablet for tablet 7ba18e3f8cb148d484010cb4e72510b2 (DEFAULT_TABLE table=default.Table_a9aa14a6383343bc83014581ac775319 [id=83d0f963c409436faa6c9a78bc0167cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.857431 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ba18e3f8cb148d484010cb4e72510b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.860872 27362 tablet_service.cc:1468] Processing CreateTablet for tablet 1396d4f8a93b412ca8397368fb98834e (DEFAULT_TABLE table=default.Table_35a3167eda1a45949f8cd8a281a88733 [id=da438a4eb783471ea0b09fe1117ec822]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.861354 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1396d4f8a93b412ca8397368fb98834e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.868196 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c180dbba07434ca797370519dbfd20b7" } new_table_name: "default.Table_74d1733a6f3d40ccb6578a52969f03f6" modify_external_catalogs: true
I20250903 20:05:09.870060 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5de0a37e50d44850a1486650571595d4" } new_table_name: "default.Table_d5626bea3c88471b846f37ae4ed79715" modify_external_catalogs: true
I20250903 20:05:09.872229 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7d8b7496fbc04f35bab6d6da573d4868" } modify_external_catalogs: true
I20250903 20:05:09.872498 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7d8b7496fbc04f35bab6d6da573d4868" } modify_external_catalogs: true
I20250903 20:05:09.896216 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 1396d4f8a93b412ca8397368fb98834e (DEFAULT_TABLE table=default.Table_35a3167eda1a45949f8cd8a281a88733 [id=da438a4eb783471ea0b09fe1117ec822]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.896924 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c1ea9c78f7e54c64be3dcad3fe7314d7" } new_table_name: "default.Table_a0ba026e7424483ba463d3d0751e3e4e" modify_external_catalogs: true
I20250903 20:05:09.902068 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_fdb8322c3c6341448be3a3c1fc482ead" } modify_external_catalogs: true
I20250903 20:05:09.902107 27358 tablet_service.cc:1468] Processing CreateTablet for tablet 9367b3a22f624c4087848099de32c080 (DEFAULT_TABLE table=default.Table_6f5d3b1c0701490fb110f36cf852a96d [id=891b259c1c6b44e2a82dc982c24e6cb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.902392 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_fdb8322c3c6341448be3a3c1fc482ead" } modify_external_catalogs: true
I20250903 20:05:09.902761 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9367b3a22f624c4087848099de32c080. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.905146 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c93f9a9bfe224141b9ff4492dfc131e1" } new_table_name: "default.Table_646dcf6e43a24b31aaa79d2f93c58a14" modify_external_catalogs: true
I20250903 20:05:09.907500 27719 raft_consensus.cc:491] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:09.907742 27719 raft_consensus.cc:513] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:09.908699 27719 leader_election.cc:290] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:09.909713 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6b4713936134681aaef1732d46c6ab2" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:09.910460 27184 leader_election.cc:343] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:09.910828 27226 tablet_service.cc:1468] Processing CreateTablet for tablet 9367b3a22f624c4087848099de32c080 (DEFAULT_TABLE table=default.Table_6f5d3b1c0701490fb110f36cf852a96d [id=891b259c1c6b44e2a82dc982c24e6cb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.910975 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6b4713936134681aaef1732d46c6ab2" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:09.911424 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9367b3a22f624c4087848099de32c080. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:09.911725 27183 leader_election.cc:343] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:09.911990 27183 leader_election.cc:304] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:09.912411 27719 raft_consensus.cc:2747] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:09.917575 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1396d4f8a93b412ca8397368fb98834e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:09.931684 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 59e120e5e39e4de8a239f13545e4b1af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59e120e5e39e4de8a239f13545e4b1af already in progress: creating tablet
W20250903 20:05:09.931985 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet fb76ba1d7cd54d6aa69d06a9a5618cd3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb76ba1d7cd54d6aa69d06a9a5618cd3 already in progress: creating tablet
I20250903 20:05:09.932310 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 161f2c2afc2f4c82b2025f7daf460bd6 (table default.Table_1a5c8c3256c24c7f8ef8597efa53dde5 [id=7caef592f80a4d6cb6df4e2c3971de49]) successfully deleted
W20250903 20:05:09.932554 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 7e0810c96eb24b8ab4278038801be245 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e0810c96eb24b8ab4278038801be245 already in progress: creating tablet
I20250903 20:05:09.933660 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 4bd2650e4d9e40f5ba5c09e367918224 (DEFAULT_TABLE table=default.Table_2ace21da933742e69b8ec85acbd919c2 [id=dcc845689a884dc69c7217043de59d32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.934181 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bd2650e4d9e40f5ba5c09e367918224. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.962932 27685 ts_tablet_manager.cc:1918] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:09.963114 27685 log.cc:1199] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/218f51a5391141b8941a2b40980147fd
I20250903 20:05:09.963042 27358 tablet_service.cc:1468] Processing CreateTablet for tablet 4692dd93b76c42c4883c7e22c6032250 (DEFAULT_TABLE table=default.Table_c1ea9c78f7e54c64be3dcad3fe7314d7 [id=f11bdb54ed5c4b1ca1e1306d39aaf25f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.963662 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4692dd93b76c42c4883c7e22c6032250. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.963686 27685 ts_tablet_manager.cc:1939] T 218f51a5391141b8941a2b40980147fd P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:09.964645 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 13) took 1318 ms. Trace:
I20250903 20:05:09.964815 27685 rpcz_store.cc:276] 0903 20:05:08.646388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:08.646459 (+ 71us) service_pool.cc:225] Handling call
0903 20:05:09.508887 (+862428us) ts_tablet_manager.cc:1145] Deleting tablet 218f51a5391141b8941a2b40980147fd
0903 20:05:09.508898 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:09.716407 (+207509us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:09.962750 (+246343us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:09.964592 (+ 1842us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":862229}
I20250903 20:05:09.966358 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 218f51a5391141b8941a2b40980147fd (table default.Table_d0d3dd3e485546579b73419a892afba3 [id=86d4245a96f94a789a163b51ae32906e]) successfully deleted
I20250903 20:05:09.971551 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 4692dd93b76c42c4883c7e22c6032250 (DEFAULT_TABLE table=default.Table_c1ea9c78f7e54c64be3dcad3fe7314d7 [id=f11bdb54ed5c4b1ca1e1306d39aaf25f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.972049 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4692dd93b76c42c4883c7e22c6032250. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.984594 27358 tablet_service.cc:1468] Processing CreateTablet for tablet 29581a9dba46479884a0e6e01b7923de (DEFAULT_TABLE table=default.Table_7e75298df3b54e3794ab39c8b0fbffc7 [id=cf5354e1f11f4a228ea25ec2baa569a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.985234 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29581a9dba46479884a0e6e01b7923de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:09.990895 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 29581a9dba46479884a0e6e01b7923de (DEFAULT_TABLE table=default.Table_7e75298df3b54e3794ab39c8b0fbffc7 [id=cf5354e1f11f4a228ea25ec2baa569a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:09.991384 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29581a9dba46479884a0e6e01b7923de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.001696 27358 tablet_service.cc:1468] Processing CreateTablet for tablet 46b5d491888147bbb101963641fd05f8 (DEFAULT_TABLE table=default.Table_b4bd57e0d5384edaa027cfcdf1b1db23 [id=3b241b63c32f42c0966e9af3dc06d701]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.002367 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46b5d491888147bbb101963641fd05f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.002417 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 1396d4f8a93b412ca8397368fb98834e (DEFAULT_TABLE table=default.Table_35a3167eda1a45949f8cd8a281a88733 [id=da438a4eb783471ea0b09fe1117ec822]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.002943 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1396d4f8a93b412ca8397368fb98834e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.014447 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 9367b3a22f624c4087848099de32c080 (DEFAULT_TABLE table=default.Table_6f5d3b1c0701490fb110f36cf852a96d [id=891b259c1c6b44e2a82dc982c24e6cb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.014976 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9367b3a22f624c4087848099de32c080. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.018560 27502 tablet_service.cc:1468] Processing CreateTablet for tablet 4692dd93b76c42c4883c7e22c6032250 (DEFAULT_TABLE table=default.Table_c1ea9c78f7e54c64be3dcad3fe7314d7 [id=f11bdb54ed5c4b1ca1e1306d39aaf25f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.018993 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4692dd93b76c42c4883c7e22c6032250. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.019243 27685 tablet_replica.cc:331] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:10.019577 27685 raft_consensus.cc:2241] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:10.019928 27685 pending_rounds.cc:70] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a: Trying to abort 1 pending ops.
I20250903 20:05:10.020030 27685 pending_rounds.cc:77] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384908695678976 op_type: NO_OP noop_request { }
I20250903 20:05:10.020184 27685 raft_consensus.cc:2887] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:10.020341 27685 raft_consensus.cc:2270] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.022571 27503 tablet_service.cc:1468] Processing CreateTablet for tablet 29581a9dba46479884a0e6e01b7923de (DEFAULT_TABLE table=default.Table_7e75298df3b54e3794ab39c8b0fbffc7 [id=cf5354e1f11f4a228ea25ec2baa569a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.022965 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29581a9dba46479884a0e6e01b7923de. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:10.027077 27717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 20:05:10.027547 27717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 20:05:10.027667 27717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250903 20:05:10.037205 27695 tablet_bootstrap.cc:492] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:10.037396 27695 ts_tablet_manager.cc:1397] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.195s user 0.048s sys 0.134s
I20250903 20:05:10.038285 27695 raft_consensus.cc:357] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.038517 27695 raft_consensus.cc:383] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.038591 27695 raft_consensus.cc:738] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.039258 27685 ts_tablet_manager.cc:1905] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:10.040938 27717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 20:05:10.041028 27717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 20:05:10.041108 27717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 20:05:10.041179 27717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 20:05:10.044217 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 43f88abc8f3640a48598c07c3d76755d
I20250903 20:05:10.044709 27062 catalog_manager.cc:6652] ReplaceTablet: tablet 43f88abc8f3640a48598c07c3d76755d deleted and replaced by tablet 68cff0e416a54a5eb37071f5a12f07ab
I20250903 20:05:10.044970 27062 master_service.cc:948] ReplaceTablet: received request to replace tablet 0a266b1bbf1540ac86d49c56df2dd4ad from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:10.045063 27719 raft_consensus.cc:491] T 07361048416c43b6bc43d8737906db99 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:10.045189 27719 raft_consensus.cc:513] T 07361048416c43b6bc43d8737906db99 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.045786 27719 leader_election.cc:290] T 07361048416c43b6bc43d8737906db99 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:10.046244 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07361048416c43b6bc43d8737906db99" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:10.046705 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 43f88abc8f3640a48598c07c3d76755d with delete_type TABLET_DATA_DELETED (replaced by tablet 68cff0e416a54a5eb37071f5a12f07ab) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:10.046808 27183 leader_election.cc:343] T 07361048416c43b6bc43d8737906db99 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:10.047133 27695 consensus_queue.cc:260] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.047936 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07361048416c43b6bc43d8737906db99" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:10.048704 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 43f88abc8f3640a48598c07c3d76755d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43f88abc8f3640a48598c07c3d76755d already in progress: creating tablet
W20250903 20:05:10.049067 27184 leader_election.cc:343] T 07361048416c43b6bc43d8737906db99 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:10.049204 27184 leader_election.cc:304] T 07361048416c43b6bc43d8737906db99 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:10.050808 27719 raft_consensus.cc:2747] T 07361048416c43b6bc43d8737906db99 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:10.053376 27695 ts_tablet_manager.cc:1428] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20250903 20:05:10.053746 27695 tablet_bootstrap.cc:492] T 43f88abc8f3640a48598c07c3d76755d P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:10.058404 27717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.65.84:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.26.65.126:39015,127.26.65.125:44059,127.26.65.124:40911
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.65.126:39015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.65.126
--webserver_port=33509
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 19:52:33 UTC on 24a791456cd2
build id 7884
ASAN enabled
I20250903 20:05:10.060716 27717 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 20:05:10.062397 27214 tablet_service.cc:1468] Processing CreateTablet for tablet c0343d2226e345669d42375de2545986 (DEFAULT_TABLE table=default.Table_79300b77de8c404a9e85783ce977861c [id=7d6d5a3d8906427e83ab3153b7bb4235]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.063073 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0343d2226e345669d42375de2545986. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.063290 27717 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250903 20:05:10.063102 27364 tablet_service.cc:1468] Processing CreateTablet for tablet c0343d2226e345669d42375de2545986 (DEFAULT_TABLE table=default.Table_79300b77de8c404a9e85783ce977861c [id=7d6d5a3d8906427e83ab3153b7bb4235]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.066972 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0343d2226e345669d42375de2545986. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.071707 27695 tablet_bootstrap.cc:654] T 43f88abc8f3640a48598c07c3d76755d P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:10.073408 27226 tablet_service.cc:1468] Processing CreateTablet for tablet 46b5d491888147bbb101963641fd05f8 (DEFAULT_TABLE table=default.Table_b4bd57e0d5384edaa027cfcdf1b1db23 [id=3b241b63c32f42c0966e9af3dc06d701]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.075101 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46b5d491888147bbb101963641fd05f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.080700 27685 ts_tablet_manager.cc:1918] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:10.080881 27685 log.cc:1199] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/b197747f8d134f978a2cd1fe3425a5d5
I20250903 20:05:10.081027 27693 tablet_bootstrap.cc:492] T 07361048416c43b6bc43d8737906db99 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:10.081291 27685 ts_tablet_manager.cc:1939] T b197747f8d134f978a2cd1fe3425a5d5 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:10.081305 27693 ts_tablet_manager.cc:1397] T 07361048416c43b6bc43d8737906db99 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.552s user 0.011s sys 0.027s
I20250903 20:05:10.082065 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 14) took 1423 ms. Trace:
I20250903 20:05:10.082262 27685 rpcz_store.cc:276] 0903 20:05:08.658368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:08.658429 (+ 61us) service_pool.cc:225] Handling call
0903 20:05:10.019118 (+1360689us) ts_tablet_manager.cc:1145] Deleting tablet b197747f8d134f978a2cd1fe3425a5d5
0903 20:05:10.019137 (+ 19us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:10.072796 (+ 53659us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:10.080535 (+ 7739us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:10.082028 (+ 1493us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1360433}
I20250903 20:05:10.082638 27685 tablet_replica.cc:331] T 9d37a5d51a184c76bb8496679dfe3991 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:10.082834 27685 raft_consensus.cc:2241] T 9d37a5d51a184c76bb8496679dfe3991 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.082981 27685 raft_consensus.cc:2270] T 9d37a5d51a184c76bb8496679dfe3991 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.083596 27685 ts_tablet_manager.cc:1905] T 9d37a5d51a184c76bb8496679dfe3991 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.083547 27693 raft_consensus.cc:357] T 07361048416c43b6bc43d8737906db99 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.083936 27693 raft_consensus.cc:383] T 07361048416c43b6bc43d8737906db99 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.084071 27693 raft_consensus.cc:738] T 07361048416c43b6bc43d8737906db99 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.084512 27693 consensus_queue.cc:260] T 07361048416c43b6bc43d8737906db99 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.086184 27692 tablet_bootstrap.cc:492] T cd3092c89c74475a999f5b3a6d77e652 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:10.086396 27692 ts_tablet_manager.cc:1397] T cd3092c89c74475a999f5b3a6d77e652 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.554s user 0.081s sys 0.261s
I20250903 20:05:10.087126 27695 tablet_bootstrap.cc:492] T 43f88abc8f3640a48598c07c3d76755d P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:10.087275 27695 ts_tablet_manager.cc:1397] T 43f88abc8f3640a48598c07c3d76755d P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20250903 20:05:10.087303 27692 raft_consensus.cc:357] T cd3092c89c74475a999f5b3a6d77e652 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.087482 27692 raft_consensus.cc:383] T cd3092c89c74475a999f5b3a6d77e652 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.087571 27692 raft_consensus.cc:738] T cd3092c89c74475a999f5b3a6d77e652 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.087826 27692 consensus_queue.cc:260] T cd3092c89c74475a999f5b3a6d77e652 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.088289 27695 raft_consensus.cc:357] T 43f88abc8f3640a48598c07c3d76755d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.088490 27695 raft_consensus.cc:383] T 43f88abc8f3640a48598c07c3d76755d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.088660 27695 raft_consensus.cc:738] T 43f88abc8f3640a48598c07c3d76755d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.088907 27695 consensus_queue.cc:260] T 43f88abc8f3640a48598c07c3d76755d P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.089479 27695 ts_tablet_manager.cc:1428] T 43f88abc8f3640a48598c07c3d76755d P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:10.089864 27695 tablet_bootstrap.cc:492] T d1e1add08013459ba0dab3b7e66f0136 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:10.091816 27695 tablet_bootstrap.cc:654] T d1e1add08013459ba0dab3b7e66f0136 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:10.093459 27685 ts_tablet_manager.cc:1918] T 9d37a5d51a184c76bb8496679dfe3991 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.093619 27685 log.cc:1199] T 9d37a5d51a184c76bb8496679dfe3991 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/9d37a5d51a184c76bb8496679dfe3991
I20250903 20:05:10.094028 27685 ts_tablet_manager.cc:1939] T 9d37a5d51a184c76bb8496679dfe3991 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
W20250903 20:05:10.095768 27753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 20:05:10.102510 27693 ts_tablet_manager.cc:1428] T 07361048416c43b6bc43d8737906db99 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20250903 20:05:10.104218 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 16) took 1426 ms. Trace:
I20250903 20:05:10.104347 27685 rpcz_store.cc:276] 0903 20:05:08.677374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:08.677648 (+ 274us) service_pool.cc:225] Handling call
0903 20:05:10.082569 (+1404921us) ts_tablet_manager.cc:1145] Deleting tablet 9d37a5d51a184c76bb8496679dfe3991
0903 20:05:10.082579 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:10.087755 (+ 5176us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:10.093317 (+ 5562us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:10.104137 (+ 10820us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1404597}
I20250903 20:05:10.104900 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 9d37a5d51a184c76bb8496679dfe3991 (table default.Table_e914faf335d44d019f91bda71758c473 [id=374b9abdf6a54b38876a96d8ec4a0368]) successfully deleted
I20250903 20:05:10.105803 27685 tablet_replica.cc:331] T 0bcfca89bc3b44d78bc1b875c796329b P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:10.106016 27685 raft_consensus.cc:2241] T 0bcfca89bc3b44d78bc1b875c796329b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.106228 27685 raft_consensus.cc:2270] T 0bcfca89bc3b44d78bc1b875c796329b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.106932 27685 ts_tablet_manager.cc:1905] T 0bcfca89bc3b44d78bc1b875c796329b P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.107576 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet 43f88abc8f3640a48598c07c3d76755d with delete_type TABLET_DATA_DELETED (replaced by tablet 68cff0e416a54a5eb37071f5a12f07ab) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:10.108457 27695 tablet_bootstrap.cc:492] T d1e1add08013459ba0dab3b7e66f0136 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:10.108671 27695 ts_tablet_manager.cc:1397] T d1e1add08013459ba0dab3b7e66f0136 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.003s
I20250903 20:05:10.109766 27695 raft_consensus.cc:357] T d1e1add08013459ba0dab3b7e66f0136 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.109998 27695 raft_consensus.cc:383] T d1e1add08013459ba0dab3b7e66f0136 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.110078 27695 raft_consensus.cc:738] T d1e1add08013459ba0dab3b7e66f0136 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
W20250903 20:05:10.110620 27757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 20:05:10.110989 27695 consensus_queue.cc:260] T d1e1add08013459ba0dab3b7e66f0136 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.111974 27693 tablet_bootstrap.cc:492] T d08c9e1e7d564886a90f6c056794becc P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:10.112979 27695 ts_tablet_manager.cc:1428] T d1e1add08013459ba0dab3b7e66f0136 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20250903 20:05:10.115641 27695 tablet_bootstrap.cc:492] T 7e0810c96eb24b8ab4278038801be245 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:10.117898 27695 tablet_bootstrap.cc:654] T 7e0810c96eb24b8ab4278038801be245 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:10.133061 27695 tablet_bootstrap.cc:492] T 7e0810c96eb24b8ab4278038801be245 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:10.133256 27695 ts_tablet_manager.cc:1397] T 7e0810c96eb24b8ab4278038801be245 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.012s
I20250903 20:05:10.133790 27693 tablet_bootstrap.cc:654] T d08c9e1e7d564886a90f6c056794becc P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:10.134550 27692 ts_tablet_manager.cc:1428] T cd3092c89c74475a999f5b3a6d77e652 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.048s user 0.006s sys 0.004s
I20250903 20:05:10.134860 27695 raft_consensus.cc:357] T 7e0810c96eb24b8ab4278038801be245 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.135051 27695 raft_consensus.cc:383] T 7e0810c96eb24b8ab4278038801be245 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.135150 27695 raft_consensus.cc:738] T 7e0810c96eb24b8ab4278038801be245 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.135375 27695 consensus_queue.cc:260] T 7e0810c96eb24b8ab4278038801be245 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.135879 27695 ts_tablet_manager.cc:1428] T 7e0810c96eb24b8ab4278038801be245 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:10.136633 27692 tablet_bootstrap.cc:492] T 07361048416c43b6bc43d8737906db99 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:10.137472 27695 tablet_bootstrap.cc:492] T 1d9a2508437b41f08821a29f58163159 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:10.139173 27695 tablet_bootstrap.cc:654] T 1d9a2508437b41f08821a29f58163159 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:10.140285 27502 tablet_service.cc:1468] Processing CreateTablet for tablet 46b5d491888147bbb101963641fd05f8 (DEFAULT_TABLE table=default.Table_b4bd57e0d5384edaa027cfcdf1b1db23 [id=3b241b63c32f42c0966e9af3dc06d701]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.140715 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46b5d491888147bbb101963641fd05f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.147972 27503 tablet_service.cc:1468] Processing CreateTablet for tablet c0343d2226e345669d42375de2545986 (DEFAULT_TABLE table=default.Table_79300b77de8c404a9e85783ce977861c [id=7d6d5a3d8906427e83ab3153b7bb4235]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.148554 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0343d2226e345669d42375de2545986. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:10.152112 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 12a7be47c9c2406596c9e00800074a7c consensus state has no leader
I20250903 20:05:10.152168 27685 ts_tablet_manager.cc:1918] T 0bcfca89bc3b44d78bc1b875c796329b P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.152350 27685 log.cc:1199] T 0bcfca89bc3b44d78bc1b875c796329b P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/0bcfca89bc3b44d78bc1b875c796329b
I20250903 20:05:10.152791 27685 ts_tablet_manager.cc:1939] T 0bcfca89bc3b44d78bc1b875c796329b P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
W20250903 20:05:10.153013 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0f4ef613b0ab4bca98294682b934f6ed consensus state has no leader
I20250903 20:05:10.153753 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 17) took 1476 ms. Trace:
I20250903 20:05:10.153927 27685 rpcz_store.cc:276] 0903 20:05:08.677582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:08.678018 (+ 436us) service_pool.cc:225] Handling call
0903 20:05:10.105213 (+1427195us) ts_tablet_manager.cc:1145] Deleting tablet 0bcfca89bc3b44d78bc1b875c796329b
0903 20:05:10.105704 (+ 491us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:10.111039 (+ 5335us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:10.152032 (+ 40993us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:10.153711 (+ 1679us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1426940}
I20250903 20:05:10.154810 27685 tablet_replica.cc:331] T 806d4fa868224194a0cf05e81b0ff377 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:10.155093 27685 raft_consensus.cc:2241] T 806d4fa868224194a0cf05e81b0ff377 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.155300 27685 raft_consensus.cc:2270] T 806d4fa868224194a0cf05e81b0ff377 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.155469 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 43f88abc8f3640a48598c07c3d76755d with delete_type TABLET_DATA_DELETED (replaced by tablet 68cff0e416a54a5eb37071f5a12f07ab) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:10.155982 27685 ts_tablet_manager.cc:1905] T 806d4fa868224194a0cf05e81b0ff377 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.161216 27692 tablet_bootstrap.cc:654] T 07361048416c43b6bc43d8737906db99 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:10.162370 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 43f88abc8f3640a48598c07c3d76755d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43f88abc8f3640a48598c07c3d76755d already in progress: creating tablet
I20250903 20:05:10.168706 27719 raft_consensus.cc:491] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:10.168948 27719 raft_consensus.cc:513] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.169788 27719 leader_election.cc:290] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:10.171895 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8a293c6fa034e9fb29682439cb3bcc2" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:10.172458 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8a293c6fa034e9fb29682439cb3bcc2" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:10.172722 27685 ts_tablet_manager.cc:1918] T 806d4fa868224194a0cf05e81b0ff377 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.172891 27685 log.cc:1199] T 806d4fa868224194a0cf05e81b0ff377 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/806d4fa868224194a0cf05e81b0ff377
I20250903 20:05:10.173015 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet a45604ad611e418e84dab793e04101a0
I20250903 20:05:10.173323 27685 ts_tablet_manager.cc:1939] T 806d4fa868224194a0cf05e81b0ff377 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:10.174331 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 22) took 1389 ms. Trace:
I20250903 20:05:10.174451 27685 rpcz_store.cc:276] 0903 20:05:08.784681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:08.785358 (+ 677us) service_pool.cc:225] Handling call
0903 20:05:10.154726 (+1369368us) ts_tablet_manager.cc:1145] Deleting tablet 806d4fa868224194a0cf05e81b0ff377
0903 20:05:10.154741 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:10.159968 (+ 5227us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:10.172572 (+ 12604us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:10.174207 (+ 1635us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1369009}
I20250903 20:05:10.174960 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 806d4fa868224194a0cf05e81b0ff377 (table default.Table_6106b8f9be34419983b266581e83bed8 [id=de7efd68262248f38a3d8d4a8d6c87db]) successfully deleted
W20250903 20:05:10.175472 27183 leader_election.cc:343] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:10.175700 27184 leader_election.cc:343] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:10.175832 27184 leader_election.cc:304] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:10.176303 27719 raft_consensus.cc:2747] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:10.176340 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0a266b1bbf1540ac86d49c56df2dd4ad consensus state has no leader
I20250903 20:05:10.177254 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet a45604ad611e418e84dab793e04101a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:10.177328 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet a45604ad611e418e84dab793e04101a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:10.178735 27692 tablet_bootstrap.cc:492] T 07361048416c43b6bc43d8737906db99 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
W20250903 20:05:10.178856 27066 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4692dd93b76c42c4883c7e22c6032250 consensus state has no leader
I20250903 20:05:10.178910 27692 ts_tablet_manager.cc:1397] T 07361048416c43b6bc43d8737906db99 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.001s
W20250903 20:05:10.179478 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet a45604ad611e418e84dab793e04101a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a45604ad611e418e84dab793e04101a0 already in progress: creating tablet
I20250903 20:05:10.179733 27692 raft_consensus.cc:357] T 07361048416c43b6bc43d8737906db99 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.179951 27692 raft_consensus.cc:383] T 07361048416c43b6bc43d8737906db99 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.180030 27692 raft_consensus.cc:738] T 07361048416c43b6bc43d8737906db99 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
W20250903 20:05:10.179967 27767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 20:05:10.180295 27692 consensus_queue.cc:260] T 07361048416c43b6bc43d8737906db99 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.180768 27692 ts_tablet_manager.cc:1428] T 07361048416c43b6bc43d8737906db99 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:10.180925 27060 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet fca7fe0dd2334f52bbba73c75ff77d3f
I20250903 20:05:10.181321 27692 tablet_bootstrap.cc:492] T 806d4fa868224194a0cf05e81b0ff377 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:10.181008 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 35) took 1049 ms. Trace:
I20250903 20:05:10.181927 27685 rpcz_store.cc:276] 0903 20:05:09.131198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:09.131623 (+ 425us) service_pool.cc:225] Handling call
0903 20:05:10.180683 (+1049060us) ts_tablet_manager.cc:1145] Deleting tablet 1d9a2508437b41f08821a29f58163159
0903 20:05:10.180699 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:10.180938 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1041347}
W20250903 20:05:10.182601 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 1d9a2508437b41f08821a29f58163159 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d9a2508437b41f08821a29f58163159 already in progress: creating tablet
I20250903 20:05:10.183015 27685 tablet_replica.cc:331] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:10.183264 27685 raft_consensus.cc:2241] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.183481 27685 raft_consensus.cc:2270] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.184369 27685 ts_tablet_manager.cc:1905] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.184628 27719 raft_consensus.cc:491] T 12a7be47c9c2406596c9e00800074a7c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:10.184784 27719 raft_consensus.cc:513] T 12a7be47c9c2406596c9e00800074a7c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.185560 27719 leader_election.cc:290] T 12a7be47c9c2406596c9e00800074a7c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:10.187052 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "12a7be47c9c2406596c9e00800074a7c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:10.187669 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "12a7be47c9c2406596c9e00800074a7c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:10.188548 27183 leader_election.cc:343] T 12a7be47c9c2406596c9e00800074a7c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:10.189234 27184 leader_election.cc:343] T 12a7be47c9c2406596c9e00800074a7c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:10.189404 27184 leader_election.cc:304] T 12a7be47c9c2406596c9e00800074a7c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:10.190011 27719 raft_consensus.cc:2747] T 12a7be47c9c2406596c9e00800074a7c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:10.194929 27685 ts_tablet_manager.cc:1918] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.195066 27685 log.cc:1199] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/e0d1b29d7f1e422e8bb5b8bbc33423e8
I20250903 20:05:10.195467 27685 ts_tablet_manager.cc:1939] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:10.196806 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet e0d1b29d7f1e422e8bb5b8bbc33423e8 (table default.Table_82f29a4a80194976bc7c8f864ec877a3 [id=49551de884e7488fb6db7249f834e12f]) successfully deleted
W20250903 20:05:10.196946 27059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d8ed7b3e292241cb9525745a797ff13a has no consensus state
I20250903 20:05:10.197469 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet a45604ad611e418e84dab793e04101a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:10.197425 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_c79b97a3bb294caf96bb5fa0d144628d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:10.198067 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c79b97a3bb294caf96bb5fa0d144628d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:10.198308 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet c12c957a36f643d9bfb60b755619cbbc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c12c957a36f643d9bfb60b755619cbbc already in progress: creating tablet
W20250903 20:05:10.199018 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 0062307c6ee54aada74a08cf0d087f17 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0062307c6ee54aada74a08cf0d087f17 already in progress: creating tablet
I20250903 20:05:10.199479 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2ace21da933742e69b8ec85acbd919c2" } modify_external_catalogs: true
I20250903 20:05:10.199656 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_6d92c85ead9f4f58855e725b7823c449"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:10.199995 27685 tablet_replica.cc:331] T cd3092c89c74475a999f5b3a6d77e652 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:10.200282 27685 raft_consensus.cc:2241] T cd3092c89c74475a999f5b3a6d77e652 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.200491 27685 raft_consensus.cc:2270] T cd3092c89c74475a999f5b3a6d77e652 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.200974 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet 07361048416c43b6bc43d8737906db99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:10.201184 27685 ts_tablet_manager.cc:1905] T cd3092c89c74475a999f5b3a6d77e652 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.201228 27687 tablet_replica.cc:331] T 07361048416c43b6bc43d8737906db99 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:10.201439 27687 raft_consensus.cc:2241] T 07361048416c43b6bc43d8737906db99 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.201618 27687 raft_consensus.cc:2270] T 07361048416c43b6bc43d8737906db99 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.202350 27692 tablet_bootstrap.cc:654] T 806d4fa868224194a0cf05e81b0ff377 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:10.203450 27687 ts_tablet_manager.cc:1905] T 07361048416c43b6bc43d8737906db99 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.199676 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2ace21da933742e69b8ec85acbd919c2" } modify_external_catalogs: true
I20250903 20:05:10.207448 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_f4196ed1d25c4d6fab512014504c51af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:10.208243 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4196ed1d25c4d6fab512014504c51af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.209288 27692 tablet_bootstrap.cc:492] T 806d4fa868224194a0cf05e81b0ff377 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:10.209455 27692 ts_tablet_manager.cc:1397] T 806d4fa868224194a0cf05e81b0ff377 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.000s
I20250903 20:05:10.209895 27685 ts_tablet_manager.cc:1918] T cd3092c89c74475a999f5b3a6d77e652 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.210083 27685 log.cc:1199] T cd3092c89c74475a999f5b3a6d77e652 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/cd3092c89c74475a999f5b3a6d77e652
I20250903 20:05:10.210548 27685 ts_tablet_manager.cc:1939] T cd3092c89c74475a999f5b3a6d77e652 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:10.210451 27692 raft_consensus.cc:357] T 806d4fa868224194a0cf05e81b0ff377 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.210680 27692 raft_consensus.cc:383] T 806d4fa868224194a0cf05e81b0ff377 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.210765 27692 raft_consensus.cc:738] T 806d4fa868224194a0cf05e81b0ff377 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.210919 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5f1e055d858e4d31bce98ac57741cc30" } modify_external_catalogs: true
I20250903 20:05:10.210997 27692 consensus_queue.cc:260] T 806d4fa868224194a0cf05e81b0ff377 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.211094 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5f1e055d858e4d31bce98ac57741cc30" } modify_external_catalogs: true
I20250903 20:05:10.211560 27692 ts_tablet_manager.cc:1428] T 806d4fa868224194a0cf05e81b0ff377 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250903 20:05:10.211911 27685 tablet_replica.cc:331] T 07361048416c43b6bc43d8737906db99 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:10.212150 27685 raft_consensus.cc:2241] T 07361048416c43b6bc43d8737906db99 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.212361 27685 raft_consensus.cc:2270] T 07361048416c43b6bc43d8737906db99 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.213095 27685 ts_tablet_manager.cc:1905] T 07361048416c43b6bc43d8737906db99 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.214035 27692 tablet_bootstrap.cc:492] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
W20250903 20:05:10.202054 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d92c85ead9f4f58855e725b7823c449 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.216465 27065 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1a5c8c3256c24c7f8ef8597efa53dde5" } modify_external_catalogs: true
I20250903 20:05:10.218137 27065 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1a5c8c3256c24c7f8ef8597efa53dde5" } modify_external_catalogs: true
I20250903 20:05:10.221010 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet cd3092c89c74475a999f5b3a6d77e652 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:10.221360 27684 tablet_replica.cc:331] T cd3092c89c74475a999f5b3a6d77e652 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:10.221576 27684 raft_consensus.cc:2241] T cd3092c89c74475a999f5b3a6d77e652 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
W20250903 20:05:10.206761 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet a45604ad611e418e84dab793e04101a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a45604ad611e418e84dab793e04101a0 already in progress: creating tablet
I20250903 20:05:10.221897 27692 tablet_bootstrap.cc:654] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:10.222191 27684 raft_consensus.cc:2270] T cd3092c89c74475a999f5b3a6d77e652 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.223068 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_43bea7dca99143f5ac13671739868cdb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:10.223781 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43bea7dca99143f5ac13671739868cdb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.224135 27684 ts_tablet_manager.cc:1905] T cd3092c89c74475a999f5b3a6d77e652 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.225327 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet cd3092c89c74475a999f5b3a6d77e652 (table default.Table_60698087f8f34bbe86a9994a117f2b37 [id=0523254ee07b4c5695b05a486945cb66]) successfully deleted
I20250903 20:05:10.230921 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet 07361048416c43b6bc43d8737906db99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:10.231367 27503 tablet_service.cc:1515] Processing DeleteTablet for tablet 806d4fa868224194a0cf05e81b0ff377 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:10.232458 27684 ts_tablet_manager.cc:1918] T cd3092c89c74475a999f5b3a6d77e652 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.232596 27684 log.cc:1199] T cd3092c89c74475a999f5b3a6d77e652 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/cd3092c89c74475a999f5b3a6d77e652
I20250903 20:05:10.206694 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_14af7742726b4562a02fd2c9b2895277"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:10.232990 27684 ts_tablet_manager.cc:1939] T cd3092c89c74475a999f5b3a6d77e652 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
W20250903 20:05:10.233199 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14af7742726b4562a02fd2c9b2895277 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.236021 27685 ts_tablet_manager.cc:1918] T 07361048416c43b6bc43d8737906db99 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.236181 27685 log.cc:1199] T 07361048416c43b6bc43d8737906db99 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/07361048416c43b6bc43d8737906db99
I20250903 20:05:10.236168 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet cd3092c89c74475a999f5b3a6d77e652 (table default.Table_60698087f8f34bbe86a9994a117f2b37 [id=0523254ee07b4c5695b05a486945cb66]) successfully deleted
I20250903 20:05:10.236636 27685 ts_tablet_manager.cc:1939] T 07361048416c43b6bc43d8737906db99 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:10.237282 27362 tablet_service.cc:1468] Processing CreateTablet for tablet 46e3a65118b744298b5536f88c22f1db (DEFAULT_TABLE table=default.Table_e1991ea3a3d14b85a4ce9160fa00bc35 [id=decb9fcf2ee64971ba8bcd8326736852]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.237737 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46e3a65118b744298b5536f88c22f1db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.240576 27685 tablet_replica.cc:331] T 943146f84c4a4adf89c5c2e91f9f04d4 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:10.240859 27685 raft_consensus.cc:2241] T 943146f84c4a4adf89c5c2e91f9f04d4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.241055 27685 raft_consensus.cc:2270] T 943146f84c4a4adf89c5c2e91f9f04d4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.241180 27684 tablet_replica.cc:331] T 07361048416c43b6bc43d8737906db99 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:10.241406 27684 raft_consensus.cc:2241] T 07361048416c43b6bc43d8737906db99 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.241572 27684 raft_consensus.cc:2270] T 07361048416c43b6bc43d8737906db99 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.241729 27685 ts_tablet_manager.cc:1905] T 943146f84c4a4adf89c5c2e91f9f04d4 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.242697 27684 ts_tablet_manager.cc:1905] T 07361048416c43b6bc43d8737906db99 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.246043 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_9d09f11dcb4b4bf09c870934bf1b6acd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:10.249882 27685 ts_tablet_manager.cc:1918] T 943146f84c4a4adf89c5c2e91f9f04d4 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.250063 27685 log.cc:1199] T 943146f84c4a4adf89c5c2e91f9f04d4 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/943146f84c4a4adf89c5c2e91f9f04d4
I20250903 20:05:10.250486 27685 ts_tablet_manager.cc:1939] T 943146f84c4a4adf89c5c2e91f9f04d4 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:10.251843 27685 tablet_replica.cc:331] T d1e1add08013459ba0dab3b7e66f0136 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:10.252101 27685 raft_consensus.cc:2241] T d1e1add08013459ba0dab3b7e66f0136 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.252220 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 07361048416c43b6bc43d8737906db99 (table default.Table_260dadc90c8e415c8dee534464e23000 [id=9cea308979eb4c7f9deaa02e1c8feee9]) successfully deleted
I20250903 20:05:10.252327 27685 raft_consensus.cc:2270] T d1e1add08013459ba0dab3b7e66f0136 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.253269 27685 ts_tablet_manager.cc:1905] T d1e1add08013459ba0dab3b7e66f0136 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.254925 27684 ts_tablet_manager.cc:1918] T 07361048416c43b6bc43d8737906db99 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.255096 27684 log.cc:1199] T 07361048416c43b6bc43d8737906db99 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/07361048416c43b6bc43d8737906db99
W20250903 20:05:10.255160 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d09f11dcb4b4bf09c870934bf1b6acd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.255487 27684 ts_tablet_manager.cc:1939] T 07361048416c43b6bc43d8737906db99 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:10.258507 27692 tablet_bootstrap.cc:492] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:10.258682 27692 ts_tablet_manager.cc:1397] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.045s user 0.009s sys 0.000s
I20250903 20:05:10.258744 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 46e3a65118b744298b5536f88c22f1db (DEFAULT_TABLE table=default.Table_e1991ea3a3d14b85a4ce9160fa00bc35 [id=decb9fcf2ee64971ba8bcd8326736852]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.259205 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46e3a65118b744298b5536f88c22f1db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.259479 27692 raft_consensus.cc:357] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.259685 27692 raft_consensus.cc:383] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.259752 27692 raft_consensus.cc:738] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.260502 27692 consensus_queue.cc:260] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.263214 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_64c39ce0250747edacd69ba845d72455"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:10.264288 27692 ts_tablet_manager.cc:1428] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
W20250903 20:05:10.264745 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64c39ce0250747edacd69ba845d72455 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.264603 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_afc0f683c5d8460484160a3a9c3e6df5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:10.265264 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_afc0f683c5d8460484160a3a9c3e6df5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.266671 27363 tablet_service.cc:1468] Processing CreateTablet for tablet 710e3bc3c4194efbbeef6bc76ee64f24 (DEFAULT_TABLE table=default.Table_2a55e0f3a78d422ab61df9f9ddb6515b [id=d945a830c86244bca66cb2652e64966d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.267160 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 710e3bc3c4194efbbeef6bc76ee64f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.267391 27692 tablet_bootstrap.cc:492] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:10.269385 27692 tablet_bootstrap.cc:654] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:10.269811 27358 tablet_service.cc:1468] Processing CreateTablet for tablet d8ed7b3e292241cb9525745a797ff13a (DEFAULT_TABLE table=default.Table_646dcf6e43a24b31aaa79d2f93c58a14 [id=ad8b3a9bedef4c188b89640ab5e2473a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.270320 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ed7b3e292241cb9525745a797ff13a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.270383 27684 tablet_replica.cc:331] T 806d4fa868224194a0cf05e81b0ff377 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:10.270550 27684 raft_consensus.cc:2241] T 806d4fa868224194a0cf05e81b0ff377 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.270735 27684 raft_consensus.cc:2270] T 806d4fa868224194a0cf05e81b0ff377 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.271536 27684 ts_tablet_manager.cc:1905] T 806d4fa868224194a0cf05e81b0ff377 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.271507 27217 tablet_service.cc:1468] Processing CreateTablet for tablet d8ed7b3e292241cb9525745a797ff13a (DEFAULT_TABLE table=default.Table_646dcf6e43a24b31aaa79d2f93c58a14 [id=ad8b3a9bedef4c188b89640ab5e2473a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.271932 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ed7b3e292241cb9525745a797ff13a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.272464 27366 tablet_service.cc:1468] Processing CreateTablet for tablet 76742d0df44c43729197f7e09f96cad8 (DEFAULT_TABLE table=default.Table_3055928e679e4dd8b8559b363b580e13 [id=9545d6a8d12e4937bb2b4cdbfc6933de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.272879 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76742d0df44c43729197f7e09f96cad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.274533 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 710e3bc3c4194efbbeef6bc76ee64f24 (DEFAULT_TABLE table=default.Table_2a55e0f3a78d422ab61df9f9ddb6515b [id=d945a830c86244bca66cb2652e64966d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.274863 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 710e3bc3c4194efbbeef6bc76ee64f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.276130 27220 tablet_service.cc:1468] Processing CreateTablet for tablet 76742d0df44c43729197f7e09f96cad8 (DEFAULT_TABLE table=default.Table_3055928e679e4dd8b8559b363b580e13 [id=9545d6a8d12e4937bb2b4cdbfc6933de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.276572 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76742d0df44c43729197f7e09f96cad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.278381 27719 raft_consensus.cc:491] T 7e0810c96eb24b8ab4278038801be245 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:10.278512 27719 raft_consensus.cc:513] T 7e0810c96eb24b8ab4278038801be245 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.279317 27719 leader_election.cc:290] T 7e0810c96eb24b8ab4278038801be245 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:10.279680 27684 ts_tablet_manager.cc:1918] T 806d4fa868224194a0cf05e81b0ff377 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.279789 27684 log.cc:1199] T 806d4fa868224194a0cf05e81b0ff377 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/806d4fa868224194a0cf05e81b0ff377
I20250903 20:05:10.280140 27684 ts_tablet_manager.cc:1939] T 806d4fa868224194a0cf05e81b0ff377 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:10.281366 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e0810c96eb24b8ab4278038801be245" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:10.281682 27060 master_service.cc:948] ReplaceTablet: received request to replace tablet e722212b281a405fb08908df0a42d864 from {username='slave'} at 127.0.0.1:45936
W20250903 20:05:10.282065 27183 leader_election.cc:343] T 7e0810c96eb24b8ab4278038801be245 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:10.293783 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 9509364ba3ec4e9bad09db09406a978a (DEFAULT_TABLE table=default.Table_03ce0e151fcc4259ba52504c9c5ddfcd [id=f261e84232234244824d0232efbb4399]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.294325 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e0810c96eb24b8ab4278038801be245" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:10.295147 27184 leader_election.cc:343] T 7e0810c96eb24b8ab4278038801be245 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:10.295329 27184 leader_election.cc:304] T 7e0810c96eb24b8ab4278038801be245 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:10.295799 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9509364ba3ec4e9bad09db09406a978a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.298355 27719 raft_consensus.cc:2747] T 7e0810c96eb24b8ab4278038801be245 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:10.298960 27220 tablet_service.cc:1468] Processing CreateTablet for tablet e5136d5388bc4c2ba71e09be40e8f412 (DEFAULT_TABLE table=default.Table_db7d825a12ef40739b70f11044170c72 [id=790f57164c7d4183a3df17ba7f230d24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.299507 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5136d5388bc4c2ba71e09be40e8f412. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.303879 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet e0d1b29d7f1e422e8bb5b8bbc33423e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:10.305641 27685 ts_tablet_manager.cc:1918] T d1e1add08013459ba0dab3b7e66f0136 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.305761 27685 log.cc:1199] T d1e1add08013459ba0dab3b7e66f0136 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/d1e1add08013459ba0dab3b7e66f0136
I20250903 20:05:10.306123 27685 ts_tablet_manager.cc:1939] T d1e1add08013459ba0dab3b7e66f0136 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:10.308074 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet d1e1add08013459ba0dab3b7e66f0136 (table default.Table_4ace9616711f4778a94a0827ff79cc07 [id=e20fa37b0e0343ed8ab3a7e814358ae6]) successfully deleted
I20250903 20:05:10.308379 27685 tablet_replica.cc:331] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
W20250903 20:05:10.308387 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 59e120e5e39e4de8a239f13545e4b1af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59e120e5e39e4de8a239f13545e4b1af already in progress: creating tablet
W20250903 20:05:10.308619 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet fb76ba1d7cd54d6aa69d06a9a5618cd3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb76ba1d7cd54d6aa69d06a9a5618cd3 already in progress: creating tablet
I20250903 20:05:10.308580 27685 raft_consensus.cc:2241] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:10.310128 27685 pending_rounds.cc:70] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a: Trying to abort 1 pending ops.
I20250903 20:05:10.310240 27685 pending_rounds.cc:77] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384909922762752 op_type: NO_OP noop_request { }
I20250903 20:05:10.310366 27685 raft_consensus.cc:2887] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:10.310479 27685 raft_consensus.cc:2270] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.306195 27693 tablet_bootstrap.cc:492] T d08c9e1e7d564886a90f6c056794becc P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:10.311946 27685 ts_tablet_manager.cc:1905] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.312088 27693 ts_tablet_manager.cc:1397] T d08c9e1e7d564886a90f6c056794becc P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.200s user 0.000s sys 0.006s
I20250903 20:05:10.313176 27693 raft_consensus.cc:357] T d08c9e1e7d564886a90f6c056794becc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.313985 27717 server_base.cc:1047] running on GCE node
I20250903 20:05:10.317411 27225 tablet_service.cc:1468] Processing CreateTablet for tablet c25aaa6ff310432bb27a0695394ec420 (DEFAULT_TABLE table=default.Table_835ced1f8b194192809c3b6fbf4d1d96 [id=23f448924ec04ef08a541a14bd2ce5df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.317848 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c25aaa6ff310432bb27a0695394ec420. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.318517 27695 tablet_bootstrap.cc:492] T 1d9a2508437b41f08821a29f58163159 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:10.318838 27695 ts_tablet_manager.cc:1397] T 1d9a2508437b41f08821a29f58163159 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.181s user 0.018s sys 0.061s
I20250903 20:05:10.320093 27695 raft_consensus.cc:357] T 1d9a2508437b41f08821a29f58163159 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:10.320847 27693 raft_consensus.cc:383] T d08c9e1e7d564886a90f6c056794becc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.311744 27360 tablet_service.cc:1468] Processing CreateTablet for tablet c25aaa6ff310432bb27a0695394ec420 (DEFAULT_TABLE table=default.Table_835ced1f8b194192809c3b6fbf4d1d96 [id=23f448924ec04ef08a541a14bd2ce5df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.321256 27693 raft_consensus.cc:738] T d08c9e1e7d564886a90f6c056794becc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.321565 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c25aaa6ff310432bb27a0695394ec420. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.321594 27693 consensus_queue.cc:260] T d08c9e1e7d564886a90f6c056794becc P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.308781 27687 ts_tablet_manager.cc:1918] T 07361048416c43b6bc43d8737906db99 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.321987 27687 log.cc:1199] T 07361048416c43b6bc43d8737906db99 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/07361048416c43b6bc43d8737906db99
I20250903 20:05:10.322618 27687 ts_tablet_manager.cc:1939] T 07361048416c43b6bc43d8737906db99 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:10.323046 27362 tablet_service.cc:1468] Processing CreateTablet for tablet 8945085bdc174e6db22a5a9477810297 (DEFAULT_TABLE table=default.Table_c9766617998c49b199a79052643d496a [id=6f7fece7a8d54d4786f1bab10da9129c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.323529 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8945085bdc174e6db22a5a9477810297. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.323881 27695 raft_consensus.cc:383] T 1d9a2508437b41f08821a29f58163159 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.324010 27695 raft_consensus.cc:738] T 1d9a2508437b41f08821a29f58163159 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.324317 27695 consensus_queue.cc:260] T 1d9a2508437b41f08821a29f58163159 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:10.325019 27695 ts_tablet_manager.cc:1428] T 1d9a2508437b41f08821a29f58163159 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250903 20:05:10.329341 27364 tablet_service.cc:1468] Processing CreateTablet for tablet 9509364ba3ec4e9bad09db09406a978a (DEFAULT_TABLE table=default.Table_03ce0e151fcc4259ba52504c9c5ddfcd [id=f261e84232234244824d0232efbb4399]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.329864 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 925c0ab4873041da82702cd9377ec7bc (DEFAULT_TABLE table=default.Table_1a5c8c3256c24c7f8ef8597efa53dde5 [id=7caef592f80a4d6cb6df4e2c3971de49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.330345 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 925c0ab4873041da82702cd9377ec7bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.331883 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 07361048416c43b6bc43d8737906db99 (table default.Table_260dadc90c8e415c8dee534464e23000 [id=9cea308979eb4c7f9deaa02e1c8feee9]) successfully deleted
I20250903 20:05:10.332679 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9509364ba3ec4e9bad09db09406a978a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:10.333031 27051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.125:44059, user_credentials={real_user=slave}} blocked reactor thread for 92157.2us
I20250903 20:05:10.333290 27366 tablet_service.cc:1468] Processing CreateTablet for tablet e5136d5388bc4c2ba71e09be40e8f412 (DEFAULT_TABLE table=default.Table_db7d825a12ef40739b70f11044170c72 [id=790f57164c7d4183a3df17ba7f230d24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.333665 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5136d5388bc4c2ba71e09be40e8f412. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.334676 27503 tablet_service.cc:1468] Processing CreateTablet for tablet 46e3a65118b744298b5536f88c22f1db (DEFAULT_TABLE table=default.Table_e1991ea3a3d14b85a4ce9160fa00bc35 [id=decb9fcf2ee64971ba8bcd8326736852]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.334932 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 07361048416c43b6bc43d8737906db99 (table default.Table_260dadc90c8e415c8dee534464e23000 [id=9cea308979eb4c7f9deaa02e1c8feee9]) successfully deleted
I20250903 20:05:10.335119 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46e3a65118b744298b5536f88c22f1db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.335201 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 806d4fa868224194a0cf05e81b0ff377 (table default.Table_6106b8f9be34419983b266581e83bed8 [id=de7efd68262248f38a3d8d4a8d6c87db]) successfully deleted
I20250903 20:05:10.336090 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 925c0ab4873041da82702cd9377ec7bc (DEFAULT_TABLE table=default.Table_1a5c8c3256c24c7f8ef8597efa53dde5 [id=7caef592f80a4d6cb6df4e2c3971de49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.336474 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 925c0ab4873041da82702cd9377ec7bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.338550 27502 tablet_service.cc:1468] Processing CreateTablet for tablet 710e3bc3c4194efbbeef6bc76ee64f24 (DEFAULT_TABLE table=default.Table_2a55e0f3a78d422ab61df9f9ddb6515b [id=d945a830c86244bca66cb2652e64966d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.338979 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 710e3bc3c4194efbbeef6bc76ee64f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.339286 27695 tablet_bootstrap.cc:492] T e722212b281a405fb08908df0a42d864 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:10.341025 27695 tablet_bootstrap.cc:654] T e722212b281a405fb08908df0a42d864 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:10.343550 27226 tablet_service.cc:1468] Processing CreateTablet for tablet 8945085bdc174e6db22a5a9477810297 (DEFAULT_TABLE table=default.Table_c9766617998c49b199a79052643d496a [id=6f7fece7a8d54d4786f1bab10da9129c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.344065 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8945085bdc174e6db22a5a9477810297. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.348819 27504 tablet_service.cc:1468] Processing CreateTablet for tablet d8ed7b3e292241cb9525745a797ff13a (DEFAULT_TABLE table=default.Table_646dcf6e43a24b31aaa79d2f93c58a14 [id=ad8b3a9bedef4c188b89640ab5e2473a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.349375 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ed7b3e292241cb9525745a797ff13a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.349359 27717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 20:05:10.350692 27501 tablet_service.cc:1468] Processing CreateTablet for tablet 9509364ba3ec4e9bad09db09406a978a (DEFAULT_TABLE table=default.Table_03ce0e151fcc4259ba52504c9c5ddfcd [id=f261e84232234244824d0232efbb4399]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.351287 27501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9509364ba3ec4e9bad09db09406a978a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.351482 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 76742d0df44c43729197f7e09f96cad8 (DEFAULT_TABLE table=default.Table_3055928e679e4dd8b8559b363b580e13 [id=9545d6a8d12e4937bb2b4cdbfc6933de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.351825 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76742d0df44c43729197f7e09f96cad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.354578 27506 tablet_service.cc:1468] Processing CreateTablet for tablet e5136d5388bc4c2ba71e09be40e8f412 (DEFAULT_TABLE table=default.Table_db7d825a12ef40739b70f11044170c72 [id=790f57164c7d4183a3df17ba7f230d24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.354667 27500 tablet_service.cc:1468] Processing CreateTablet for tablet c25aaa6ff310432bb27a0695394ec420 (DEFAULT_TABLE table=default.Table_835ced1f8b194192809c3b6fbf4d1d96 [id=23f448924ec04ef08a541a14bd2ce5df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.355073 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5136d5388bc4c2ba71e09be40e8f412. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.358655 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 8945085bdc174e6db22a5a9477810297 (DEFAULT_TABLE table=default.Table_c9766617998c49b199a79052643d496a [id=6f7fece7a8d54d4786f1bab10da9129c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.359370 27499 tablet_service.cc:1468] Processing CreateTablet for tablet 925c0ab4873041da82702cd9377ec7bc (DEFAULT_TABLE table=default.Table_1a5c8c3256c24c7f8ef8597efa53dde5 [id=7caef592f80a4d6cb6df4e2c3971de49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.361387 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8945085bdc174e6db22a5a9477810297. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.362233 27685 ts_tablet_manager.cc:1918] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:10.362380 27685 log.cc:1199] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/161f2c2afc2f4c82b2025f7daf460bd6
I20250903 20:05:10.362882 27685 ts_tablet_manager.cc:1939] T 161f2c2afc2f4c82b2025f7daf460bd6 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:10.362938 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 925c0ab4873041da82702cd9377ec7bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.331987 27693 ts_tablet_manager.cc:1428] T d08c9e1e7d564886a90f6c056794becc P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20250903 20:05:10.364189 27693 tablet_bootstrap.cc:492] T 806d4fa868224194a0cf05e81b0ff377 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:10.364800 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c25aaa6ff310432bb27a0695394ec420. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.366050 27693 tablet_bootstrap.cc:654] T 806d4fa868224194a0cf05e81b0ff377 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:10.367303 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 161f2c2afc2f4c82b2025f7daf460bd6 (table default.Table_1a5c8c3256c24c7f8ef8597efa53dde5 [id=7caef592f80a4d6cb6df4e2c3971de49]) successfully deleted
I20250903 20:05:10.369436 27685 tablet_replica.cc:331] T 7e0810c96eb24b8ab4278038801be245 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:10.369661 27685 raft_consensus.cc:2241] T 7e0810c96eb24b8ab4278038801be245 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.369836 27685 raft_consensus.cc:2270] T 7e0810c96eb24b8ab4278038801be245 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.372294 27685 ts_tablet_manager.cc:1905] T 7e0810c96eb24b8ab4278038801be245 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.381597 27685 ts_tablet_manager.cc:1918] T 7e0810c96eb24b8ab4278038801be245 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.381778 27685 log.cc:1199] T 7e0810c96eb24b8ab4278038801be245 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/7e0810c96eb24b8ab4278038801be245
I20250903 20:05:10.382129 27685 ts_tablet_manager.cc:1939] T 7e0810c96eb24b8ab4278038801be245 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:10.383670 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 0a266b1bbf1540ac86d49c56df2dd4ad
I20250903 20:05:10.384200 27062 catalog_manager.cc:6652] ReplaceTablet: tablet 0a266b1bbf1540ac86d49c56df2dd4ad deleted and replaced by tablet 73f65e7515994b898eec2ea23903aff4
I20250903 20:05:10.386478 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet e722212b281a405fb08908df0a42d864
I20250903 20:05:10.386760 27685 tablet_replica.cc:331] T 43f88abc8f3640a48598c07c3d76755d P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:10.386950 27685 raft_consensus.cc:2241] T 43f88abc8f3640a48598c07c3d76755d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.387765 27685 raft_consensus.cc:2270] T 43f88abc8f3640a48598c07c3d76755d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.389340 27063 master_service.cc:948] ReplaceTablet: received request to replace tablet 7e0810c96eb24b8ab4278038801be245 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:10.389272 27685 ts_tablet_manager.cc:1905] T 43f88abc8f3640a48598c07c3d76755d P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.389776 27063 master_service.cc:948] ReplaceTablet: received request to replace tablet 0a266b1bbf1540ac86d49c56df2dd4ad from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:10.390151 27063 master_service.cc:948] ReplaceTablet: received request to replace tablet f3a33e90dc1f4540aee457de96f5b95b from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:10.390807 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 4bd2650e4d9e40f5ba5c09e367918224
I20250903 20:05:10.391467 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_edfc268f076142a2b7c2ef149edf7f5c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:10.392098 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_edfc268f076142a2b7c2ef149edf7f5c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.393446 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 925c0ab4873041da82702cd9377ec7bc
I20250903 20:05:10.394090 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e3f5c55d553840099371cad3a73e2ca5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:10.394696 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3f5c55d553840099371cad3a73e2ca5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.396246 27057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a0ba026e7424483ba463d3d0751e3e4e" } new_table_name: "default.Table_7dfd9faa02434381b40a56369804413b" modify_external_catalogs: true
I20250903 20:05:10.397814 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_74d1733a6f3d40ccb6578a52969f03f6" } new_table_name: "default.Table_97be735f442746fe92867a2df71838a0" modify_external_catalogs: true
I20250903 20:05:10.399415 27059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f6203681100e4faeb016f0e6afc283f7" } modify_external_catalogs: true
I20250903 20:05:10.399580 27059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f6203681100e4faeb016f0e6afc283f7" } modify_external_catalogs: true
I20250903 20:05:10.400486 27717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 20:05:10.401978 27693 tablet_bootstrap.cc:492] T 806d4fa868224194a0cf05e81b0ff377 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:10.402120 27693 ts_tablet_manager.cc:1397] T 806d4fa868224194a0cf05e81b0ff377 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.010s
I20250903 20:05:10.402805 27685 ts_tablet_manager.cc:1918] T 43f88abc8f3640a48598c07c3d76755d P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.402946 27685 log.cc:1199] T 43f88abc8f3640a48598c07c3d76755d P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/43f88abc8f3640a48598c07c3d76755d
I20250903 20:05:10.403385 27685 ts_tablet_manager.cc:1939] T 43f88abc8f3640a48598c07c3d76755d P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:10.403402 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 7e0810c96eb24b8ab4278038801be245 (table default.Table_db7d825a12ef40739b70f11044170c72 [id=790f57164c7d4183a3df17ba7f230d24]) successfully deleted
I20250903 20:05:10.403738 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet e722212b281a405fb08908df0a42d864 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:10.403980 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet 1d9a2508437b41f08821a29f58163159 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:10.404155 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet 4bd2650e4d9e40f5ba5c09e367918224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:10.404378 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 925c0ab4873041da82702cd9377ec7bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:10.404484 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 43f88abc8f3640a48598c07c3d76755d (table default.Table_3e3261a8bf49425b8203ec49357abd67 [id=71d850a0cea14b0ba337a822ad99d513]) successfully deleted
I20250903 20:05:10.404961 27693 raft_consensus.cc:357] T 806d4fa868224194a0cf05e81b0ff377 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.405164 27693 raft_consensus.cc:383] T 806d4fa868224194a0cf05e81b0ff377 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250903 20:05:10.405172 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet a45604ad611e418e84dab793e04101a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a45604ad611e418e84dab793e04101a0 already in progress: creating tablet
I20250903 20:05:10.405265 27693 raft_consensus.cc:738] T 806d4fa868224194a0cf05e81b0ff377 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.405539 27693 consensus_queue.cc:260] T 806d4fa868224194a0cf05e81b0ff377 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:10.405740 27052 catalog_manager.cc:4895] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet e0d1b29d7f1e422e8bb5b8bbc33423e8 because the tablet was not found. No further retry: Not found: Tablet not found: e0d1b29d7f1e422e8bb5b8bbc33423e8
W20250903 20:05:10.406369 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet e722212b281a405fb08908df0a42d864 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e722212b281a405fb08908df0a42d864 already in progress: creating tablet
I20250903 20:05:10.407177 27685 tablet_replica.cc:331] T 1d9a2508437b41f08821a29f58163159 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:10.407372 27685 raft_consensus.cc:2241] T 1d9a2508437b41f08821a29f58163159 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.407547 27685 raft_consensus.cc:2270] T 1d9a2508437b41f08821a29f58163159 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.406668 27693 ts_tablet_manager.cc:1428] T 806d4fa868224194a0cf05e81b0ff377 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250903 20:05:10.406668 27366 tablet_service.cc:1515] Processing DeleteTablet for tablet d08c9e1e7d564886a90f6c056794becc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:10.406831 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet e722212b281a405fb08908df0a42d864 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:10.408565 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet e722212b281a405fb08908df0a42d864 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:10.407007 27363 tablet_service.cc:1515] Processing DeleteTablet for tablet 4bd2650e4d9e40f5ba5c09e367918224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:10.408864 27693 tablet_bootstrap.cc:492] T cd3092c89c74475a999f5b3a6d77e652 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:10.408888 27687 tablet_replica.cc:331] T d08c9e1e7d564886a90f6c056794becc P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:10.409046 27687 raft_consensus.cc:2241] T d08c9e1e7d564886a90f6c056794becc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.409230 27687 raft_consensus.cc:2270] T d08c9e1e7d564886a90f6c056794becc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.409425 27692 tablet_bootstrap.cc:492] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:10.409598 27692 ts_tablet_manager.cc:1397] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.142s user 0.019s sys 0.033s
I20250903 20:05:10.409976 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet 0a266b1bbf1540ac86d49c56df2dd4ad with delete_type TABLET_DATA_DELETED (replaced by tablet 73f65e7515994b898eec2ea23903aff4) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:10.407084 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet 925c0ab4873041da82702cd9377ec7bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:10.410519 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet 0a266b1bbf1540ac86d49c56df2dd4ad with delete_type TABLET_DATA_DELETED (replaced by tablet 73f65e7515994b898eec2ea23903aff4) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:10.410461 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3e3261a8bf49425b8203ec49357abd67" } new_table_name: "default.Table_a5f8f31c12b742b597c8b35d662a607f" modify_external_catalogs: true
I20250903 20:05:10.411027 27687 ts_tablet_manager.cc:1905] T d08c9e1e7d564886a90f6c056794becc P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.412442 27693 tablet_bootstrap.cc:654] T cd3092c89c74475a999f5b3a6d77e652 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:10.412936 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f9e41274ca534e46b7906480d7fc32fe" } modify_external_catalogs: true
I20250903 20:05:10.413141 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f9e41274ca534e46b7906480d7fc32fe" } modify_external_catalogs: true
I20250903 20:05:10.413658 27692 raft_consensus.cc:357] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.413843 27692 raft_consensus.cc:383] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.413924 27692 raft_consensus.cc:738] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.414142 27692 consensus_queue.cc:260] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.414666 27692 ts_tablet_manager.cc:1428] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250903 20:05:10.408361 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet 0a266b1bbf1540ac86d49c56df2dd4ad with delete_type TABLET_DATA_DELETED (replaced by tablet 73f65e7515994b898eec2ea23903aff4) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:10.415179 27685 ts_tablet_manager.cc:1905] T 1d9a2508437b41f08821a29f58163159 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.415649 27692 tablet_bootstrap.cc:492] T b6b4713936134681aaef1732d46c6ab2 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:10.417397 27692 tablet_bootstrap.cc:654] T b6b4713936134681aaef1732d46c6ab2 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:10.417742 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet e722212b281a405fb08908df0a42d864 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e722212b281a405fb08908df0a42d864 already in progress: creating tablet
W20250903 20:05:10.417994 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 0a266b1bbf1540ac86d49c56df2dd4ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a266b1bbf1540ac86d49c56df2dd4ad already in progress: creating tablet
I20250903 20:05:10.420534 27717 hybrid_clock.cc:648] HybridClock initialized: now 1756929910420395 us; error 122 us; skew 500 ppm
I20250903 20:05:10.421044 27717 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 20:05:10.421839 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f20a070ead204fcab2d81f7025c09666" } new_table_name: "default.Table_c557d48ec717410fb860dfdc7ce9957b" modify_external_catalogs: true
I20250903 20:05:10.422713 27503 tablet_service.cc:1515] Processing DeleteTablet for tablet 925c0ab4873041da82702cd9377ec7bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:10.423396 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet 4bd2650e4d9e40f5ba5c09e367918224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:10.423707 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet e0d1b29d7f1e422e8bb5b8bbc33423e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:10.423740 27687 ts_tablet_manager.cc:1918] T d08c9e1e7d564886a90f6c056794becc P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.423974 27684 tablet_replica.cc:331] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:10.423992 27687 log.cc:1199] T d08c9e1e7d564886a90f6c056794becc P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/d08c9e1e7d564886a90f6c056794becc
I20250903 20:05:10.424203 27684 raft_consensus.cc:2241] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
W20250903 20:05:10.424301 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 925c0ab4873041da82702cd9377ec7bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 925c0ab4873041da82702cd9377ec7bc already in progress: creating tablet
I20250903 20:05:10.424405 27684 raft_consensus.cc:2270] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.424901 27687 ts_tablet_manager.cc:1939] T d08c9e1e7d564886a90f6c056794becc P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:10.426620 27684 ts_tablet_manager.cc:1905] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.426833 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet d08c9e1e7d564886a90f6c056794becc (table default.Table_562dd57ebbc14070b3c772c08e93d7df [id=02a9013803934e17884b377e69dd6cac]) successfully deleted
W20250903 20:05:10.429335 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet e722212b281a405fb08908df0a42d864 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e722212b281a405fb08908df0a42d864 already in progress: creating tablet
W20250903 20:05:10.430537 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 4bd2650e4d9e40f5ba5c09e367918224 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4bd2650e4d9e40f5ba5c09e367918224 already in progress: creating tablet
W20250903 20:05:10.431565 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 925c0ab4873041da82702cd9377ec7bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 925c0ab4873041da82702cd9377ec7bc already in progress: creating tablet
W20250903 20:05:10.432670 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 0a266b1bbf1540ac86d49c56df2dd4ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a266b1bbf1540ac86d49c56df2dd4ad already in progress: creating tablet
I20250903 20:05:10.434885 27685 ts_tablet_manager.cc:1918] T 1d9a2508437b41f08821a29f58163159 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.435631 27684 ts_tablet_manager.cc:1918] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.435813 27684 log.cc:1199] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/e0d1b29d7f1e422e8bb5b8bbc33423e8
I20250903 20:05:10.436261 27684 ts_tablet_manager.cc:1939] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
W20250903 20:05:10.436924 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 4bd2650e4d9e40f5ba5c09e367918224 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4bd2650e4d9e40f5ba5c09e367918224 already in progress: creating tablet
I20250903 20:05:10.437316 27685 log.cc:1199] T 1d9a2508437b41f08821a29f58163159 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/1d9a2508437b41f08821a29f58163159
I20250903 20:05:10.437950 27685 ts_tablet_manager.cc:1939] T 1d9a2508437b41f08821a29f58163159 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:10.438297 27717 webserver.cc:480] Webserver started at http://127.26.65.126:33509/ using document root <none> and password file <none>
W20250903 20:05:10.438851 27057 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4692dd93b76c42c4883c7e22c6032250 consensus state has no leader
I20250903 20:05:10.439512 27717 fs_manager.cc:362] Metadata directory not provided
I20250903 20:05:10.439682 27717 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 20:05:10.440248 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c9766617998c49b199a79052643d496a" } new_table_name: "default.Table_a0582ccb94a64603a5fbdd0e531a9edf" modify_external_catalogs: true
I20250903 20:05:10.440538 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 1d9a2508437b41f08821a29f58163159 (table default.Table_c106f52e6b6b458391a12b3043e7eedf [id=cbae62c266f24526affd029472f41b94]) successfully deleted
I20250903 20:05:10.440634 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet 806d4fa868224194a0cf05e81b0ff377 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:10.440725 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_8b2ba396ac1b424493dbc17297d2d133"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:10.440886 27687 tablet_replica.cc:331] T 806d4fa868224194a0cf05e81b0ff377 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:10.441061 27687 raft_consensus.cc:2241] T 806d4fa868224194a0cf05e81b0ff377 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.441239 27687 raft_consensus.cc:2270] T 806d4fa868224194a0cf05e81b0ff377 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:10.441320 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b2ba396ac1b424493dbc17297d2d133 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.442116 27687 ts_tablet_manager.cc:1905] T 806d4fa868224194a0cf05e81b0ff377 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.444976 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet e0d1b29d7f1e422e8bb5b8bbc33423e8 (table default.Table_82f29a4a80194976bc7c8f864ec877a3 [id=49551de884e7488fb6db7249f834e12f]) successfully deleted
I20250903 20:05:10.445904 27695 tablet_bootstrap.cc:492] T e722212b281a405fb08908df0a42d864 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
W20250903 20:05:10.445899 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 4bd2650e4d9e40f5ba5c09e367918224 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4bd2650e4d9e40f5ba5c09e367918224 already in progress: creating tablet
I20250903 20:05:10.446066 27695 ts_tablet_manager.cc:1397] T e722212b281a405fb08908df0a42d864 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.107s user 0.020s sys 0.016s
I20250903 20:05:10.446945 27695 raft_consensus.cc:357] T e722212b281a405fb08908df0a42d864 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.447163 27695 raft_consensus.cc:383] T e722212b281a405fb08908df0a42d864 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.447260 27695 raft_consensus.cc:738] T e722212b281a405fb08908df0a42d864 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.447538 27695 consensus_queue.cc:260] T e722212b281a405fb08908df0a42d864 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.448632 27693 tablet_bootstrap.cc:492] T cd3092c89c74475a999f5b3a6d77e652 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:10.448840 27693 ts_tablet_manager.cc:1397] T cd3092c89c74475a999f5b3a6d77e652 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
W20250903 20:05:10.449822 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 925c0ab4873041da82702cd9377ec7bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 925c0ab4873041da82702cd9377ec7bc already in progress: creating tablet
W20250903 20:05:10.451208 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 0a266b1bbf1540ac86d49c56df2dd4ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a266b1bbf1540ac86d49c56df2dd4ad already in progress: creating tablet
I20250903 20:05:10.452867 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet f3a33e90dc1f4540aee457de96f5b95b
I20250903 20:05:10.454041 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet f3a33e90dc1f4540aee457de96f5b95b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:10.454176 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 12a7be47c9c2406596c9e00800074a7c consensus state has no leader
W20250903 20:05:10.454828 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet f3a33e90dc1f4540aee457de96f5b95b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3a33e90dc1f4540aee457de96f5b95b already in progress: creating tablet
W20250903 20:05:10.457544 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 68cff0e416a54a5eb37071f5a12f07ab has no consensus state
W20250903 20:05:10.458014 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b5bdc27275cd4156afe767a1fb33a8bd consensus state has no leader
I20250903 20:05:10.458395 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet f3a33e90dc1f4540aee457de96f5b95b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:10.459578 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet f3a33e90dc1f4540aee457de96f5b95b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3a33e90dc1f4540aee457de96f5b95b already in progress: creating tablet
I20250903 20:05:10.459699 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet f3a33e90dc1f4540aee457de96f5b95b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:10.459771 27693 raft_consensus.cc:357] T cd3092c89c74475a999f5b3a6d77e652 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.459983 27693 raft_consensus.cc:383] T cd3092c89c74475a999f5b3a6d77e652 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.460058 27693 raft_consensus.cc:738] T cd3092c89c74475a999f5b3a6d77e652 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.460347 27693 consensus_queue.cc:260] T cd3092c89c74475a999f5b3a6d77e652 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.460872 27693 ts_tablet_manager.cc:1428] T cd3092c89c74475a999f5b3a6d77e652 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20250903 20:05:10.461246 27693 tablet_bootstrap.cc:492] T fd499b81344c4b58bf77f3622422763c P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:10.461668 27695 ts_tablet_manager.cc:1428] T e722212b281a405fb08908df0a42d864 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20250903 20:05:10.461869 27687 ts_tablet_manager.cc:1918] T 806d4fa868224194a0cf05e81b0ff377 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.462774 27687 log.cc:1199] T 806d4fa868224194a0cf05e81b0ff377 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/806d4fa868224194a0cf05e81b0ff377
I20250903 20:05:10.463207 27687 ts_tablet_manager.cc:1939] T 806d4fa868224194a0cf05e81b0ff377 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:10.463387 27693 tablet_bootstrap.cc:654] T fd499b81344c4b58bf77f3622422763c P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:10.464323 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 806d4fa868224194a0cf05e81b0ff377 (table default.Table_6106b8f9be34419983b266581e83bed8 [id=de7efd68262248f38a3d8d4a8d6c87db]) successfully deleted
W20250903 20:05:10.464696 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet f3a33e90dc1f4540aee457de96f5b95b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3a33e90dc1f4540aee457de96f5b95b already in progress: creating tablet
I20250903 20:05:10.464774 27695 tablet_bootstrap.cc:492] T f99c05e72b9a4ff9a1e7c269ca029b6a P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:10.466756 27695 tablet_bootstrap.cc:654] T f99c05e72b9a4ff9a1e7c269ca029b6a P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:10.468794 27717 fs_manager.cc:714] Time spent opening directory manager: real 0.026s user 0.007s sys 0.006s
I20250903 20:05:10.472553 27695 tablet_bootstrap.cc:492] T f99c05e72b9a4ff9a1e7c269ca029b6a P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:10.473004 27695 ts_tablet_manager.cc:1397] T f99c05e72b9a4ff9a1e7c269ca029b6a P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20250903 20:05:10.474949 27692 tablet_bootstrap.cc:492] T b6b4713936134681aaef1732d46c6ab2 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:10.475178 27692 ts_tablet_manager.cc:1397] T b6b4713936134681aaef1732d46c6ab2 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.060s user 0.013s sys 0.004s
I20250903 20:05:10.475447 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet cd3092c89c74475a999f5b3a6d77e652 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:10.476091 27692 raft_consensus.cc:357] T b6b4713936134681aaef1732d46c6ab2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.476279 27687 tablet_replica.cc:331] T cd3092c89c74475a999f5b3a6d77e652 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:10.476320 27692 raft_consensus.cc:383] T b6b4713936134681aaef1732d46c6ab2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.476387 27692 raft_consensus.cc:738] T b6b4713936134681aaef1732d46c6ab2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.476473 27687 raft_consensus.cc:2241] T cd3092c89c74475a999f5b3a6d77e652 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.476670 27687 raft_consensus.cc:2270] T cd3092c89c74475a999f5b3a6d77e652 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.476889 27693 tablet_bootstrap.cc:492] T fd499b81344c4b58bf77f3622422763c P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:10.476786 27695 raft_consensus.cc:357] T f99c05e72b9a4ff9a1e7c269ca029b6a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.477030 27695 raft_consensus.cc:383] T f99c05e72b9a4ff9a1e7c269ca029b6a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.477051 27693 ts_tablet_manager.cc:1397] T fd499b81344c4b58bf77f3622422763c P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.012s
I20250903 20:05:10.477111 27695 raft_consensus.cc:738] T f99c05e72b9a4ff9a1e7c269ca029b6a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.477378 27695 consensus_queue.cc:260] T f99c05e72b9a4ff9a1e7c269ca029b6a P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:10.477766 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8945085bdc174e6db22a5a9477810297 consensus state has no leader
I20250903 20:05:10.477896 27695 ts_tablet_manager.cc:1428] T f99c05e72b9a4ff9a1e7c269ca029b6a P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:10.478024 27693 raft_consensus.cc:357] T fd499b81344c4b58bf77f3622422763c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.478281 27693 raft_consensus.cc:383] T fd499b81344c4b58bf77f3622422763c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.478379 27693 raft_consensus.cc:738] T fd499b81344c4b58bf77f3622422763c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.478715 27692 consensus_queue.cc:260] T b6b4713936134681aaef1732d46c6ab2 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.479640 27693 consensus_queue.cc:260] T fd499b81344c4b58bf77f3622422763c P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.480860 27693 ts_tablet_manager.cc:1428] T fd499b81344c4b58bf77f3622422763c P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250903 20:05:10.481271 27693 tablet_bootstrap.cc:492] T 3906285457f449e6824d1f044bffca24 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:10.482016 27692 ts_tablet_manager.cc:1428] T b6b4713936134681aaef1732d46c6ab2 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250903 20:05:10.482442 27692 tablet_bootstrap.cc:492] T fd499b81344c4b58bf77f3622422763c P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:10.482861 27057 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet c8a293c6fa034e9fb29682439cb3bcc2 (table default.Table_37e1d397cf99443d87211c3a8e1b9b82 [id=5cfc8827e4ca4f4a94424c48d6f71eb5]). Expected version 2 got 0
I20250903 20:05:10.485308 27693 tablet_bootstrap.cc:654] T 3906285457f449e6824d1f044bffca24 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:10.485572 27784 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 20:05:10.487089 27692 tablet_bootstrap.cc:654] T fd499b81344c4b58bf77f3622422763c P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:10.487182 27050 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 12a7be47c9c2406596c9e00800074a7c (table default.Table_97be735f442746fe92867a2df71838a0 [id=fdc7899c8b334d309b834ae1daa14fa0], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 12a7be47c9c2406596c9e00800074a7c consensus state has no leader
I20250903 20:05:10.487725 27717 fs_manager.cc:730] Time spent opening block manager: real 0.014s user 0.003s sys 0.005s
I20250903 20:05:10.489552 27214 tablet_service.cc:1468] Processing CreateTablet for tablet cfa8a1d9011e460681443a6b77b95958 (DEFAULT_TABLE table=default.Table_f4196ed1d25c4d6fab512014504c51af [id=058a64daaf104dfdace7d05d52388b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.489982 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfa8a1d9011e460681443a6b77b95958. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.490172 27217 tablet_service.cc:1468] Processing CreateTablet for tablet f7ab722c37264973b204cd4f1f95052d (DEFAULT_TABLE table=default.Table_b2a82a2c310140caaeac5d0dcb4bb61a [id=e2548ad469e54a41af7c0181357d2467]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.490526 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7ab722c37264973b204cd4f1f95052d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.492660 27059 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet f99c05e72b9a4ff9a1e7c269ca029b6a (table default.Table_0c3cf60b884e47eaaa60b3e0e80d56a3 [id=0b57a09164704002b7fcd0b8f08da1e9]). Expected version 1 got 0
I20250903 20:05:10.493780 27360 tablet_service.cc:1468] Processing CreateTablet for tablet cfa8a1d9011e460681443a6b77b95958 (DEFAULT_TABLE table=default.Table_f4196ed1d25c4d6fab512014504c51af [id=058a64daaf104dfdace7d05d52388b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.493832 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 4268316c28e8471a9f28807d20dbb395 (DEFAULT_TABLE table=default.Table_c79b97a3bb294caf96bb5fa0d144628d [id=f70d7ea93e154e6199f6766316353862]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.494201 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4268316c28e8471a9f28807d20dbb395. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.494282 27695 tablet_bootstrap.cc:492] T 0062307c6ee54aada74a08cf0d087f17 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:10.493294 27226 tablet_service.cc:1468] Processing CreateTablet for tablet 68cff0e416a54a5eb37071f5a12f07ab (DEFAULT_TABLE table=default.Table_a5f8f31c12b742b597c8b35d662a607f [id=71d850a0cea14b0ba337a822ad99d513]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.497119 27499 tablet_service.cc:1468] Processing CreateTablet for tablet cfa8a1d9011e460681443a6b77b95958 (DEFAULT_TABLE table=default.Table_f4196ed1d25c4d6fab512014504c51af [id=058a64daaf104dfdace7d05d52388b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.497370 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 8023fa7537234ce783c2bf90b575b00c (DEFAULT_TABLE table=default.Table_6d92c85ead9f4f58855e725b7823c449 [id=7032fe3b789041d380f5db114f241e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.497530 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68cff0e416a54a5eb37071f5a12f07ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.497548 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfa8a1d9011e460681443a6b77b95958. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.497718 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8023fa7537234ce783c2bf90b575b00c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.498370 27223 tablet_service.cc:1515] Processing DeleteTablet for tablet e722212b281a405fb08908df0a42d864 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:10.499840 27220 tablet_service.cc:1468] Processing CreateTablet for tablet 8023fa7537234ce783c2bf90b575b00c (DEFAULT_TABLE table=default.Table_6d92c85ead9f4f58855e725b7823c449 [id=7032fe3b789041d380f5db114f241e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.494195 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfa8a1d9011e460681443a6b77b95958. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.502424 27223 tablet_service.cc:1468] Processing CreateTablet for tablet e20fcd28c3ae4355a033d2b8429f84e2 (DEFAULT_TABLE table=default.Table_53cc54536434460ba6637a083afaddb2 [id=0e611bffbb74451d9e24eed4a553c2ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.502535 27502 tablet_service.cc:1468] Processing CreateTablet for tablet 4268316c28e8471a9f28807d20dbb395 (DEFAULT_TABLE table=default.Table_c79b97a3bb294caf96bb5fa0d144628d [id=f70d7ea93e154e6199f6766316353862]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.502914 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4268316c28e8471a9f28807d20dbb395. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.494493 27358 tablet_service.cc:1468] Processing CreateTablet for tablet e20fcd28c3ae4355a033d2b8429f84e2 (DEFAULT_TABLE table=default.Table_53cc54536434460ba6637a083afaddb2 [id=0e611bffbb74451d9e24eed4a553c2ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.499284 27362 tablet_service.cc:1468] Processing CreateTablet for tablet 4268316c28e8471a9f28807d20dbb395 (DEFAULT_TABLE table=default.Table_c79b97a3bb294caf96bb5fa0d144628d [id=f70d7ea93e154e6199f6766316353862]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.499501 27366 tablet_service.cc:1468] Processing CreateTablet for tablet 68cff0e416a54a5eb37071f5a12f07ab (DEFAULT_TABLE table=default.Table_a5f8f31c12b742b597c8b35d662a607f [id=71d850a0cea14b0ba337a822ad99d513]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.506585 27501 tablet_service.cc:1468] Processing CreateTablet for tablet 68cff0e416a54a5eb37071f5a12f07ab (DEFAULT_TABLE table=default.Table_a5f8f31c12b742b597c8b35d662a607f [id=71d850a0cea14b0ba337a822ad99d513]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.498873 27364 tablet_service.cc:1468] Processing CreateTablet for tablet f7ab722c37264973b204cd4f1f95052d (DEFAULT_TABLE table=default.Table_b2a82a2c310140caaeac5d0dcb4bb61a [id=e2548ad469e54a41af7c0181357d2467]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.506966 27501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68cff0e416a54a5eb37071f5a12f07ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.499075 27363 tablet_service.cc:1468] Processing CreateTablet for tablet 8023fa7537234ce783c2bf90b575b00c (DEFAULT_TABLE table=default.Table_6d92c85ead9f4f58855e725b7823c449 [id=7032fe3b789041d380f5db114f241e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.507313 27685 tablet_replica.cc:331] T e722212b281a405fb08908df0a42d864 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:10.507525 27685 raft_consensus.cc:2241] T e722212b281a405fb08908df0a42d864 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.507654 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e20fcd28c3ae4355a033d2b8429f84e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.507771 27685 raft_consensus.cc:2270] T e722212b281a405fb08908df0a42d864 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.508245 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68cff0e416a54a5eb37071f5a12f07ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.509007 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7ab722c37264973b204cd4f1f95052d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.509168 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4268316c28e8471a9f28807d20dbb395. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.509348 27500 tablet_service.cc:1468] Processing CreateTablet for tablet e20fcd28c3ae4355a033d2b8429f84e2 (DEFAULT_TABLE table=default.Table_53cc54536434460ba6637a083afaddb2 [id=0e611bffbb74451d9e24eed4a553c2ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.509716 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e20fcd28c3ae4355a033d2b8429f84e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.510046 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8023fa7537234ce783c2bf90b575b00c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.510200 27685 ts_tablet_manager.cc:1905] T e722212b281a405fb08908df0a42d864 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.510365 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet e0d1b29d7f1e422e8bb5b8bbc33423e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:10.497084 27503 tablet_service.cc:1468] Processing CreateTablet for tablet f7ab722c37264973b204cd4f1f95052d (DEFAULT_TABLE table=default.Table_b2a82a2c310140caaeac5d0dcb4bb61a [id=e2548ad469e54a41af7c0181357d2467]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.511123 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7ab722c37264973b204cd4f1f95052d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.512009 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8023fa7537234ce783c2bf90b575b00c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.514645 27223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e20fcd28c3ae4355a033d2b8429f84e2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:10.516628 27051 catalog_manager.cc:4895] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet e0d1b29d7f1e422e8bb5b8bbc33423e8 because the tablet was not found. No further retry: Not found: Tablet not found: e0d1b29d7f1e422e8bb5b8bbc33423e8
I20250903 20:05:10.516701 27685 ts_tablet_manager.cc:1918] T e722212b281a405fb08908df0a42d864 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.516819 27685 log.cc:1199] T e722212b281a405fb08908df0a42d864 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/e722212b281a405fb08908df0a42d864
I20250903 20:05:10.517184 27685 ts_tablet_manager.cc:1939] T e722212b281a405fb08908df0a42d864 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:10.517839 27062 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet b6b4713936134681aaef1732d46c6ab2 (table default.Table_b6f25e1f7a484577a9cbff11e0deb857 [id=d0a75545b91c48bba47fb581bb0d1a96]). Expected version 1 got 0
I20250903 20:05:10.518185 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet e722212b281a405fb08908df0a42d864 (table default.Table_5f1e055d858e4d31bce98ac57741cc30 [id=7fd6a8bf57a846f78b1bf70fa3ea5175]) successfully deleted
I20250903 20:05:10.519670 27062 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet fd499b81344c4b58bf77f3622422763c (table default.Table_e914faf335d44d019f91bda71758c473 [id=374b9abdf6a54b38876a96d8ec4a0368]). Expected version 4 got 0
I20250903 20:05:10.520812 27687 ts_tablet_manager.cc:1905] T cd3092c89c74475a999f5b3a6d77e652 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.523145 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet e1a49d0599654552bf1c72857031699b
I20250903 20:05:10.524705 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet e1a49d0599654552bf1c72857031699b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:10.525503 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet e1a49d0599654552bf1c72857031699b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1a49d0599654552bf1c72857031699b already in progress: creating tablet
I20250903 20:05:10.525856 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet e1a49d0599654552bf1c72857031699b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:10.526539 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet e1a49d0599654552bf1c72857031699b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1a49d0599654552bf1c72857031699b already in progress: creating tablet
I20250903 20:05:10.525372 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet e1a49d0599654552bf1c72857031699b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:10.528173 27695 tablet_bootstrap.cc:654] T 0062307c6ee54aada74a08cf0d087f17 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:10.535944 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_18040dcdd59e4eaf92f9aa8748571a1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:10.536813 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18040dcdd59e4eaf92f9aa8748571a1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.539492 27695 tablet_bootstrap.cc:492] T 0062307c6ee54aada74a08cf0d087f17 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:10.539687 27695 ts_tablet_manager.cc:1397] T 0062307c6ee54aada74a08cf0d087f17 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.005s
I20250903 20:05:10.539953 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ffb4d9c1a2d945f1af90b41164c2875a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:10.540616 27695 raft_consensus.cc:357] T 0062307c6ee54aada74a08cf0d087f17 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.540818 27695 raft_consensus.cc:383] T 0062307c6ee54aada74a08cf0d087f17 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250903 20:05:10.540870 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ffb4d9c1a2d945f1af90b41164c2875a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.540931 27695 raft_consensus.cc:738] T 0062307c6ee54aada74a08cf0d087f17 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.541164 27695 consensus_queue.cc:260] T 0062307c6ee54aada74a08cf0d087f17 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.543057 27695 ts_tablet_manager.cc:1428] T 0062307c6ee54aada74a08cf0d087f17 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20250903 20:05:10.546437 27695 tablet_bootstrap.cc:492] T c12c957a36f643d9bfb60b755619cbbc P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:10.548673 27695 tablet_bootstrap.cc:654] T c12c957a36f643d9bfb60b755619cbbc P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:10.554142 27693 tablet_bootstrap.cc:492] T 3906285457f449e6824d1f044bffca24 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:10.554420 27693 ts_tablet_manager.cc:1397] T 3906285457f449e6824d1f044bffca24 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.073s user 0.002s sys 0.006s
I20250903 20:05:10.555308 27693 raft_consensus.cc:357] T 3906285457f449e6824d1f044bffca24 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.555534 27693 raft_consensus.cc:383] T 3906285457f449e6824d1f044bffca24 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.555635 27693 raft_consensus.cc:738] T 3906285457f449e6824d1f044bffca24 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.555918 27693 consensus_queue.cc:260] T 3906285457f449e6824d1f044bffca24 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.557545 27693 ts_tablet_manager.cc:1428] T 3906285457f449e6824d1f044bffca24 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250903 20:05:10.559638 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet 0062307c6ee54aada74a08cf0d087f17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:10.560667 27685 tablet_replica.cc:331] T 0062307c6ee54aada74a08cf0d087f17 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:10.561044 27685 raft_consensus.cc:2241] T 0062307c6ee54aada74a08cf0d087f17 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.561424 27685 raft_consensus.cc:2270] T 0062307c6ee54aada74a08cf0d087f17 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.541436 27717 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/data,/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/wal
uuid: "8857b5d4ea1e4f45aafb8e029837cca2"
format_stamp: "Formatted at 2025-09-03 20:05:04 on dist-test-slave-fs2w"
I20250903 20:05:10.562050 27692 tablet_bootstrap.cc:492] T fd499b81344c4b58bf77f3622422763c P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:10.562283 27692 ts_tablet_manager.cc:1397] T fd499b81344c4b58bf77f3622422763c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.080s user 0.004s sys 0.004s
I20250903 20:05:10.562399 27685 ts_tablet_manager.cc:1905] T 0062307c6ee54aada74a08cf0d087f17 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.562441 27717 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 20:05:10.563566 27692 raft_consensus.cc:357] T fd499b81344c4b58bf77f3622422763c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.563778 27692 raft_consensus.cc:383] T fd499b81344c4b58bf77f3622422763c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.563899 27692 raft_consensus.cc:738] T fd499b81344c4b58bf77f3622422763c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.564204 27692 consensus_queue.cc:260] T fd499b81344c4b58bf77f3622422763c P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.565044 27692 ts_tablet_manager.cc:1428] T fd499b81344c4b58bf77f3622422763c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:10.565402 27692 tablet_bootstrap.cc:492] T 43f88abc8f3640a48598c07c3d76755d P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:10.567196 27693 tablet_bootstrap.cc:492] T 1d9a2508437b41f08821a29f58163159 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:10.569720 27693 tablet_bootstrap.cc:654] T 1d9a2508437b41f08821a29f58163159 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:10.570540 27692 tablet_bootstrap.cc:654] T 43f88abc8f3640a48598c07c3d76755d P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:10.576100 27059 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 3906285457f449e6824d1f044bffca24 (table default.Table_264581b9b25640c58ed0dd863bf6bb8e [id=b35d1fca6bcd4bb6a1581c2c81cac712]). Expected version 1 got 0
I20250903 20:05:10.577128 27685 ts_tablet_manager.cc:1918] T 0062307c6ee54aada74a08cf0d087f17 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.577322 27685 log.cc:1199] T 0062307c6ee54aada74a08cf0d087f17 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/0062307c6ee54aada74a08cf0d087f17
I20250903 20:05:10.577745 27685 ts_tablet_manager.cc:1939] T 0062307c6ee54aada74a08cf0d087f17 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:10.577780 27062 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet fd499b81344c4b58bf77f3622422763c (table default.Table_e914faf335d44d019f91bda71758c473 [id=374b9abdf6a54b38876a96d8ec4a0368]). Expected version 4 got 0
I20250903 20:05:10.582597 27362 tablet_service.cc:1468] Processing CreateTablet for tablet 31342583c026438cbbae14106e75bbe5 (DEFAULT_TABLE table=default.Table_e3f5c55d553840099371cad3a73e2ca5 [id=a950fbaecd7341989961ff3ea9a713e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.583040 27500 tablet_service.cc:1468] Processing CreateTablet for tablet 04ccb3e4a8bd4789b9f19edd4a5b6145 (DEFAULT_TABLE table=default.Table_edfc268f076142a2b7c2ef149edf7f5c [id=b287b55c1542426fb4fd0372ef6bcd05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.583163 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31342583c026438cbbae14106e75bbe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.583508 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04ccb3e4a8bd4789b9f19edd4a5b6145. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.585222 27062 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 3906285457f449e6824d1f044bffca24 (table default.Table_264581b9b25640c58ed0dd863bf6bb8e [id=b35d1fca6bcd4bb6a1581c2c81cac712]). Expected version 1 got 0
I20250903 20:05:10.586565 27503 tablet_service.cc:1468] Processing CreateTablet for tablet 31342583c026438cbbae14106e75bbe5 (DEFAULT_TABLE table=default.Table_e3f5c55d553840099371cad3a73e2ca5 [id=a950fbaecd7341989961ff3ea9a713e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.586935 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31342583c026438cbbae14106e75bbe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.587519 27687 ts_tablet_manager.cc:1918] T cd3092c89c74475a999f5b3a6d77e652 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.588760 27364 tablet_service.cc:1468] Processing CreateTablet for tablet 79a0553c912049cca5971b1ad6f7e11c (DEFAULT_TABLE table=default.Table_afc0f683c5d8460484160a3a9c3e6df5 [id=b89ea241456b4393994ad07d839053f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.588893 27363 tablet_service.cc:1468] Processing CreateTablet for tablet 2a5c74a281b44019872a6442d41bc373 (DEFAULT_TABLE table=default.Table_9d09f11dcb4b4bf09c870934bf1b6acd [id=a9ddfed19dab46a7bb4e049483e3996f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.589184 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79a0553c912049cca5971b1ad6f7e11c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.589951 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 2a5c74a281b44019872a6442d41bc373 (DEFAULT_TABLE table=default.Table_9d09f11dcb4b4bf09c870934bf1b6acd [id=a9ddfed19dab46a7bb4e049483e3996f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.590420 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a5c74a281b44019872a6442d41bc373. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.590487 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 79a0553c912049cca5971b1ad6f7e11c (DEFAULT_TABLE table=default.Table_afc0f683c5d8460484160a3a9c3e6df5 [id=b89ea241456b4393994ad07d839053f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.590842 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79a0553c912049cca5971b1ad6f7e11c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.594539 27502 tablet_service.cc:1468] Processing CreateTablet for tablet 691cb69759654fdc9223844049a06d02 (DEFAULT_TABLE table=default.Table_64c39ce0250747edacd69ba845d72455 [id=dbfd528fcaba4b519108e3c4a5a5c36f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.594913 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 691cb69759654fdc9223844049a06d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.595031 27223 tablet_service.cc:1468] Processing CreateTablet for tablet 691cb69759654fdc9223844049a06d02 (DEFAULT_TABLE table=default.Table_64c39ce0250747edacd69ba845d72455 [id=dbfd528fcaba4b519108e3c4a5a5c36f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.595391 27223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 691cb69759654fdc9223844049a06d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.595758 27062 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet fd499b81344c4b58bf77f3622422763c (table default.Table_e914faf335d44d019f91bda71758c473 [id=374b9abdf6a54b38876a96d8ec4a0368]). Expected version 4 got 0
I20250903 20:05:10.597958 27499 tablet_service.cc:1468] Processing CreateTablet for tablet 262a3549867a473aaaa7dfcc0b5872cd (DEFAULT_TABLE table=default.Table_14af7742726b4562a02fd2c9b2895277 [id=6d72901e5102476890f466fd406a7127]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.598321 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 262a3549867a473aaaa7dfcc0b5872cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.598547 27501 tablet_service.cc:1468] Processing CreateTablet for tablet 2a5c74a281b44019872a6442d41bc373 (DEFAULT_TABLE table=default.Table_9d09f11dcb4b4bf09c870934bf1b6acd [id=a9ddfed19dab46a7bb4e049483e3996f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.598860 27501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a5c74a281b44019872a6442d41bc373. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.599319 27366 tablet_service.cc:1468] Processing CreateTablet for tablet 691cb69759654fdc9223844049a06d02 (DEFAULT_TABLE table=default.Table_64c39ce0250747edacd69ba845d72455 [id=dbfd528fcaba4b519108e3c4a5a5c36f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.599372 27226 tablet_service.cc:1468] Processing CreateTablet for tablet 79a0553c912049cca5971b1ad6f7e11c (DEFAULT_TABLE table=default.Table_afc0f683c5d8460484160a3a9c3e6df5 [id=b89ea241456b4393994ad07d839053f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.599704 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 691cb69759654fdc9223844049a06d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.599785 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 04ccb3e4a8bd4789b9f19edd4a5b6145 (DEFAULT_TABLE table=default.Table_edfc268f076142a2b7c2ef149edf7f5c [id=b287b55c1542426fb4fd0372ef6bcd05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.600180 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79a0553c912049cca5971b1ad6f7e11c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.588779 27687 log.cc:1199] T cd3092c89c74475a999f5b3a6d77e652 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/cd3092c89c74475a999f5b3a6d77e652
I20250903 20:05:10.601337 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 04ccb3e4a8bd4789b9f19edd4a5b6145 (DEFAULT_TABLE table=default.Table_edfc268f076142a2b7c2ef149edf7f5c [id=b287b55c1542426fb4fd0372ef6bcd05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.601691 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04ccb3e4a8bd4789b9f19edd4a5b6145. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.602197 27692 tablet_bootstrap.cc:492] T 43f88abc8f3640a48598c07c3d76755d P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:10.602387 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04ccb3e4a8bd4789b9f19edd4a5b6145. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.602409 27692 ts_tablet_manager.cc:1397] T 43f88abc8f3640a48598c07c3d76755d P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.005s
I20250903 20:05:10.603284 27692 raft_consensus.cc:357] T 43f88abc8f3640a48598c07c3d76755d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.603478 27692 raft_consensus.cc:383] T 43f88abc8f3640a48598c07c3d76755d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.603569 27692 raft_consensus.cc:738] T 43f88abc8f3640a48598c07c3d76755d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.603873 27692 consensus_queue.cc:260] T 43f88abc8f3640a48598c07c3d76755d P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.604602 27692 ts_tablet_manager.cc:1428] T 43f88abc8f3640a48598c07c3d76755d P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:10.604923 27692 tablet_bootstrap.cc:492] T c12c957a36f643d9bfb60b755619cbbc P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:10.606668 27692 tablet_bootstrap.cc:654] T c12c957a36f643d9bfb60b755619cbbc P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:10.606974 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_264581b9b25640c58ed0dd863bf6bb8e" } modify_external_catalogs: true
I20250903 20:05:10.607247 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_264581b9b25640c58ed0dd863bf6bb8e" } modify_external_catalogs: true
I20250903 20:05:10.609170 27059 master_service.cc:948] ReplaceTablet: received request to replace tablet 0f4ef613b0ab4bca98294682b934f6ed from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:10.616394 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 0f4ef613b0ab4bca98294682b934f6ed
I20250903 20:05:10.616978 27059 catalog_manager.cc:6652] ReplaceTablet: tablet 0f4ef613b0ab4bca98294682b934f6ed deleted and replaced by tablet 264127511a234e9fbf409f3aa2860ccc
I20250903 20:05:10.618577 27358 tablet_service.cc:1468] Processing CreateTablet for tablet 262a3549867a473aaaa7dfcc0b5872cd (DEFAULT_TABLE table=default.Table_14af7742726b4562a02fd2c9b2895277 [id=6d72901e5102476890f466fd406a7127]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.618953 27220 tablet_service.cc:1468] Processing CreateTablet for tablet 31342583c026438cbbae14106e75bbe5 (DEFAULT_TABLE table=default.Table_e3f5c55d553840099371cad3a73e2ca5 [id=a950fbaecd7341989961ff3ea9a713e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.619398 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31342583c026438cbbae14106e75bbe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.620393 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a5c74a281b44019872a6442d41bc373. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.620749 27059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7dfd9faa02434381b40a56369804413b" } modify_external_catalogs: true
I20250903 20:05:10.620942 27059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7dfd9faa02434381b40a56369804413b" } modify_external_catalogs: true
I20250903 20:05:10.622327 27693 tablet_bootstrap.cc:492] T 1d9a2508437b41f08821a29f58163159 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:10.622510 27693 ts_tablet_manager.cc:1397] T 1d9a2508437b41f08821a29f58163159 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.055s user 0.003s sys 0.005s
I20250903 20:05:10.624145 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f4ef613b0ab4bca98294682b934f6ed with delete_type TABLET_DATA_DELETED (replaced by tablet 264127511a234e9fbf409f3aa2860ccc) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:10.624495 27687 ts_tablet_manager.cc:1939] T cd3092c89c74475a999f5b3a6d77e652 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:10.625043 27501 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f4ef613b0ab4bca98294682b934f6ed with delete_type TABLET_DATA_DELETED (replaced by tablet 264127511a234e9fbf409f3aa2860ccc) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:10.625629 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_8b2ba396ac1b424493dbc17297d2d133" } modify_external_catalogs: true
I20250903 20:05:10.625813 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_8b2ba396ac1b424493dbc17297d2d133" } modify_external_catalogs: true
I20250903 20:05:10.626298 27693 raft_consensus.cc:357] T 1d9a2508437b41f08821a29f58163159 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:10.626497 27693 raft_consensus.cc:383] T 1d9a2508437b41f08821a29f58163159 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.626605 27693 raft_consensus.cc:738] T 1d9a2508437b41f08821a29f58163159 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.626842 27693 consensus_queue.cc:260] T 1d9a2508437b41f08821a29f58163159 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:10.627249 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet cd3092c89c74475a999f5b3a6d77e652 (table default.Table_60698087f8f34bbe86a9994a117f2b37 [id=0523254ee07b4c5695b05a486945cb66]) successfully deleted
I20250903 20:05:10.627364 27693 ts_tablet_manager.cc:1428] T 1d9a2508437b41f08821a29f58163159 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
W20250903 20:05:10.627595 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet e1a49d0599654552bf1c72857031699b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1a49d0599654552bf1c72857031699b already in progress: creating tablet
I20250903 20:05:10.627689 27693 tablet_bootstrap.cc:492] T f99c05e72b9a4ff9a1e7c269ca029b6a P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
W20250903 20:05:10.627851 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 0f4ef613b0ab4bca98294682b934f6ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f4ef613b0ab4bca98294682b934f6ed already in progress: creating tablet
I20250903 20:05:10.628434 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_8266aa4c05d94b34ad3e45b3035e3039"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:10.628993 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8266aa4c05d94b34ad3e45b3035e3039 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.629813 27500 tablet_service.cc:1515] Processing DeleteTablet for tablet 43f88abc8f3640a48598c07c3d76755d with delete_type TABLET_DATA_DELETED (replaced by tablet 68cff0e416a54a5eb37071f5a12f07ab) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:10.630769 27684 tablet_replica.cc:331] T 43f88abc8f3640a48598c07c3d76755d P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:10.630961 27684 raft_consensus.cc:2241] T 43f88abc8f3640a48598c07c3d76755d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.631145 27684 raft_consensus.cc:2270] T 43f88abc8f3640a48598c07c3d76755d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.631623 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b2824f7f5f324e6ebcacb1b0645de8ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:10.632161 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2824f7f5f324e6ebcacb1b0645de8ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.633409 27684 ts_tablet_manager.cc:1905] T 43f88abc8f3640a48598c07c3d76755d P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.634929 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_a642c53ef0da461ba2f33dd4010e7129"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:10.635010 27719 raft_consensus.cc:491] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:10.635165 27719 raft_consensus.cc:513] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:10.635471 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a642c53ef0da461ba2f33dd4010e7129 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.636401 27719 leader_election.cc:290] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:10.637146 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd499b81344c4b58bf77f3622422763c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:10.637276 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f4ef613b0ab4bca98294682b934f6ed with delete_type TABLET_DATA_DELETED (replaced by tablet 264127511a234e9fbf409f3aa2860ccc) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:10.637481 27525 raft_consensus.cc:2466] T fd499b81344c4b58bf77f3622422763c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:10.638149 27183 leader_election.cc:304] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:10.638531 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 0062307c6ee54aada74a08cf0d087f17 (table default.Table_6d1b2694cd1a4963ac658f25026e9ef5 [id=f4bc244b05e840b9843b4d448d4f28aa]) successfully deleted
I20250903 20:05:10.640257 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_9c5c22a18c1d4913abdfab7d6a589058"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:10.640728 27719 raft_consensus.cc:2802] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250903 20:05:10.640895 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c5c22a18c1d4913abdfab7d6a589058 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.642153 27719 raft_consensus.cc:491] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:10.642264 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 262a3549867a473aaaa7dfcc0b5872cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.642326 27719 raft_consensus.cc:3058] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:10.642434 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd499b81344c4b58bf77f3622422763c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:10.642911 27385 raft_consensus.cc:2466] T fd499b81344c4b58bf77f3622422763c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:10.645020 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_cc799e62ff6b47e1af28b7006551bbf5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:10.645794 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc799e62ff6b47e1af28b7006551bbf5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.646368 27693 tablet_bootstrap.cc:654] T f99c05e72b9a4ff9a1e7c269ca029b6a P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:10.647843 27719 raft_consensus.cc:513] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.648810 27719 leader_election.cc:290] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:10.649295 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd499b81344c4b58bf77f3622422763c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:10.649600 27525 raft_consensus.cc:3058] T fd499b81344c4b58bf77f3622422763c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:10.650761 27684 ts_tablet_manager.cc:1918] T 43f88abc8f3640a48598c07c3d76755d P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.650889 27684 log.cc:1199] T 43f88abc8f3640a48598c07c3d76755d P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/43f88abc8f3640a48598c07c3d76755d
I20250903 20:05:10.651293 27684 ts_tablet_manager.cc:1939] T 43f88abc8f3640a48598c07c3d76755d P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:10.653604 27525 raft_consensus.cc:2466] T fd499b81344c4b58bf77f3622422763c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:10.656206 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd499b81344c4b58bf77f3622422763c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:10.656656 27064 master_service.cc:948] ReplaceTablet: received request to replace tablet be278233c7c34061988b267a44ee674f from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:10.657464 27385 raft_consensus.cc:3058] T fd499b81344c4b58bf77f3622422763c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:10.659148 27183 leader_election.cc:304] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:10.660250 27719 raft_consensus.cc:2802] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:10.660740 27385 raft_consensus.cc:2466] T fd499b81344c4b58bf77f3622422763c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:10.661721 27061 master_service.cc:948] ReplaceTablet: received request to replace tablet 76742d0df44c43729197f7e09f96cad8 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:10.666270 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 262a3549867a473aaaa7dfcc0b5872cd (DEFAULT_TABLE table=default.Table_14af7742726b4562a02fd2c9b2895277 [id=6d72901e5102476890f466fd406a7127]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.666751 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 262a3549867a473aaaa7dfcc0b5872cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.670409 27213 tablet_service.cc:1468] Processing CreateTablet for tablet 44db427ee53a4d0a907dfd63282d6d4f (DEFAULT_TABLE table=default.Table_43bea7dca99143f5ac13671739868cdb [id=5c647f2080d043f3851e5767b1efd161]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.670837 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44db427ee53a4d0a907dfd63282d6d4f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:10.670855 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 0f4ef613b0ab4bca98294682b934f6ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f4ef613b0ab4bca98294682b934f6ed already in progress: creating tablet
I20250903 20:05:10.671350 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 43f88abc8f3640a48598c07c3d76755d (table default.Table_a5f8f31c12b742b597c8b35d662a607f [id=71d850a0cea14b0ba337a822ad99d513]) successfully deleted
I20250903 20:05:10.672317 27500 tablet_service.cc:1468] Processing CreateTablet for tablet 44db427ee53a4d0a907dfd63282d6d4f (DEFAULT_TABLE table=default.Table_43bea7dca99143f5ac13671739868cdb [id=5c647f2080d043f3851e5767b1efd161]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.672787 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44db427ee53a4d0a907dfd63282d6d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.674558 27366 tablet_service.cc:1468] Processing CreateTablet for tablet c9c870fb14e3469fb2979d8462b49121 (DEFAULT_TABLE table=default.Table_8b2ba396ac1b424493dbc17297d2d133 [id=3403a412096a4ef98878426a43899d7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.675104 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9c870fb14e3469fb2979d8462b49121. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.677067 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 73f65e7515994b898eec2ea23903aff4 (DEFAULT_TABLE table=default.Table_d5626bea3c88471b846f37ae4ed79715 [id=75e5cb0bab9b46a6b5caec7c99efc7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.677475 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73f65e7515994b898eec2ea23903aff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.678609 27364 tablet_service.cc:1468] Processing CreateTablet for tablet 73f65e7515994b898eec2ea23903aff4 (DEFAULT_TABLE table=default.Table_d5626bea3c88471b846f37ae4ed79715 [id=75e5cb0bab9b46a6b5caec7c99efc7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.678645 27224 tablet_service.cc:1468] Processing CreateTablet for tablet c9c870fb14e3469fb2979d8462b49121 (DEFAULT_TABLE table=default.Table_8b2ba396ac1b424493dbc17297d2d133 [id=3403a412096a4ef98878426a43899d7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.679047 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9c870fb14e3469fb2979d8462b49121. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.679206 27362 tablet_service.cc:1468] Processing CreateTablet for tablet 44db427ee53a4d0a907dfd63282d6d4f (DEFAULT_TABLE table=default.Table_43bea7dca99143f5ac13671739868cdb [id=5c647f2080d043f3851e5767b1efd161]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.684433 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73f65e7515994b898eec2ea23903aff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.685014 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44db427ee53a4d0a907dfd63282d6d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.685673 27503 tablet_service.cc:1468] Processing CreateTablet for tablet c9c870fb14e3469fb2979d8462b49121 (DEFAULT_TABLE table=default.Table_8b2ba396ac1b424493dbc17297d2d133 [id=3403a412096a4ef98878426a43899d7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.686082 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9c870fb14e3469fb2979d8462b49121. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.686780 27692 tablet_bootstrap.cc:492] T c12c957a36f643d9bfb60b755619cbbc P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:10.686930 27692 ts_tablet_manager.cc:1397] T c12c957a36f643d9bfb60b755619cbbc P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.082s user 0.009s sys 0.004s
I20250903 20:05:10.687029 27221 tablet_service.cc:1468] Processing CreateTablet for tablet 73f65e7515994b898eec2ea23903aff4 (DEFAULT_TABLE table=default.Table_d5626bea3c88471b846f37ae4ed79715 [id=75e5cb0bab9b46a6b5caec7c99efc7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.687412 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73f65e7515994b898eec2ea23903aff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.687736 27692 raft_consensus.cc:357] T c12c957a36f643d9bfb60b755619cbbc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.687930 27692 raft_consensus.cc:383] T c12c957a36f643d9bfb60b755619cbbc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.688021 27692 raft_consensus.cc:738] T c12c957a36f643d9bfb60b755619cbbc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.690713 27692 consensus_queue.cc:260] T c12c957a36f643d9bfb60b755619cbbc P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.691488 27692 ts_tablet_manager.cc:1428] T c12c957a36f643d9bfb60b755619cbbc P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:10.691835 27692 tablet_bootstrap.cc:492] T 1d9a2508437b41f08821a29f58163159 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:10.693675 27692 tablet_bootstrap.cc:654] T 1d9a2508437b41f08821a29f58163159 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:10.695986 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 0f4ef613b0ab4bca98294682b934f6ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f4ef613b0ab4bca98294682b934f6ed already in progress: creating tablet
I20250903 20:05:10.701716 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 4692dd93b76c42c4883c7e22c6032250
I20250903 20:05:10.703599 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_56d2ca07338b4ce4ad732c5b2401812b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:10.704154 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56d2ca07338b4ce4ad732c5b2401812b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.705263 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 3906285457f449e6824d1f044bffca24
I20250903 20:05:10.705976 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_3aff8eb0c058463b831146139686b58e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:10.706489 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3aff8eb0c058463b831146139686b58e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.707544 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet c9c870fb14e3469fb2979d8462b49121
I20250903 20:05:10.708164 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a3f1ce69cd8943b88954a6ded73a8713" } new_table_name: "default.Table_7325977ff92e45bab0b59b79579ecae4" modify_external_catalogs: true
I20250903 20:05:10.709496 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0c3cf60b884e47eaaa60b3e0e80d56a3" } modify_external_catalogs: true
I20250903 20:05:10.709669 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0c3cf60b884e47eaaa60b3e0e80d56a3" } modify_external_catalogs: true
I20250903 20:05:10.711069 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a9aa14a6383343bc83014581ac775319" } new_table_name: "default.Table_9691d111178f4b9c9024a0a66fc3f03b" modify_external_catalogs: true
I20250903 20:05:10.712785 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_18040dcdd59e4eaf92f9aa8748571a1c" } new_table_name: "default.Table_b520cd881f58424dadf3fa4a013812ee" modify_external_catalogs: true
I20250903 20:05:10.714011 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet be278233c7c34061988b267a44ee674f
I20250903 20:05:10.714538 27064 catalog_manager.cc:6652] ReplaceTablet: tablet be278233c7c34061988b267a44ee674f deleted and replaced by tablet 5bc3b9771a0042fd87e706204dcbb712
I20250903 20:05:10.714886 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2facdec2620546d5bf3dcdbaaf76a56e" } new_table_name: "default.Table_459903503b5148d09d3cb8f2f5519fab" modify_external_catalogs: true
I20250903 20:05:10.715898 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 76742d0df44c43729197f7e09f96cad8
I20250903 20:05:10.716378 27061 catalog_manager.cc:6652] ReplaceTablet: tablet 76742d0df44c43729197f7e09f96cad8 deleted and replaced by tablet 0940159eaea740f99be2b67d5abacd2d
I20250903 20:05:10.716681 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_54bf0e4b392545b08001021008e93985"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:10.717060 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54bf0e4b392545b08001021008e93985 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.723186 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet be278233c7c34061988b267a44ee674f with delete_type TABLET_DATA_DELETED (replaced by tablet 5bc3b9771a0042fd87e706204dcbb712) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:10.723482 27057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2a55e0f3a78d422ab61df9f9ddb6515b" } new_table_name: "default.Table_452fdb47678d4520828b9a3db6ddb237" modify_external_catalogs: true
I20250903 20:05:10.723186 27501 tablet_service.cc:1515] Processing DeleteTablet for tablet 76742d0df44c43729197f7e09f96cad8 with delete_type TABLET_DATA_DELETED (replaced by tablet 0940159eaea740f99be2b67d5abacd2d) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:10.728449 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 4692dd93b76c42c4883c7e22c6032250 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:10.728791 27366 tablet_service.cc:1515] Processing DeleteTablet for tablet 1d9a2508437b41f08821a29f58163159 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:10.729053 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet 3906285457f449e6824d1f044bffca24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:10.729107 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet c9c870fb14e3469fb2979d8462b49121 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:10.729269 27358 tablet_service.cc:1515] Processing DeleteTablet for tablet be278233c7c34061988b267a44ee674f with delete_type TABLET_DATA_DELETED (replaced by tablet 5bc3b9771a0042fd87e706204dcbb712) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:10.729460 27366 tablet_service.cc:1515] Processing DeleteTablet for tablet 76742d0df44c43729197f7e09f96cad8 with delete_type TABLET_DATA_DELETED (replaced by tablet 0940159eaea740f99be2b67d5abacd2d) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:10.729935 27687 tablet_replica.cc:331] T 1d9a2508437b41f08821a29f58163159 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:10.730166 27687 raft_consensus.cc:2241] T 1d9a2508437b41f08821a29f58163159 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.730396 27687 raft_consensus.cc:2270] T 1d9a2508437b41f08821a29f58163159 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.730654 27221 tablet_service.cc:1515] Processing DeleteTablet for tablet be278233c7c34061988b267a44ee674f with delete_type TABLET_DATA_DELETED (replaced by tablet 5bc3b9771a0042fd87e706204dcbb712) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:10.730800 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet be278233c7c34061988b267a44ee674f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be278233c7c34061988b267a44ee674f already in progress: creating tablet
W20250903 20:05:10.731040 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 76742d0df44c43729197f7e09f96cad8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76742d0df44c43729197f7e09f96cad8 already in progress: creating tablet
I20250903 20:05:10.731122 27501 tablet_service.cc:1515] Processing DeleteTablet for tablet c12c957a36f643d9bfb60b755619cbbc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:10.731269 27224 tablet_service.cc:1515] Processing DeleteTablet for tablet c9c870fb14e3469fb2979d8462b49121 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:10.731419 27684 tablet_replica.cc:331] T c12c957a36f643d9bfb60b755619cbbc P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:10.731520 27687 ts_tablet_manager.cc:1905] T 1d9a2508437b41f08821a29f58163159 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.731623 27684 raft_consensus.cc:2241] T c12c957a36f643d9bfb60b755619cbbc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.731807 27684 raft_consensus.cc:2270] T c12c957a36f643d9bfb60b755619cbbc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.731812 27213 tablet_service.cc:1515] Processing DeleteTablet for tablet 3906285457f449e6824d1f044bffca24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:10.732153 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3055928e679e4dd8b8559b363b580e13" } modify_external_catalogs: true
I20250903 20:05:10.732317 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 76742d0df44c43729197f7e09f96cad8 with delete_type TABLET_DATA_DELETED (replaced by tablet 0940159eaea740f99be2b67d5abacd2d) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:10.732355 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3055928e679e4dd8b8559b363b580e13" } modify_external_catalogs: true
W20250903 20:05:10.732568 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet be278233c7c34061988b267a44ee674f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be278233c7c34061988b267a44ee674f already in progress: creating tablet
W20250903 20:05:10.732825 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet c9c870fb14e3469fb2979d8462b49121 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9c870fb14e3469fb2979d8462b49121 already in progress: creating tablet
W20250903 20:05:10.733070 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 3906285457f449e6824d1f044bffca24 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3906285457f449e6824d1f044bffca24 already in progress: creating tablet
W20250903 20:05:10.733286 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 76742d0df44c43729197f7e09f96cad8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76742d0df44c43729197f7e09f96cad8 already in progress: creating tablet
W20250903 20:05:10.733484 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 4692dd93b76c42c4883c7e22c6032250 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4692dd93b76c42c4883c7e22c6032250 already in progress: creating tablet
I20250903 20:05:10.734275 27684 ts_tablet_manager.cc:1905] T c12c957a36f643d9bfb60b755619cbbc P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.739879 27687 ts_tablet_manager.cc:1918] T 1d9a2508437b41f08821a29f58163159 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.740136 27687 log.cc:1199] T 1d9a2508437b41f08821a29f58163159 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/1d9a2508437b41f08821a29f58163159
I20250903 20:05:10.740605 27687 ts_tablet_manager.cc:1939] T 1d9a2508437b41f08821a29f58163159 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:10.741940 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 1d9a2508437b41f08821a29f58163159 (table default.Table_c106f52e6b6b458391a12b3043e7eedf [id=cbae62c266f24526affd029472f41b94]) successfully deleted
I20250903 20:05:10.742566 27687 tablet_replica.cc:331] T 3906285457f449e6824d1f044bffca24 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:10.742806 27687 raft_consensus.cc:2241] T 3906285457f449e6824d1f044bffca24 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.743073 27687 raft_consensus.cc:2270] T 3906285457f449e6824d1f044bffca24 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.744004 27687 ts_tablet_manager.cc:1905] T 3906285457f449e6824d1f044bffca24 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.745865 27755 raft_consensus.cc:491] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:10.746052 27755 raft_consensus.cc:513] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.746820 27755 leader_election.cc:290] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:10.747470 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8a293c6fa034e9fb29682439cb3bcc2" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:10.748248 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8a293c6fa034e9fb29682439cb3bcc2" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:10.748467 27249 raft_consensus.cc:2466] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
W20250903 20:05:10.748989 27462 leader_election.cc:343] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:10.749337 27462 leader_election.cc:304] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:10.749992 27755 raft_consensus.cc:2802] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:10.750119 27755 raft_consensus.cc:491] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:10.750204 27755 raft_consensus.cc:3058] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:10.750154 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet f99c05e72b9a4ff9a1e7c269ca029b6a
W20250903 20:05:10.752279 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ba18e3f8cb148d484010cb4e72510b2 consensus state has no leader
W20250903 20:05:10.752497 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eced730bdbc14a8c9f6b7b7e53435fd0 has no consensus state
W20250903 20:05:10.752728 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5bc3b9771a0042fd87e706204dcbb712 has no consensus state
W20250903 20:05:10.750178 27066 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 99a21a3e49934a79a66700c0b0d86307 consensus state has no leader
I20250903 20:05:10.754034 27755 raft_consensus.cc:513] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.754469 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_26a46b6cf89f459b99526bc640729603"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:10.755174 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26a46b6cf89f459b99526bc640729603 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.755164 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8a293c6fa034e9fb29682439cb3bcc2" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:10.756692 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet 4692dd93b76c42c4883c7e22c6032250 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:10.758713 27366 tablet_service.cc:1515] Processing DeleteTablet for tablet f99c05e72b9a4ff9a1e7c269ca029b6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:10.759578 27695 tablet_bootstrap.cc:492] T c12c957a36f643d9bfb60b755619cbbc P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
W20250903 20:05:10.759708 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 4692dd93b76c42c4883c7e22c6032250 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4692dd93b76c42c4883c7e22c6032250 already in progress: creating tablet
I20250903 20:05:10.759774 27695 ts_tablet_manager.cc:1397] T c12c957a36f643d9bfb60b755619cbbc P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.213s user 0.023s sys 0.025s
I20250903 20:05:10.760697 27501 tablet_service.cc:1515] Processing DeleteTablet for tablet f99c05e72b9a4ff9a1e7c269ca029b6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:10.760663 27462 leader_election.cc:343] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:10.760797 27500 tablet_service.cc:1515] Processing DeleteTablet for tablet 3906285457f449e6824d1f044bffca24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:10.760637 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet c9c870fb14e3469fb2979d8462b49121 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:10.761081 27503 tablet_service.cc:1515] Processing DeleteTablet for tablet 4692dd93b76c42c4883c7e22c6032250 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:10.761771 27719 raft_consensus.cc:695] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:10.761946 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8a293c6fa034e9fb29682439cb3bcc2" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:10.762249 27249 raft_consensus.cc:3058] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:10.762177 27719 consensus_queue.cc:237] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.763147 27695 raft_consensus.cc:357] T c12c957a36f643d9bfb60b755619cbbc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.763387 27695 raft_consensus.cc:383] T c12c957a36f643d9bfb60b755619cbbc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.763490 27695 raft_consensus.cc:738] T c12c957a36f643d9bfb60b755619cbbc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.764782 27695 consensus_queue.cc:260] T c12c957a36f643d9bfb60b755619cbbc P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.765393 27695 ts_tablet_manager.cc:1428] T c12c957a36f643d9bfb60b755619cbbc P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20250903 20:05:10.765580 27249 raft_consensus.cc:2466] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:10.766350 27032 master_runner.cc:186] Error getting master registration for 127.26.65.126:39015: Network error: Client connection negotiation failed: client connection to 127.26.65.126:39015: connect: Connection refused (error 111),
I20250903 20:05:10.766533 27032 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250903 20:05:10.767010 27462 leader_election.cc:304] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:10.768301 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_091b5cecb0b04277a78dc078b4f800a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:10.768460 27755 leader_election.cc:290] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:10.769025 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_091b5cecb0b04277a78dc078b4f800a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.769295 27755 raft_consensus.cc:2802] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:10.769461 27755 raft_consensus.cc:695] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:10.770239 27755 consensus_queue.cc:237] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.770720 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet f99c05e72b9a4ff9a1e7c269ca029b6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:10.771021 27685 tablet_replica.cc:331] T f99c05e72b9a4ff9a1e7c269ca029b6a P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:10.771232 27685 raft_consensus.cc:2241] T f99c05e72b9a4ff9a1e7c269ca029b6a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.771420 27685 raft_consensus.cc:2270] T f99c05e72b9a4ff9a1e7c269ca029b6a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.772346 27685 ts_tablet_manager.cc:1905] T f99c05e72b9a4ff9a1e7c269ca029b6a P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.774492 27684 ts_tablet_manager.cc:1918] T c12c957a36f643d9bfb60b755619cbbc P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.774655 27684 log.cc:1199] T c12c957a36f643d9bfb60b755619cbbc P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/c12c957a36f643d9bfb60b755619cbbc
I20250903 20:05:10.775092 27684 ts_tablet_manager.cc:1939] T c12c957a36f643d9bfb60b755619cbbc P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
W20250903 20:05:10.775591 27057 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 710e3bc3c4194efbbeef6bc76ee64f24 consensus state has no leader
I20250903 20:05:10.776470 27058 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 0940159eaea740f99be2b67d5abacd2d
I20250903 20:05:10.779152 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet c12c957a36f643d9bfb60b755619cbbc (table default.Table_7d476d0d69e246e28664c7c5b84f3ba2 [id=7c51b3299c2948189100608937aa1ef9]) successfully deleted
W20250903 20:05:10.779693 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet f99c05e72b9a4ff9a1e7c269ca029b6a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f99c05e72b9a4ff9a1e7c269ca029b6a already in progress: creating tablet
W20250903 20:05:10.779934 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 3906285457f449e6824d1f044bffca24 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3906285457f449e6824d1f044bffca24 already in progress: creating tablet
W20250903 20:05:10.780151 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet c9c870fb14e3469fb2979d8462b49121 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9c870fb14e3469fb2979d8462b49121 already in progress: creating tablet
W20250903 20:05:10.780534 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 4692dd93b76c42c4883c7e22c6032250 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4692dd93b76c42c4883c7e22c6032250 already in progress: creating tablet
I20250903 20:05:10.781610 27685 ts_tablet_manager.cc:1918] T f99c05e72b9a4ff9a1e7c269ca029b6a P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.781802 27685 log.cc:1199] T f99c05e72b9a4ff9a1e7c269ca029b6a P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/f99c05e72b9a4ff9a1e7c269ca029b6a
I20250903 20:05:10.782349 27685 ts_tablet_manager.cc:1939] T f99c05e72b9a4ff9a1e7c269ca029b6a P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:10.783486 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet f99c05e72b9a4ff9a1e7c269ca029b6a (table default.Table_0c3cf60b884e47eaaa60b3e0e80d56a3 [id=0b57a09164704002b7fcd0b8f08da1e9]) successfully deleted
I20250903 20:05:10.786397 27695 tablet_bootstrap.cc:492] T 3906285457f449e6824d1f044bffca24 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:10.788642 27504 tablet_service.cc:1468] Processing CreateTablet for tablet eced730bdbc14a8c9f6b7b7e53435fd0 (DEFAULT_TABLE table=default.Table_b520cd881f58424dadf3fa4a013812ee [id=2c179236954b41cfa9242e16946ef9bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.789417 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eced730bdbc14a8c9f6b7b7e53435fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.789918 27695 tablet_bootstrap.cc:654] T 3906285457f449e6824d1f044bffca24 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:10.791596 27500 tablet_service.cc:1468] Processing CreateTablet for tablet 264127511a234e9fbf409f3aa2860ccc (DEFAULT_TABLE table=default.Table_9b700cf2d51b4fca81d158e22d72a3f3 [id=d20347a8196e44459448223a3af2dead]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.792171 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 264127511a234e9fbf409f3aa2860ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.792747 27066 catalog_manager.cc:5582] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a reported cstate change: term changed from 0 to 1, leader changed from <none> to 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65). New cstate: current_term: 1 leader_uuid: "89f8f0ee68b940db9ce250c0604f934a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: HEALTHY } } }
I20250903 20:05:10.793118 27692 tablet_bootstrap.cc:492] T 1d9a2508437b41f08821a29f58163159 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:10.793202 27066 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet fd499b81344c4b58bf77f3622422763c (table default.Table_e914faf335d44d019f91bda71758c473 [id=374b9abdf6a54b38876a96d8ec4a0368]). Expected version 4 got 0
I20250903 20:05:10.793324 27692 ts_tablet_manager.cc:1397] T 1d9a2508437b41f08821a29f58163159 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.102s user 0.006s sys 0.004s
I20250903 20:05:10.793593 27059 catalog_manager.cc:5582] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f reported cstate change: term changed from 0 to 1, leader changed from <none> to e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67). New cstate: current_term: 1 leader_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:10.794005 27059 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet c8a293c6fa034e9fb29682439cb3bcc2 (table default.Table_37e1d397cf99443d87211c3a8e1b9b82 [id=5cfc8827e4ca4f4a94424c48d6f71eb5]). Expected version 2 got 0
I20250903 20:05:10.795346 27503 tablet_service.cc:1468] Processing CreateTablet for tablet e8bc99de4ffe43d58351d170b7c61d8a (DEFAULT_TABLE table=default.Table_ffb4d9c1a2d945f1af90b41164c2875a [id=416ff29f66d14564826a6734dde1c604]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.795734 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8bc99de4ffe43d58351d170b7c61d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.795688 27692 raft_consensus.cc:357] T 1d9a2508437b41f08821a29f58163159 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:10.795894 27692 raft_consensus.cc:383] T 1d9a2508437b41f08821a29f58163159 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.795989 27692 raft_consensus.cc:738] T 1d9a2508437b41f08821a29f58163159 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.796231 27692 consensus_queue.cc:260] T 1d9a2508437b41f08821a29f58163159 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:10.796905 27692 ts_tablet_manager.cc:1428] T 1d9a2508437b41f08821a29f58163159 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20250903 20:05:10.797228 27692 tablet_bootstrap.cc:492] T d1e1add08013459ba0dab3b7e66f0136 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:10.798902 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 264127511a234e9fbf409f3aa2860ccc (DEFAULT_TABLE table=default.Table_9b700cf2d51b4fca81d158e22d72a3f3 [id=d20347a8196e44459448223a3af2dead]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.799156 27362 tablet_service.cc:1468] Processing CreateTablet for tablet 264127511a234e9fbf409f3aa2860ccc (DEFAULT_TABLE table=default.Table_9b700cf2d51b4fca81d158e22d72a3f3 [id=d20347a8196e44459448223a3af2dead]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.799316 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 264127511a234e9fbf409f3aa2860ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.799645 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 264127511a234e9fbf409f3aa2860ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.799602 27358 tablet_service.cc:1468] Processing CreateTablet for tablet eced730bdbc14a8c9f6b7b7e53435fd0 (DEFAULT_TABLE table=default.Table_b520cd881f58424dadf3fa4a013812ee [id=2c179236954b41cfa9242e16946ef9bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.799154 27366 tablet_service.cc:1468] Processing CreateTablet for tablet e8bc99de4ffe43d58351d170b7c61d8a (DEFAULT_TABLE table=default.Table_ffb4d9c1a2d945f1af90b41164c2875a [id=416ff29f66d14564826a6734dde1c604]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.799975 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eced730bdbc14a8c9f6b7b7e53435fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.800199 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8bc99de4ffe43d58351d170b7c61d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.802564 27226 tablet_service.cc:1468] Processing CreateTablet for tablet eced730bdbc14a8c9f6b7b7e53435fd0 (DEFAULT_TABLE table=default.Table_b520cd881f58424dadf3fa4a013812ee [id=2c179236954b41cfa9242e16946ef9bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.802994 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eced730bdbc14a8c9f6b7b7e53435fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.803941 27058 master_service.cc:948] ReplaceTablet: received request to replace tablet 59e120e5e39e4de8a239f13545e4b1af from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:10.804606 27057 master_service.cc:948] ReplaceTablet: received request to replace tablet f7ab722c37264973b204cd4f1f95052d from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:10.809492 27220 tablet_service.cc:1468] Processing CreateTablet for tablet e8bc99de4ffe43d58351d170b7c61d8a (DEFAULT_TABLE table=default.Table_ffb4d9c1a2d945f1af90b41164c2875a [id=416ff29f66d14564826a6734dde1c604]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.809926 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8bc99de4ffe43d58351d170b7c61d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.818789 27692 tablet_bootstrap.cc:654] T d1e1add08013459ba0dab3b7e66f0136 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:10.820621 27695 tablet_bootstrap.cc:492] T 3906285457f449e6824d1f044bffca24 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:10.820794 27695 ts_tablet_manager.cc:1397] T 3906285457f449e6824d1f044bffca24 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.055s user 0.001s sys 0.003s
I20250903 20:05:10.821579 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet f7ab722c37264973b204cd4f1f95052d
I20250903 20:05:10.821718 27695 raft_consensus.cc:357] T 3906285457f449e6824d1f044bffca24 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.821908 27695 raft_consensus.cc:383] T 3906285457f449e6824d1f044bffca24 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.822001 27695 raft_consensus.cc:738] T 3906285457f449e6824d1f044bffca24 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.822119 27057 catalog_manager.cc:6652] ReplaceTablet: tablet f7ab722c37264973b204cd4f1f95052d deleted and replaced by tablet 57a1167457b84ea8a493be232b1d088a
I20250903 20:05:10.822378 27695 consensus_queue.cc:260] T 3906285457f449e6824d1f044bffca24 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.822932 27695 ts_tablet_manager.cc:1428] T 3906285457f449e6824d1f044bffca24 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250903 20:05:10.823228 27695 tablet_bootstrap.cc:492] T 0a266b1bbf1540ac86d49c56df2dd4ad P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:10.824427 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet f7ab722c37264973b204cd4f1f95052d with delete_type TABLET_DATA_DELETED (replaced by tablet 57a1167457b84ea8a493be232b1d088a) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:10.825930 27695 tablet_bootstrap.cc:654] T 0a266b1bbf1540ac86d49c56df2dd4ad P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:10.826787 27501 tablet_service.cc:1515] Processing DeleteTablet for tablet 1d9a2508437b41f08821a29f58163159 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:10.828166 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet f7ab722c37264973b204cd4f1f95052d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f7ab722c37264973b204cd4f1f95052d already in progress: creating tablet
I20250903 20:05:10.828508 27684 tablet_replica.cc:331] T 1d9a2508437b41f08821a29f58163159 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:10.828706 27684 raft_consensus.cc:2241] T 1d9a2508437b41f08821a29f58163159 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.828907 27684 raft_consensus.cc:2270] T 1d9a2508437b41f08821a29f58163159 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.829658 27684 ts_tablet_manager.cc:1905] T 1d9a2508437b41f08821a29f58163159 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.833554 27687 ts_tablet_manager.cc:1918] T 3906285457f449e6824d1f044bffca24 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.834389 27223 tablet_service.cc:1515] Processing DeleteTablet for tablet 3906285457f449e6824d1f044bffca24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:10.834808 27692 tablet_bootstrap.cc:492] T d1e1add08013459ba0dab3b7e66f0136 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:10.834987 27692 ts_tablet_manager.cc:1397] T d1e1add08013459ba0dab3b7e66f0136 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20250903 20:05:10.836148 27692 raft_consensus.cc:357] T d1e1add08013459ba0dab3b7e66f0136 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.836372 27692 raft_consensus.cc:383] T d1e1add08013459ba0dab3b7e66f0136 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.837699 27692 raft_consensus.cc:738] T d1e1add08013459ba0dab3b7e66f0136 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.839149 27685 tablet_replica.cc:331] T 3906285457f449e6824d1f044bffca24 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:10.835302 27693 tablet_bootstrap.cc:492] T f99c05e72b9a4ff9a1e7c269ca029b6a P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:10.839399 27066 master_service.cc:948] ReplaceTablet: received request to replace tablet c25aaa6ff310432bb27a0695394ec420 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:10.839613 27693 ts_tablet_manager.cc:1397] T f99c05e72b9a4ff9a1e7c269ca029b6a P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.212s user 0.016s sys 0.019s
I20250903 20:05:10.839828 27684 ts_tablet_manager.cc:1918] T 1d9a2508437b41f08821a29f58163159 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.839962 27684 log.cc:1199] T 1d9a2508437b41f08821a29f58163159 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/1d9a2508437b41f08821a29f58163159
I20250903 20:05:10.840169 27685 raft_consensus.cc:2241] T 3906285457f449e6824d1f044bffca24 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.840382 27684 ts_tablet_manager.cc:1939] T 1d9a2508437b41f08821a29f58163159 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:10.840710 27685 raft_consensus.cc:2270] T 3906285457f449e6824d1f044bffca24 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.835891 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet f7ab722c37264973b204cd4f1f95052d with delete_type TABLET_DATA_DELETED (replaced by tablet 57a1167457b84ea8a493be232b1d088a) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:10.841060 27693 raft_consensus.cc:357] T f99c05e72b9a4ff9a1e7c269ca029b6a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.841315 27693 raft_consensus.cc:383] T f99c05e72b9a4ff9a1e7c269ca029b6a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.841421 27693 raft_consensus.cc:738] T f99c05e72b9a4ff9a1e7c269ca029b6a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.841526 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 1d9a2508437b41f08821a29f58163159 (table default.Table_c106f52e6b6b458391a12b3043e7eedf [id=cbae62c266f24526affd029472f41b94]) successfully deleted
I20250903 20:05:10.841743 27693 consensus_queue.cc:260] T f99c05e72b9a4ff9a1e7c269ca029b6a P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.841876 27692 consensus_queue.cc:260] T d1e1add08013459ba0dab3b7e66f0136 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.842631 27685 ts_tablet_manager.cc:1905] T 3906285457f449e6824d1f044bffca24 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.843282 27692 ts_tablet_manager.cc:1428] T d1e1add08013459ba0dab3b7e66f0136 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250903 20:05:10.843720 27693 ts_tablet_manager.cc:1428] T f99c05e72b9a4ff9a1e7c269ca029b6a P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:10.842875 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet c12c957a36f643d9bfb60b755619cbbc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:10.844030 27693 tablet_bootstrap.cc:492] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:10.844472 27692 tablet_bootstrap.cc:492] T 0a266b1bbf1540ac86d49c56df2dd4ad P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:10.847595 27500 tablet_service.cc:1515] Processing DeleteTablet for tablet d1e1add08013459ba0dab3b7e66f0136 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:10.847940 27693 tablet_bootstrap.cc:654] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:10.848026 27692 tablet_bootstrap.cc:654] T 0a266b1bbf1540ac86d49c56df2dd4ad P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:10.848680 27684 tablet_replica.cc:331] T d1e1add08013459ba0dab3b7e66f0136 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:10.848883 27684 raft_consensus.cc:2241] T d1e1add08013459ba0dab3b7e66f0136 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.849068 27684 raft_consensus.cc:2270] T d1e1add08013459ba0dab3b7e66f0136 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.851469 27695 tablet_bootstrap.cc:492] T 0a266b1bbf1540ac86d49c56df2dd4ad P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:10.851663 27695 ts_tablet_manager.cc:1397] T 0a266b1bbf1540ac86d49c56df2dd4ad P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.003s
I20250903 20:05:10.851876 27687 log.cc:1199] T 3906285457f449e6824d1f044bffca24 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/3906285457f449e6824d1f044bffca24
I20250903 20:05:10.852559 27695 raft_consensus.cc:357] T 0a266b1bbf1540ac86d49c56df2dd4ad P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.852936 27695 raft_consensus.cc:383] T 0a266b1bbf1540ac86d49c56df2dd4ad P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.853018 27695 raft_consensus.cc:738] T 0a266b1bbf1540ac86d49c56df2dd4ad P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.852645 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet f7ab722c37264973b204cd4f1f95052d with delete_type TABLET_DATA_DELETED (replaced by tablet 57a1167457b84ea8a493be232b1d088a) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:10.853695 27687 ts_tablet_manager.cc:1939] T 3906285457f449e6824d1f044bffca24 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:10.853284 27695 consensus_queue.cc:260] T 0a266b1bbf1540ac86d49c56df2dd4ad P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.854635 27695 ts_tablet_manager.cc:1428] T 0a266b1bbf1540ac86d49c56df2dd4ad P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20250903 20:05:10.855005 27695 tablet_bootstrap.cc:492] T 0f4ef613b0ab4bca98294682b934f6ed P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:10.856130 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 3906285457f449e6824d1f044bffca24 (table default.Table_264581b9b25640c58ed0dd863bf6bb8e [id=b35d1fca6bcd4bb6a1581c2c81cac712]) successfully deleted
I20250903 20:05:10.856626 27684 ts_tablet_manager.cc:1905] T d1e1add08013459ba0dab3b7e66f0136 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:10.859311 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet c9c870fb14e3469fb2979d8462b49121 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9c870fb14e3469fb2979d8462b49121 already in progress: creating tablet
I20250903 20:05:10.860235 27362 tablet_service.cc:1468] Processing CreateTablet for tablet a10b079e7d5a440580ac00195fe88c65 (DEFAULT_TABLE table=default.Table_56d2ca07338b4ce4ad732c5b2401812b [id=6d6f5e8cadf9445d8695c075cfabb2ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.860671 27695 tablet_bootstrap.cc:654] T 0f4ef613b0ab4bca98294682b934f6ed P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:10.860831 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a10b079e7d5a440580ac00195fe88c65. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:10.863942 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet be278233c7c34061988b267a44ee674f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be278233c7c34061988b267a44ee674f already in progress: creating tablet
I20250903 20:05:10.865862 27500 tablet_service.cc:1468] Processing CreateTablet for tablet a10b079e7d5a440580ac00195fe88c65 (DEFAULT_TABLE table=default.Table_56d2ca07338b4ce4ad732c5b2401812b [id=6d6f5e8cadf9445d8695c075cfabb2ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.866407 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a10b079e7d5a440580ac00195fe88c65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.866467 27358 tablet_service.cc:1468] Processing CreateTablet for tablet fdc77810f27a40758e063b8b1150ab71 (DEFAULT_TABLE table=default.Table_9c5c22a18c1d4913abdfab7d6a589058 [id=84da55ea3b4946f0b87d8d6389c63403]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.866888 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdc77810f27a40758e063b8b1150ab71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.867133 27223 tablet_service.cc:1468] Processing CreateTablet for tablet fdc77810f27a40758e063b8b1150ab71 (DEFAULT_TABLE table=default.Table_9c5c22a18c1d4913abdfab7d6a589058 [id=84da55ea3b4946f0b87d8d6389c63403]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.867229 27214 tablet_service.cc:1468] Processing CreateTablet for tablet a10b079e7d5a440580ac00195fe88c65 (DEFAULT_TABLE table=default.Table_56d2ca07338b4ce4ad732c5b2401812b [id=6d6f5e8cadf9445d8695c075cfabb2ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.867579 27223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdc77810f27a40758e063b8b1150ab71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.867930 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a10b079e7d5a440580ac00195fe88c65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.869719 27366 tablet_service.cc:1468] Processing CreateTablet for tablet ecd339ec12e84e88acec427630cc8668 (DEFAULT_TABLE table=default.Table_54bf0e4b392545b08001021008e93985 [id=a803b1a5d2404f74aa4671eebd23e8ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.870146 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecd339ec12e84e88acec427630cc8668. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.871230 27687 tablet_replica.cc:331] T f99c05e72b9a4ff9a1e7c269ca029b6a P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:10.872014 27687 raft_consensus.cc:2241] T f99c05e72b9a4ff9a1e7c269ca029b6a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.872251 27501 tablet_service.cc:1468] Processing CreateTablet for tablet ecd339ec12e84e88acec427630cc8668 (DEFAULT_TABLE table=default.Table_54bf0e4b392545b08001021008e93985 [id=a803b1a5d2404f74aa4671eebd23e8ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.872695 27501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecd339ec12e84e88acec427630cc8668. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.876299 27687 raft_consensus.cc:2270] T f99c05e72b9a4ff9a1e7c269ca029b6a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.878278 27504 tablet_service.cc:1468] Processing CreateTablet for tablet fdc77810f27a40758e063b8b1150ab71 (DEFAULT_TABLE table=default.Table_9c5c22a18c1d4913abdfab7d6a589058 [id=84da55ea3b4946f0b87d8d6389c63403]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.879215 27693 tablet_bootstrap.cc:492] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:10.879366 27693 ts_tablet_manager.cc:1397] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.007s
I20250903 20:05:10.879734 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdc77810f27a40758e063b8b1150ab71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.880164 27223 tablet_service.cc:1468] Processing CreateTablet for tablet ecd339ec12e84e88acec427630cc8668 (DEFAULT_TABLE table=default.Table_54bf0e4b392545b08001021008e93985 [id=a803b1a5d2404f74aa4671eebd23e8ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.880499 27687 ts_tablet_manager.cc:1905] T f99c05e72b9a4ff9a1e7c269ca029b6a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.880643 27223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecd339ec12e84e88acec427630cc8668. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.883895 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet 0a266b1bbf1540ac86d49c56df2dd4ad with delete_type TABLET_DATA_DELETED (replaced by tablet 73f65e7515994b898eec2ea23903aff4) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:10.884907 27249 raft_consensus.cc:1273] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Refusing update from remote peer e72ef1dbcc5143cf9f04a808f101c80f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 20:05:10.885802 27755 consensus_queue.cc:1035] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:10.886763 27220 tablet_service.cc:1468] Processing CreateTablet for tablet 6ea37668d64d4feeb92a162711609c13 (DEFAULT_TABLE table=default.Table_cc799e62ff6b47e1af28b7006551bbf5 [id=d97016a400e34272830add1e220c65ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.887171 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ea37668d64d4feeb92a162711609c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.888831 27226 tablet_service.cc:1468] Processing CreateTablet for tablet deb84fc3465443d99c11097e53f5f575 (DEFAULT_TABLE table=default.Table_a642c53ef0da461ba2f33dd4010e7129 [id=95919dc5590942bd8e47047598b6d6bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.890177 27692 tablet_bootstrap.cc:492] T 0a266b1bbf1540ac86d49c56df2dd4ad P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:10.890383 27692 ts_tablet_manager.cc:1397] T 0a266b1bbf1540ac86d49c56df2dd4ad P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.047s user 0.005s sys 0.000s
I20250903 20:05:10.891083 27503 tablet_service.cc:1468] Processing CreateTablet for tablet 6ea37668d64d4feeb92a162711609c13 (DEFAULT_TABLE table=default.Table_cc799e62ff6b47e1af28b7006551bbf5 [id=d97016a400e34272830add1e220c65ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.892561 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ea37668d64d4feeb92a162711609c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.894613 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet c25aaa6ff310432bb27a0695394ec420
I20250903 20:05:10.894659 27684 ts_tablet_manager.cc:1918] T d1e1add08013459ba0dab3b7e66f0136 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.894790 27684 log.cc:1199] T d1e1add08013459ba0dab3b7e66f0136 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/d1e1add08013459ba0dab3b7e66f0136
I20250903 20:05:10.895162 27684 ts_tablet_manager.cc:1939] T d1e1add08013459ba0dab3b7e66f0136 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:10.895669 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet c25aaa6ff310432bb27a0695394ec420 with delete_type TABLET_DATA_DELETED (replaced by tablet 3f78a951bb024cf290116b0fe438ddb4) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:10.891220 27692 raft_consensus.cc:357] T 0a266b1bbf1540ac86d49c56df2dd4ad P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.895917 27066 catalog_manager.cc:6652] ReplaceTablet: tablet c25aaa6ff310432bb27a0695394ec420 deleted and replaced by tablet 3f78a951bb024cf290116b0fe438ddb4
I20250903 20:05:10.896030 27692 raft_consensus.cc:383] T 0a266b1bbf1540ac86d49c56df2dd4ad P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.896164 27692 raft_consensus.cc:738] T 0a266b1bbf1540ac86d49c56df2dd4ad P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.891721 27501 tablet_service.cc:1468] Processing CreateTablet for tablet deb84fc3465443d99c11097e53f5f575 (DEFAULT_TABLE table=default.Table_a642c53ef0da461ba2f33dd4010e7129 [id=95919dc5590942bd8e47047598b6d6bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.896560 27692 consensus_queue.cc:260] T 0a266b1bbf1540ac86d49c56df2dd4ad P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.899482 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet d1e1add08013459ba0dab3b7e66f0136 (table default.Table_4ace9616711f4778a94a0827ff79cc07 [id=e20fa37b0e0343ed8ab3a7e814358ae6]) successfully deleted
W20250903 20:05:10.900435 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet c25aaa6ff310432bb27a0695394ec420 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c25aaa6ff310432bb27a0695394ec420 already in progress: creating tablet
I20250903 20:05:10.896996 27501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deb84fc3465443d99c11097e53f5f575. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.897382 27692 ts_tablet_manager.cc:1428] T 0a266b1bbf1540ac86d49c56df2dd4ad P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250903 20:05:10.901626 27692 tablet_bootstrap.cc:492] T f99c05e72b9a4ff9a1e7c269ca029b6a P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:10.901510 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 0a266b1bbf1540ac86d49c56df2dd4ad with delete_type TABLET_DATA_DELETED (replaced by tablet 73f65e7515994b898eec2ea23903aff4) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:10.904313 27692 tablet_bootstrap.cc:654] T f99c05e72b9a4ff9a1e7c269ca029b6a P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:10.896579 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet c25aaa6ff310432bb27a0695394ec420 with delete_type TABLET_DATA_DELETED (replaced by tablet 3f78a951bb024cf290116b0fe438ddb4) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:10.909049 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deb84fc3465443d99c11097e53f5f575. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.909876 27684 tablet_replica.cc:331] T 0a266b1bbf1540ac86d49c56df2dd4ad P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:10.910110 27684 raft_consensus.cc:2241] T 0a266b1bbf1540ac86d49c56df2dd4ad P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.910310 27685 ts_tablet_manager.cc:1918] T 3906285457f449e6824d1f044bffca24 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.910514 27685 log.cc:1199] T 3906285457f449e6824d1f044bffca24 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/3906285457f449e6824d1f044bffca24
I20250903 20:05:10.910938 27685 ts_tablet_manager.cc:1939] T 3906285457f449e6824d1f044bffca24 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:10.911494 27684 raft_consensus.cc:2270] T 0a266b1bbf1540ac86d49c56df2dd4ad P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.913053 27684 ts_tablet_manager.cc:1905] T 0a266b1bbf1540ac86d49c56df2dd4ad P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.914523 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 5bc3b9771a0042fd87e706204dcbb712 (DEFAULT_TABLE table=default.Table_459903503b5148d09d3cb8f2f5519fab [id=3c69612aaf164dc899dcc252d85636f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.914978 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bc3b9771a0042fd87e706204dcbb712. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.915227 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 5bc3b9771a0042fd87e706204dcbb712 (DEFAULT_TABLE table=default.Table_459903503b5148d09d3cb8f2f5519fab [id=3c69612aaf164dc899dcc252d85636f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.916025 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bc3b9771a0042fd87e706204dcbb712. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.916085 27225 tablet_service.cc:1468] Processing CreateTablet for tablet c9129a3e773a4fecb28ceee27778a294 (DEFAULT_TABLE table=default.Table_b2824f7f5f324e6ebcacb1b0645de8ec [id=50898d980302442fbd70cb88f8f4faba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.916478 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 3906285457f449e6824d1f044bffca24 (table default.Table_264581b9b25640c58ed0dd863bf6bb8e [id=b35d1fca6bcd4bb6a1581c2c81cac712]) successfully deleted
I20250903 20:05:10.916859 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9129a3e773a4fecb28ceee27778a294. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.917100 27685 tablet_replica.cc:331] T c12c957a36f643d9bfb60b755619cbbc P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:10.917307 27685 raft_consensus.cc:2241] T c12c957a36f643d9bfb60b755619cbbc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:10.917519 27685 raft_consensus.cc:2270] T c12c957a36f643d9bfb60b755619cbbc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:10.919339 27500 tablet_service.cc:1468] Processing CreateTablet for tablet c9129a3e773a4fecb28ceee27778a294 (DEFAULT_TABLE table=default.Table_b2824f7f5f324e6ebcacb1b0645de8ec [id=50898d980302442fbd70cb88f8f4faba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.919828 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9129a3e773a4fecb28ceee27778a294. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.923074 27685 ts_tablet_manager.cc:1905] T c12c957a36f643d9bfb60b755619cbbc P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:10.924043 27059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_cc799e62ff6b47e1af28b7006551bbf5" } modify_external_catalogs: true
I20250903 20:05:10.924285 27059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_cc799e62ff6b47e1af28b7006551bbf5" } modify_external_catalogs: true
I20250903 20:05:10.924731 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_80ad865643824ab88e2d63eb1b66d079"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:10.924906 27064 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_afc0f683c5d8460484160a3a9c3e6df5" } modify_external_catalogs: true
I20250903 20:05:10.925061 27064 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_afc0f683c5d8460484160a3a9c3e6df5" } modify_external_catalogs: true
W20250903 20:05:10.925423 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80ad865643824ab88e2d63eb1b66d079 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.926509 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_3f5617eea30e4038945f21ba8ca6be65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:10.926977 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f5617eea30e4038945f21ba8ca6be65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.928069 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e822637ebd11495cbb3c65907c6a7c78"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:10.928557 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e822637ebd11495cbb3c65907c6a7c78 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.924733 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b0b8582c6c054baab4688a30cfe5f767"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:10.929118 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0b8582c6c054baab4688a30cfe5f767 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.932121 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_d8651e0216bf42e08446c9cdc2ed5b0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:10.932771 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8651e0216bf42e08446c9cdc2ed5b0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.936080 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_9691d111178f4b9c9024a0a66fc3f03b" } modify_external_catalogs: true
I20250903 20:05:10.936385 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_9691d111178f4b9c9024a0a66fc3f03b" } modify_external_catalogs: true
I20250903 20:05:10.943711 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 6ea37668d64d4feeb92a162711609c13
I20250903 20:05:10.944525 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 79a0553c912049cca5971b1ad6f7e11c
I20250903 20:05:10.947028 27502 tablet_service.cc:1468] Processing CreateTablet for tablet 8b3121fce1c44d16a0b083313dc67b7a (DEFAULT_TABLE table=default.Table_8266aa4c05d94b34ad3e45b3035e3039 [id=158a2fc69fce4c54917277cc90439126]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.947453 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b3121fce1c44d16a0b083313dc67b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.947671 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 1fef9400227449ed892daff588999caa (DEFAULT_TABLE table=default.Table_3aff8eb0c058463b831146139686b58e [id=5568f788d1414c83b4359b5adf28632a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.948056 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fef9400227449ed892daff588999caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.952082 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_232f4e198e7746f8a49e0481f1166138"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:10.952622 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_232f4e198e7746f8a49e0481f1166138 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.952857 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ea37668d64d4feeb92a162711609c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:10.954912 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_983904dd0bb4486493150ce49e43a229"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:10.955513 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_983904dd0bb4486493150ce49e43a229 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.955488 27684 ts_tablet_manager.cc:1918] T 0a266b1bbf1540ac86d49c56df2dd4ad P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:10.955644 27684 log.cc:1199] T 0a266b1bbf1540ac86d49c56df2dd4ad P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/0a266b1bbf1540ac86d49c56df2dd4ad
I20250903 20:05:10.955965 27684 ts_tablet_manager.cc:1939] T 0a266b1bbf1540ac86d49c56df2dd4ad P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:10.956248 27213 tablet_service.cc:1468] Processing CreateTablet for tablet 8b3121fce1c44d16a0b083313dc67b7a (DEFAULT_TABLE table=default.Table_8266aa4c05d94b34ad3e45b3035e3039 [id=158a2fc69fce4c54917277cc90439126]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.957499 27692 tablet_bootstrap.cc:492] T f99c05e72b9a4ff9a1e7c269ca029b6a P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:10.957664 27692 ts_tablet_manager.cc:1397] T f99c05e72b9a4ff9a1e7c269ca029b6a P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.056s user 0.008s sys 0.005s
I20250903 20:05:10.957793 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet 79a0553c912049cca5971b1ad6f7e11c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:10.958452 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 0a266b1bbf1540ac86d49c56df2dd4ad (table default.Table_d5626bea3c88471b846f37ae4ed79715 [id=75e5cb0bab9b46a6b5caec7c99efc7b8]) successfully deleted
I20250903 20:05:10.958534 27692 raft_consensus.cc:357] T f99c05e72b9a4ff9a1e7c269ca029b6a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.958750 27692 raft_consensus.cc:383] T f99c05e72b9a4ff9a1e7c269ca029b6a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.959169 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_8266aa4c05d94b34ad3e45b3035e3039" } new_table_name: "default.Table_41538df113834b45ae2213b105fb9706" modify_external_catalogs: true
I20250903 20:05:10.959415 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c557d48ec717410fb860dfdc7ce9957b" } modify_external_catalogs: true
I20250903 20:05:10.959609 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c557d48ec717410fb860dfdc7ce9957b" } modify_external_catalogs: true
I20250903 20:05:10.961009 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b3121fce1c44d16a0b083313dc67b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.958842 27692 raft_consensus.cc:738] T f99c05e72b9a4ff9a1e7c269ca029b6a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.963585 27692 consensus_queue.cc:260] T f99c05e72b9a4ff9a1e7c269ca029b6a P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.965104 27692 ts_tablet_manager.cc:1428] T f99c05e72b9a4ff9a1e7c269ca029b6a P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250903 20:05:10.965265 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ea37668d64d4feeb92a162711609c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:10.966248 27692 tablet_bootstrap.cc:492] T 3906285457f449e6824d1f044bffca24 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
W20250903 20:05:10.966410 27462 consensus_peers.cc:489] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:10.968473 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet c25aaa6ff310432bb27a0695394ec420 with delete_type TABLET_DATA_DELETED (replaced by tablet 3f78a951bb024cf290116b0fe438ddb4) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:10.968530 27692 tablet_bootstrap.cc:654] T 3906285457f449e6824d1f044bffca24 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:10.968817 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 5bc3b9771a0042fd87e706204dcbb712 (DEFAULT_TABLE table=default.Table_459903503b5148d09d3cb8f2f5519fab [id=3c69612aaf164dc899dcc252d85636f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.969406 27363 tablet_service.cc:1468] Processing CreateTablet for tablet c9129a3e773a4fecb28ceee27778a294 (DEFAULT_TABLE table=default.Table_b2824f7f5f324e6ebcacb1b0645de8ec [id=50898d980302442fbd70cb88f8f4faba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.969743 27366 tablet_service.cc:1468] Processing CreateTablet for tablet deb84fc3465443d99c11097e53f5f575 (DEFAULT_TABLE table=default.Table_a642c53ef0da461ba2f33dd4010e7129 [id=95919dc5590942bd8e47047598b6d6bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.969926 27358 tablet_service.cc:1468] Processing CreateTablet for tablet 6ea37668d64d4feeb92a162711609c13 (DEFAULT_TABLE table=default.Table_cc799e62ff6b47e1af28b7006551bbf5 [id=d97016a400e34272830add1e220c65ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.970641 27693 raft_consensus.cc:357] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.970783 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 79a0553c912049cca5971b1ad6f7e11c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:10.970881 27693 raft_consensus.cc:383] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.970992 27692 tablet_bootstrap.cc:492] T 3906285457f449e6824d1f044bffca24 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:10.970991 27693 raft_consensus.cc:738] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.971027 27359 tablet_service.cc:1468] Processing CreateTablet for tablet 1fef9400227449ed892daff588999caa (DEFAULT_TABLE table=default.Table_3aff8eb0c058463b831146139686b58e [id=5568f788d1414c83b4359b5adf28632a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.971156 27692 ts_tablet_manager.cc:1397] T 3906285457f449e6824d1f044bffca24 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20250903 20:05:10.971310 27693 consensus_queue.cc:260] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.971585 27357 tablet_service.cc:1468] Processing CreateTablet for tablet 8b3121fce1c44d16a0b083313dc67b7a (DEFAULT_TABLE table=default.Table_8266aa4c05d94b34ad3e45b3035e3039 [id=158a2fc69fce4c54917277cc90439126]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.972273 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bc3b9771a0042fd87e706204dcbb712. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.972939 27693 ts_tablet_manager.cc:1428] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.093s user 0.001s sys 0.002s
I20250903 20:05:10.973400 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ea37668d64d4feeb92a162711609c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.973394 27692 raft_consensus.cc:357] T 3906285457f449e6824d1f044bffca24 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.973619 27692 raft_consensus.cc:383] T 3906285457f449e6824d1f044bffca24 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:10.973691 27692 raft_consensus.cc:738] T 3906285457f449e6824d1f044bffca24 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:10.973754 27693 tablet_bootstrap.cc:492] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:10.974345 27692 consensus_queue.cc:260] T 3906285457f449e6824d1f044bffca24 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:10.976109 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 1fef9400227449ed892daff588999caa (DEFAULT_TABLE table=default.Table_3aff8eb0c058463b831146139686b58e [id=5568f788d1414c83b4359b5adf28632a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:10.976536 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fef9400227449ed892daff588999caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.977625 27692 ts_tablet_manager.cc:1428] T 3906285457f449e6824d1f044bffca24 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250903 20:05:10.980829 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e3f5c55d553840099371cad3a73e2ca5" } new_table_name: "default.Table_943bea896c744410a2a4d80ffb9ca439" modify_external_catalogs: true
I20250903 20:05:10.981138 27058 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet b6b4713936134681aaef1732d46c6ab2 (table default.Table_b6f25e1f7a484577a9cbff11e0deb857 [id=d0a75545b91c48bba47fb581bb0d1a96]). Expected version 1 got 0
I20250903 20:05:10.981184 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_aab58ad5394041179974552994469305"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:10.981777 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aab58ad5394041179974552994469305 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:10.981886 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f4196ed1d25c4d6fab512014504c51af" } new_table_name: "default.Table_1d5b22110e4b45c193eb0faaae131c67" modify_external_catalogs: true
I20250903 20:05:10.982611 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b4bd57e0d5384edaa027cfcdf1b1db23" } new_table_name: "default.Table_b88b7b0fe749402e90c3826d01e8cc8e" modify_external_catalogs: true
I20250903 20:05:10.984158 27692 tablet_bootstrap.cc:492] T 0062307c6ee54aada74a08cf0d087f17 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:10.987432 27695 tablet_bootstrap.cc:492] T 0f4ef613b0ab4bca98294682b934f6ed P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:10.987608 27695 ts_tablet_manager.cc:1397] T 0f4ef613b0ab4bca98294682b934f6ed P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.133s user 0.014s sys 0.004s
I20250903 20:05:10.987692 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9129a3e773a4fecb28ceee27778a294. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.990447 27717 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 20:05:10.991706 27717 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 20:05:10.992048 27717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 20:05:10.993378 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deb84fc3465443d99c11097e53f5f575. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:10.994766 27503 tablet_service.cc:1515] Processing DeleteTablet for tablet 79a0553c912049cca5971b1ad6f7e11c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:10.995878 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 79a0553c912049cca5971b1ad6f7e11c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79a0553c912049cca5971b1ad6f7e11c already in progress: creating tablet
I20250903 20:05:10.998368 27500 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ea37668d64d4feeb92a162711609c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:10.999116 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 6ea37668d64d4feeb92a162711609c13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ea37668d64d4feeb92a162711609c13 already in progress: creating tablet
I20250903 20:05:11.003304 27695 raft_consensus.cc:357] T 0f4ef613b0ab4bca98294682b934f6ed P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.003515 27695 raft_consensus.cc:383] T 0f4ef613b0ab4bca98294682b934f6ed P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.003618 27695 raft_consensus.cc:738] T 0f4ef613b0ab4bca98294682b934f6ed P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.003855 27695 consensus_queue.cc:260] T 0f4ef613b0ab4bca98294682b934f6ed P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.004223 27692 tablet_bootstrap.cc:654] T 0062307c6ee54aada74a08cf0d087f17 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.004433 27695 ts_tablet_manager.cc:1428] T 0f4ef613b0ab4bca98294682b934f6ed P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20250903 20:05:11.004803 27695 tablet_bootstrap.cc:492] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:11.011739 27685 ts_tablet_manager.cc:1918] T c12c957a36f643d9bfb60b755619cbbc P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:11.011935 27685 log.cc:1199] T c12c957a36f643d9bfb60b755619cbbc P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/c12c957a36f643d9bfb60b755619cbbc
I20250903 20:05:11.012312 27685 ts_tablet_manager.cc:1939] T c12c957a36f643d9bfb60b755619cbbc P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:11.013913 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet c12c957a36f643d9bfb60b755619cbbc (table default.Table_7d476d0d69e246e28664c7c5b84f3ba2 [id=7c51b3299c2948189100608937aa1ef9]) successfully deleted
W20250903 20:05:11.014761 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet f7ab722c37264973b204cd4f1f95052d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f7ab722c37264973b204cd4f1f95052d already in progress: creating tablet
I20250903 20:05:11.015151 27685 tablet_replica.cc:331] T 0a266b1bbf1540ac86d49c56df2dd4ad P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:11.015738 27685 raft_consensus.cc:2241] T 0a266b1bbf1540ac86d49c56df2dd4ad P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:11.016028 27685 raft_consensus.cc:2270] T 0a266b1bbf1540ac86d49c56df2dd4ad P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:11.017035 27685 ts_tablet_manager.cc:1905] T 0a266b1bbf1540ac86d49c56df2dd4ad P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:11.021786 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8b3121fce1c44d16a0b083313dc67b7a consensus state has no leader
I20250903 20:05:11.023705 27057 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet b6b4713936134681aaef1732d46c6ab2 (table default.Table_b6f25e1f7a484577a9cbff11e0deb857 [id=d0a75545b91c48bba47fb581bb0d1a96]). Expected version 1 got 0
I20250903 20:05:11.024175 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 7ba18e3f8cb148d484010cb4e72510b2
I20250903 20:05:11.028003 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_10864de211fa4474949fecdfbe4c1e9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:11.028795 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10864de211fa4474949fecdfbe4c1e9e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:11.029361 27685 ts_tablet_manager.cc:1918] T 0a266b1bbf1540ac86d49c56df2dd4ad P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:11.029417 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 3906285457f449e6824d1f044bffca24 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.030131 27685 log.cc:1199] T 0a266b1bbf1540ac86d49c56df2dd4ad P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/0a266b1bbf1540ac86d49c56df2dd4ad
I20250903 20:05:11.030678 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ba18e3f8cb148d484010cb4e72510b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:11.030707 27685 ts_tablet_manager.cc:1939] T 0a266b1bbf1540ac86d49c56df2dd4ad P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:11.032085 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 0a266b1bbf1540ac86d49c56df2dd4ad (table default.Table_d5626bea3c88471b846f37ae4ed79715 [id=75e5cb0bab9b46a6b5caec7c99efc7b8]) successfully deleted
W20250903 20:05:11.033159 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet c25aaa6ff310432bb27a0695394ec420 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c25aaa6ff310432bb27a0695394ec420 already in progress: creating tablet
I20250903 20:05:11.033907 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_4810ae9efbde43bc8c237c6e135aa5aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:11.034086 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 6ea37668d64d4feeb92a162711609c13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ea37668d64d4feeb92a162711609c13 already in progress: creating tablet
I20250903 20:05:11.034409 27684 tablet_replica.cc:331] T 3906285457f449e6824d1f044bffca24 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:11.034528 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fef9400227449ed892daff588999caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.034634 27684 raft_consensus.cc:2241] T 3906285457f449e6824d1f044bffca24 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
W20250903 20:05:11.034663 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4810ae9efbde43bc8c237c6e135aa5aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:11.034812 27684 raft_consensus.cc:2270] T 3906285457f449e6824d1f044bffca24 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:11.034947 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 79a0553c912049cca5971b1ad6f7e11c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79a0553c912049cca5971b1ad6f7e11c already in progress: creating tablet
W20250903 20:05:11.035738 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 7ba18e3f8cb148d484010cb4e72510b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ba18e3f8cb148d484010cb4e72510b2 already in progress: creating tablet
I20250903 20:05:11.036568 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ba18e3f8cb148d484010cb4e72510b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.037261 27695 tablet_bootstrap.cc:654] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.038290 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b3121fce1c44d16a0b083313dc67b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.024528 27062 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet c8a293c6fa034e9fb29682439cb3bcc2 (table default.Table_37e1d397cf99443d87211c3a8e1b9b82 [id=5cfc8827e4ca4f4a94424c48d6f71eb5]). Expected version 2 got 0
I20250903 20:05:11.041416 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ba18e3f8cb148d484010cb4e72510b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:11.024788 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_9b700cf2d51b4fca81d158e22d72a3f3" } new_table_name: "default.Table_1ade5bba3a18432c808ee7c4b910d247" modify_external_catalogs: true
I20250903 20:05:11.041981 27693 tablet_bootstrap.cc:654] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:11.042529 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 31342583c026438cbbae14106e75bbe5 consensus state has no leader
I20250903 20:05:11.042800 27060 master_service.cc:948] ReplaceTablet: received request to replace tablet 12a7be47c9c2406596c9e00800074a7c from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:11.025385 27059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7e75298df3b54e3794ab39c8b0fbffc7" } modify_external_catalogs: true
I20250903 20:05:11.043361 27059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7e75298df3b54e3794ab39c8b0fbffc7" } modify_external_catalogs: true
I20250903 20:05:11.025693 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet b5bdc27275cd4156afe767a1fb33a8bd
I20250903 20:05:11.044411 27061 master_service.cc:948] ReplaceTablet: received request to replace tablet d8ed7b3e292241cb9525745a797ff13a from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:11.044493 27063 master_service.cc:948] ReplaceTablet: received request to replace tablet 710e3bc3c4194efbbeef6bc76ee64f24 from {username='slave'} at 127.0.0.1:45936
W20250903 20:05:11.044848 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 46b5d491888147bbb101963641fd05f8 consensus state has no leader
W20250903 20:05:11.046015 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cfa8a1d9011e460681443a6b77b95958 consensus state has no leader
I20250903 20:05:11.046478 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet b5bdc27275cd4156afe767a1fb33a8bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:11.047914 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet b5bdc27275cd4156afe767a1fb33a8bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5bdc27275cd4156afe767a1fb33a8bd already in progress: creating tablet
I20250903 20:05:11.048746 27692 tablet_bootstrap.cc:492] T 0062307c6ee54aada74a08cf0d087f17 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:11.048916 27692 ts_tablet_manager.cc:1397] T 0062307c6ee54aada74a08cf0d087f17 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.065s user 0.002s sys 0.002s
I20250903 20:05:11.049172 27695 tablet_bootstrap.cc:492] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:11.049181 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet b5bdc27275cd4156afe767a1fb33a8bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.049335 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet f99c05e72b9a4ff9a1e7c269ca029b6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.049468 27695 ts_tablet_manager.cc:1397] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.045s user 0.000s sys 0.004s
I20250903 20:05:11.049896 27692 raft_consensus.cc:357] T 0062307c6ee54aada74a08cf0d087f17 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.050019 27684 ts_tablet_manager.cc:1905] T 3906285457f449e6824d1f044bffca24 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:11.050105 27692 raft_consensus.cc:383] T 0062307c6ee54aada74a08cf0d087f17 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.050248 27692 raft_consensus.cc:738] T 0062307c6ee54aada74a08cf0d087f17 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.050535 27692 consensus_queue.cc:260] T 0062307c6ee54aada74a08cf0d087f17 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.050836 27695 raft_consensus.cc:357] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.051095 27695 raft_consensus.cc:383] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.051213 27695 raft_consensus.cc:738] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.051563 27695 consensus_queue.cc:260] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.057186 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet 0062307c6ee54aada74a08cf0d087f17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.057691 27692 ts_tablet_manager.cc:1428] T 0062307c6ee54aada74a08cf0d087f17 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20250903 20:05:11.060079 27065 master_service.cc:948] ReplaceTablet: received request to replace tablet 8945085bdc174e6db22a5a9477810297 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:11.063616 27692 tablet_bootstrap.cc:492] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:11.064970 27695 ts_tablet_manager.cc:1428] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20250903 20:05:11.068842 27692 tablet_bootstrap.cc:654] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.069099 27695 tablet_bootstrap.cc:492] T eaa11d17303142ff935705c16f0b9690 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:11.069927 27684 ts_tablet_manager.cc:1918] T 3906285457f449e6824d1f044bffca24 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:11.070119 27684 log.cc:1199] T 3906285457f449e6824d1f044bffca24 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/3906285457f449e6824d1f044bffca24
I20250903 20:05:11.070653 27684 ts_tablet_manager.cc:1939] T 3906285457f449e6824d1f044bffca24 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:11.071854 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 3906285457f449e6824d1f044bffca24 (table default.Table_264581b9b25640c58ed0dd863bf6bb8e [id=b35d1fca6bcd4bb6a1581c2c81cac712]) successfully deleted
W20250903 20:05:11.072175 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 7ba18e3f8cb148d484010cb4e72510b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ba18e3f8cb148d484010cb4e72510b2 already in progress: creating tablet
I20250903 20:05:11.073678 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet b5bdc27275cd4156afe767a1fb33a8bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:11.076198 27695 tablet_bootstrap.cc:654] T eaa11d17303142ff935705c16f0b9690 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.079751 27687 ts_tablet_manager.cc:1918] T f99c05e72b9a4ff9a1e7c269ca029b6a P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:11.079979 27687 log.cc:1199] T f99c05e72b9a4ff9a1e7c269ca029b6a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/f99c05e72b9a4ff9a1e7c269ca029b6a
I20250903 20:05:11.080415 27687 ts_tablet_manager.cc:1939] T f99c05e72b9a4ff9a1e7c269ca029b6a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:11.081015 27692 tablet_bootstrap.cc:492] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:11.081198 27692 ts_tablet_manager.cc:1397] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.005s
I20250903 20:05:11.081686 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet f99c05e72b9a4ff9a1e7c269ca029b6a (table default.Table_0c3cf60b884e47eaaa60b3e0e80d56a3 [id=0b57a09164704002b7fcd0b8f08da1e9]) successfully deleted
I20250903 20:05:11.082101 27692 raft_consensus.cc:357] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.082374 27692 raft_consensus.cc:383] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.082453 27692 raft_consensus.cc:738] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.082736 27692 consensus_queue.cc:260] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:11.083259 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet f7ab722c37264973b204cd4f1f95052d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f7ab722c37264973b204cd4f1f95052d already in progress: creating tablet
W20250903 20:05:11.083534 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 6ea37668d64d4feeb92a162711609c13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ea37668d64d4feeb92a162711609c13 already in progress: creating tablet
W20250903 20:05:11.083729 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet c25aaa6ff310432bb27a0695394ec420 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c25aaa6ff310432bb27a0695394ec420 already in progress: creating tablet
W20250903 20:05:11.083921 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 79a0553c912049cca5971b1ad6f7e11c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79a0553c912049cca5971b1ad6f7e11c already in progress: creating tablet
W20250903 20:05:11.084123 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 7ba18e3f8cb148d484010cb4e72510b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ba18e3f8cb148d484010cb4e72510b2 already in progress: creating tablet
W20250903 20:05:11.084345 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet b5bdc27275cd4156afe767a1fb33a8bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5bdc27275cd4156afe767a1fb33a8bd already in progress: creating tablet
I20250903 20:05:11.086126 27684 tablet_replica.cc:331] T f99c05e72b9a4ff9a1e7c269ca029b6a P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:11.086341 27684 raft_consensus.cc:2241] T f99c05e72b9a4ff9a1e7c269ca029b6a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:11.086520 27684 raft_consensus.cc:2270] T f99c05e72b9a4ff9a1e7c269ca029b6a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:11.086647 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet b5bdc27275cd4156afe767a1fb33a8bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5bdc27275cd4156afe767a1fb33a8bd already in progress: creating tablet
I20250903 20:05:11.081889 27822 tablet.cc:1722] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250903 20:05:11.090512 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 264127511a234e9fbf409f3aa2860ccc consensus state has no leader
I20250903 20:05:11.093200 27684 ts_tablet_manager.cc:1905] T f99c05e72b9a4ff9a1e7c269ca029b6a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:11.094897 27692 ts_tablet_manager.cc:1428] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20250903 20:05:11.098635 27692 tablet_bootstrap.cc:492] T 7e0810c96eb24b8ab4278038801be245 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:11.101351 27357 tablet_service.cc:1468] Processing CreateTablet for tablet 4c543ac4df33429b9787c6d32bf1ebeb (DEFAULT_TABLE table=default.Table_26a46b6cf89f459b99526bc640729603 [id=ec0522912bda4887b0c40e98ac12768a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.102005 27502 tablet_service.cc:1468] Processing CreateTablet for tablet e9eb24dd5a6243c49a3389320a3f78a0 (DEFAULT_TABLE table=default.Table_091b5cecb0b04277a78dc078b4f800a6 [id=1b9cb57cf2b445098305391cc7935b69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.102499 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9eb24dd5a6243c49a3389320a3f78a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.102967 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c543ac4df33429b9787c6d32bf1ebeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.105860 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 4c543ac4df33429b9787c6d32bf1ebeb (DEFAULT_TABLE table=default.Table_26a46b6cf89f459b99526bc640729603 [id=ec0522912bda4887b0c40e98ac12768a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.106289 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c543ac4df33429b9787c6d32bf1ebeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.101785 27359 tablet_service.cc:1468] Processing CreateTablet for tablet e9eb24dd5a6243c49a3389320a3f78a0 (DEFAULT_TABLE table=default.Table_091b5cecb0b04277a78dc078b4f800a6 [id=1b9cb57cf2b445098305391cc7935b69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.107249 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9eb24dd5a6243c49a3389320a3f78a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.109367 27366 tablet_service.cc:1468] Processing CreateTablet for tablet 3f78a951bb024cf290116b0fe438ddb4 (DEFAULT_TABLE table=default.Table_835ced1f8b194192809c3b6fbf4d1d96 [id=23f448924ec04ef08a541a14bd2ce5df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.109761 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f78a951bb024cf290116b0fe438ddb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.112761 27500 tablet_service.cc:1468] Processing CreateTablet for tablet 3f78a951bb024cf290116b0fe438ddb4 (DEFAULT_TABLE table=default.Table_835ced1f8b194192809c3b6fbf4d1d96 [id=23f448924ec04ef08a541a14bd2ce5df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.113131 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f78a951bb024cf290116b0fe438ddb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.114573 27363 tablet_service.cc:1468] Processing CreateTablet for tablet 57a1167457b84ea8a493be232b1d088a (DEFAULT_TABLE table=default.Table_b2a82a2c310140caaeac5d0dcb4bb61a [id=e2548ad469e54a41af7c0181357d2467]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.114975 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57a1167457b84ea8a493be232b1d088a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.117727 27503 tablet_service.cc:1468] Processing CreateTablet for tablet 57a1167457b84ea8a493be232b1d088a (DEFAULT_TABLE table=default.Table_b2a82a2c310140caaeac5d0dcb4bb61a [id=e2548ad469e54a41af7c0181357d2467]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.122004 27755 raft_consensus.cc:491] T 0062307c6ee54aada74a08cf0d087f17 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:11.122291 27755 raft_consensus.cc:513] T 0062307c6ee54aada74a08cf0d087f17 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.123099 27755 leader_election.cc:290] T 0062307c6ee54aada74a08cf0d087f17 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:11.123821 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57a1167457b84ea8a493be232b1d088a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.123822 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0062307c6ee54aada74a08cf0d087f17" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:11.124428 27462 leader_election.cc:343] T 0062307c6ee54aada74a08cf0d087f17 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:11.139250 27462 leader_election.cc:343] T 0062307c6ee54aada74a08cf0d087f17 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 0062307c6ee54aada74a08cf0d087f17
I20250903 20:05:11.139430 27462 leader_election.cc:304] T 0062307c6ee54aada74a08cf0d087f17 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:11.139914 27213 tablet_service.cc:1468] Processing CreateTablet for tablet e9eb24dd5a6243c49a3389320a3f78a0 (DEFAULT_TABLE table=default.Table_091b5cecb0b04277a78dc078b4f800a6 [id=1b9cb57cf2b445098305391cc7935b69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.140337 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9eb24dd5a6243c49a3389320a3f78a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.141347 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 710e3bc3c4194efbbeef6bc76ee64f24
I20250903 20:05:11.141554 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 12a7be47c9c2406596c9e00800074a7c
I20250903 20:05:11.141848 27063 catalog_manager.cc:6652] ReplaceTablet: tablet 710e3bc3c4194efbbeef6bc76ee64f24 deleted and replaced by tablet 4be9bdc390de4a74a396003c9c3c2b71
I20250903 20:05:11.141918 27060 catalog_manager.cc:6652] ReplaceTablet: tablet 12a7be47c9c2406596c9e00800074a7c deleted and replaced by tablet 912eee5b48ef467bbd6bbe34f8aa390c
I20250903 20:05:11.142341 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 8945085bdc174e6db22a5a9477810297
I20250903 20:05:11.142333 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 3f78a951bb024cf290116b0fe438ddb4 (DEFAULT_TABLE table=default.Table_835ced1f8b194192809c3b6fbf4d1d96 [id=23f448924ec04ef08a541a14bd2ce5df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.142688 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f78a951bb024cf290116b0fe438ddb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.142832 27065 catalog_manager.cc:6652] ReplaceTablet: tablet 8945085bdc174e6db22a5a9477810297 deleted and replaced by tablet f71d3dd9af2543299296f21381d1d563
I20250903 20:05:11.142966 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 12a7be47c9c2406596c9e00800074a7c with delete_type TABLET_DATA_DELETED (replaced by tablet 912eee5b48ef467bbd6bbe34f8aa390c) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:11.143142 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet d8ed7b3e292241cb9525745a797ff13a
I20250903 20:05:11.143590 27061 catalog_manager.cc:6652] ReplaceTablet: tablet d8ed7b3e292241cb9525745a797ff13a deleted and replaced by tablet 639ecb9caa4a423899ab8eb35bb716d7
I20250903 20:05:11.143839 27057 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_37e1d397cf99443d87211c3a8e1b9b82 [id=5cfc8827e4ca4f4a94424c48d6f71eb5] alter complete (version 2)
I20250903 20:05:11.144318 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet d8ed7b3e292241cb9525745a797ff13a with delete_type TABLET_DATA_DELETED (replaced by tablet 639ecb9caa4a423899ab8eb35bb716d7) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:11.144804 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 12a7be47c9c2406596c9e00800074a7c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12a7be47c9c2406596c9e00800074a7c already in progress: creating tablet
I20250903 20:05:11.145215 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet 710e3bc3c4194efbbeef6bc76ee64f24 with delete_type TABLET_DATA_DELETED (replaced by tablet 4be9bdc390de4a74a396003c9c3c2b71) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:11.145756 27057 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet c8a293c6fa034e9fb29682439cb3bcc2 (table default.Table_37e1d397cf99443d87211c3a8e1b9b82 [id=5cfc8827e4ca4f4a94424c48d6f71eb5]). Expected version 2 got 0
W20250903 20:05:11.145920 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet d8ed7b3e292241cb9525745a797ff13a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8ed7b3e292241cb9525745a797ff13a already in progress: creating tablet
W20250903 20:05:11.146191 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 710e3bc3c4194efbbeef6bc76ee64f24 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 710e3bc3c4194efbbeef6bc76ee64f24 already in progress: creating tablet
I20250903 20:05:11.146473 27223 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f4ef613b0ab4bca98294682b934f6ed with delete_type TABLET_DATA_DELETED (replaced by tablet 264127511a234e9fbf409f3aa2860ccc) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:11.146700 27366 tablet_service.cc:1515] Processing DeleteTablet for tablet 8945085bdc174e6db22a5a9477810297 with delete_type TABLET_DATA_DELETED (replaced by tablet f71d3dd9af2543299296f21381d1d563) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:11.137537 27823 tablet.cc:1722] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250903 20:05:11.147264 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 8945085bdc174e6db22a5a9477810297 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8945085bdc174e6db22a5a9477810297 already in progress: creating tablet
I20250903 20:05:11.147346 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 29581a9dba46479884a0e6e01b7923de
I20250903 20:05:11.148027 27685 tablet_replica.cc:331] T 0f4ef613b0ab4bca98294682b934f6ed P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:11.148227 27685 raft_consensus.cc:2241] T 0f4ef613b0ab4bca98294682b934f6ed P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:11.149027 27685 raft_consensus.cc:2270] T 0f4ef613b0ab4bca98294682b934f6ed P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:11.149291 27226 tablet_service.cc:1468] Processing CreateTablet for tablet 4c543ac4df33429b9787c6d32bf1ebeb (DEFAULT_TABLE table=default.Table_26a46b6cf89f459b99526bc640729603 [id=ec0522912bda4887b0c40e98ac12768a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.150521 27223 tablet_service.cc:1468] Processing CreateTablet for tablet 57a1167457b84ea8a493be232b1d088a (DEFAULT_TABLE table=default.Table_b2a82a2c310140caaeac5d0dcb4bb61a [id=e2548ad469e54a41af7c0181357d2467]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.152366 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c543ac4df33429b9787c6d32bf1ebeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.152660 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet 8945085bdc174e6db22a5a9477810297 with delete_type TABLET_DATA_DELETED (replaced by tablet f71d3dd9af2543299296f21381d1d563) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.152736 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet d8ed7b3e292241cb9525745a797ff13a with delete_type TABLET_DATA_DELETED (replaced by tablet 639ecb9caa4a423899ab8eb35bb716d7) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.153653 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_41538df113834b45ae2213b105fb9706" } modify_external_catalogs: true
I20250903 20:05:11.153795 27501 tablet_service.cc:1515] Processing DeleteTablet for tablet 12a7be47c9c2406596c9e00800074a7c with delete_type TABLET_DATA_DELETED (replaced by tablet 912eee5b48ef467bbd6bbe34f8aa390c) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.153856 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_41538df113834b45ae2213b105fb9706" } modify_external_catalogs: true
I20250903 20:05:11.154023 27500 tablet_service.cc:1515] Processing DeleteTablet for tablet 710e3bc3c4194efbbeef6bc76ee64f24 with delete_type TABLET_DATA_DELETED (replaced by tablet 4be9bdc390de4a74a396003c9c3c2b71) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.154421 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet 29581a9dba46479884a0e6e01b7923de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.154428 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a642c53ef0da461ba2f33dd4010e7129" } modify_external_catalogs: true
I20250903 20:05:11.154618 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a642c53ef0da461ba2f33dd4010e7129" } modify_external_catalogs: true
I20250903 20:05:11.154670 27685 ts_tablet_manager.cc:1905] T 0f4ef613b0ab4bca98294682b934f6ed P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:11.154902 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet 12a7be47c9c2406596c9e00800074a7c with delete_type TABLET_DATA_DELETED (replaced by tablet 912eee5b48ef467bbd6bbe34f8aa390c) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:11.155030 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet 710e3bc3c4194efbbeef6bc76ee64f24 with delete_type TABLET_DATA_DELETED (replaced by tablet 4be9bdc390de4a74a396003c9c3c2b71) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:11.155233 27223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57a1167457b84ea8a493be232b1d088a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.157290 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet 8945085bdc174e6db22a5a9477810297 with delete_type TABLET_DATA_DELETED (replaced by tablet f71d3dd9af2543299296f21381d1d563) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:11.157544 27755 raft_consensus.cc:2747] T 0062307c6ee54aada74a08cf0d087f17 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:11.159832 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet fb76ba1d7cd54d6aa69d06a9a5618cd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:11.160460 27366 tablet_service.cc:1515] Processing DeleteTablet for tablet 29581a9dba46479884a0e6e01b7923de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:11.162428 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 29581a9dba46479884a0e6e01b7923de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29581a9dba46479884a0e6e01b7923de already in progress: creating tablet
I20250903 20:05:11.160686 27693 tablet_bootstrap.cc:492] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:11.163663 27525 raft_consensus.cc:1273] T fd499b81344c4b58bf77f3622422763c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Refusing update from remote peer 89f8f0ee68b940db9ce250c0604f934a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 20:05:11.164515 27385 raft_consensus.cc:1273] T fd499b81344c4b58bf77f3622422763c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Refusing update from remote peer 89f8f0ee68b940db9ce250c0604f934a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 20:05:11.164598 27692 tablet_bootstrap.cc:654] T 7e0810c96eb24b8ab4278038801be245 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.165272 27719 consensus_queue.cc:1035] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:11.167006 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet 29581a9dba46479884a0e6e01b7923de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:11.170085 27684 ts_tablet_manager.cc:1918] T f99c05e72b9a4ff9a1e7c269ca029b6a P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:11.170249 27684 log.cc:1199] T f99c05e72b9a4ff9a1e7c269ca029b6a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/f99c05e72b9a4ff9a1e7c269ca029b6a
I20250903 20:05:11.170622 27684 ts_tablet_manager.cc:1939] T f99c05e72b9a4ff9a1e7c269ca029b6a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:11.170670 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet d8ed7b3e292241cb9525745a797ff13a with delete_type TABLET_DATA_DELETED (replaced by tablet 639ecb9caa4a423899ab8eb35bb716d7) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:11.171988 27693 ts_tablet_manager.cc:1397] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.198s user 0.016s sys 0.007s
I20250903 20:05:11.172783 27684 tablet_replica.cc:331] T 0062307c6ee54aada74a08cf0d087f17 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:11.172993 27684 raft_consensus.cc:2241] T 0062307c6ee54aada74a08cf0d087f17 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:11.173161 27684 raft_consensus.cc:2270] T 0062307c6ee54aada74a08cf0d087f17 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:11.173357 27693 raft_consensus.cc:357] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.173601 27693 raft_consensus.cc:383] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.173678 27693 raft_consensus.cc:738] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.175206 27693 consensus_queue.cc:260] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.175622 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet f99c05e72b9a4ff9a1e7c269ca029b6a (table default.Table_0c3cf60b884e47eaaa60b3e0e80d56a3 [id=0b57a09164704002b7fcd0b8f08da1e9]) successfully deleted
I20250903 20:05:11.176379 27693 ts_tablet_manager.cc:1428] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250903 20:05:11.176724 27693 tablet_bootstrap.cc:492] T c8a293c6fa034e9fb29682439cb3bcc2 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:11.182760 27719 consensus_queue.cc:1035] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20250903 20:05:11.185928 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 8b3121fce1c44d16a0b083313dc67b7a
I20250903 20:05:11.189824 27695 tablet_bootstrap.cc:492] T eaa11d17303142ff935705c16f0b9690 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:11.190049 27695 ts_tablet_manager.cc:1397] T eaa11d17303142ff935705c16f0b9690 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.124s user 0.009s sys 0.006s
I20250903 20:05:11.191395 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_dacd8238c8334c1c9f66e745f5036a06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:11.191977 27053 consensus_peers.cc:489] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 -> Peer 8857b5d4ea1e4f45aafb8e029837cca2 (127.26.65.126:39015): Couldn't send request to peer 8857b5d4ea1e4f45aafb8e029837cca2. Status: Network error: Client connection negotiation failed: client connection to 127.26.65.126:39015: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250903 20:05:11.192039 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dacd8238c8334c1c9f66e745f5036a06 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:11.192070 27366 tablet_service.cc:1468] Processing CreateTablet for tablet de8ac9f984be46ec8f021928de12a090 (DEFAULT_TABLE table=default.Table_10864de211fa4474949fecdfbe4c1e9e [id=2650c8259ef4489695c56b6a2ba898d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.192737 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de8ac9f984be46ec8f021928de12a090. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.193481 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet 8b3121fce1c44d16a0b083313dc67b7a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:11.193411 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_afa82ec7c58343adb623622c56033b9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:11.193994 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_afa82ec7c58343adb623622c56033b9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:11.194180 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 8b3121fce1c44d16a0b083313dc67b7a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b3121fce1c44d16a0b083313dc67b7a already in progress: creating tablet
I20250903 20:05:11.195358 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_8b0346420a89417b9ae2005be49e9f3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:11.196117 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b0346420a89417b9ae2005be49e9f3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:11.198527 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet 8b3121fce1c44d16a0b083313dc67b7a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.191395 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_62f55f5f45434b01b7b7e2d6d1ddc6fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:11.199349 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62f55f5f45434b01b7b7e2d6d1ddc6fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:11.201742 27695 raft_consensus.cc:357] T eaa11d17303142ff935705c16f0b9690 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.201956 27695 raft_consensus.cc:383] T eaa11d17303142ff935705c16f0b9690 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.202441 27695 raft_consensus.cc:738] T eaa11d17303142ff935705c16f0b9690 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.202718 27695 consensus_queue.cc:260] T eaa11d17303142ff935705c16f0b9690 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.204516 27695 ts_tablet_manager.cc:1428] T eaa11d17303142ff935705c16f0b9690 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250903 20:05:11.204860 27695 tablet_bootstrap.cc:492] T be278233c7c34061988b267a44ee674f P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:11.205574 27217 tablet_service.cc:1468] Processing CreateTablet for tablet de8ac9f984be46ec8f021928de12a090 (DEFAULT_TABLE table=default.Table_10864de211fa4474949fecdfbe4c1e9e [id=2650c8259ef4489695c56b6a2ba898d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.206387 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de8ac9f984be46ec8f021928de12a090. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.207496 27357 tablet_service.cc:1468] Processing CreateTablet for tablet 07565ea2b5f34dcfb407ff3bfde17243 (DEFAULT_TABLE table=default.Table_4810ae9efbde43bc8c237c6e135aa5aa [id=dbeb164ccbf544cf947af9bf656559c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.207966 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07565ea2b5f34dcfb407ff3bfde17243. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.208557 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet 8b3121fce1c44d16a0b083313dc67b7a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:11.209843 27502 tablet_service.cc:1468] Processing CreateTablet for tablet de8ac9f984be46ec8f021928de12a090 (DEFAULT_TABLE table=default.Table_10864de211fa4474949fecdfbe4c1e9e [id=2650c8259ef4489695c56b6a2ba898d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.210301 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de8ac9f984be46ec8f021928de12a090. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.213505 27214 tablet_service.cc:1468] Processing CreateTablet for tablet fbfabb6ff1714d2e921f45256b543759 (DEFAULT_TABLE table=default.Table_aab58ad5394041179974552994469305 [id=1889dee4122f4e69ac2e3d22fb4126a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.213928 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbfabb6ff1714d2e921f45256b543759. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.214345 27226 tablet_service.cc:1468] Processing CreateTablet for tablet 07565ea2b5f34dcfb407ff3bfde17243 (DEFAULT_TABLE table=default.Table_4810ae9efbde43bc8c237c6e135aa5aa [id=dbeb164ccbf544cf947af9bf656559c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.215485 27060 master_service.cc:948] ReplaceTablet: received request to replace tablet 262a3549867a473aaaa7dfcc0b5872cd from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:11.216120 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07565ea2b5f34dcfb407ff3bfde17243. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.217836 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 07565ea2b5f34dcfb407ff3bfde17243 (DEFAULT_TABLE table=default.Table_4810ae9efbde43bc8c237c6e135aa5aa [id=dbeb164ccbf544cf947af9bf656559c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.218287 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07565ea2b5f34dcfb407ff3bfde17243. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.218253 27499 tablet_service.cc:1468] Processing CreateTablet for tablet bf2545037c214bb1b5060575e3cdf81a (DEFAULT_TABLE table=default.Table_983904dd0bb4486493150ce49e43a229 [id=42c554f2f532414a88b50818e2dbb458]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.218510 27225 tablet_service.cc:1468] Processing CreateTablet for tablet bf2545037c214bb1b5060575e3cdf81a (DEFAULT_TABLE table=default.Table_983904dd0bb4486493150ce49e43a229 [id=42c554f2f532414a88b50818e2dbb458]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.218739 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf2545037c214bb1b5060575e3cdf81a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.218868 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf2545037c214bb1b5060575e3cdf81a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.221668 27362 tablet_service.cc:1468] Processing CreateTablet for tablet 5fa7cc0c03234223af382982ca737e27 (DEFAULT_TABLE table=default.Table_d8651e0216bf42e08446c9cdc2ed5b0a [id=66eb535587dc49e5b1ee72124cca0374]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.221911 27224 tablet_service.cc:1468] Processing CreateTablet for tablet 6238740ce1894e899be9a40321f0ab73 (DEFAULT_TABLE table=default.Table_232f4e198e7746f8a49e0481f1166138 [id=53074cd97dec40c7b0e1444183d6707a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.222182 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fa7cc0c03234223af382982ca737e27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.222496 27503 tablet_service.cc:1468] Processing CreateTablet for tablet fbfabb6ff1714d2e921f45256b543759 (DEFAULT_TABLE table=default.Table_aab58ad5394041179974552994469305 [id=1889dee4122f4e69ac2e3d22fb4126a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.222846 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbfabb6ff1714d2e921f45256b543759. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.223981 27684 ts_tablet_manager.cc:1905] T 0062307c6ee54aada74a08cf0d087f17 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:11.224843 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6238740ce1894e899be9a40321f0ab73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.225050 27220 tablet_service.cc:1468] Processing CreateTablet for tablet 5fa7cc0c03234223af382982ca737e27 (DEFAULT_TABLE table=default.Table_d8651e0216bf42e08446c9cdc2ed5b0a [id=66eb535587dc49e5b1ee72124cca0374]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.225404 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fa7cc0c03234223af382982ca737e27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.229533 27221 tablet_service.cc:1468] Processing CreateTablet for tablet 9a76a17b64a74fbfbb4f2e92a02ae1cd (DEFAULT_TABLE table=default.Table_e822637ebd11495cbb3c65907c6a7c78 [id=297ca0b9100044d783d2c610c137f77b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.229987 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a76a17b64a74fbfbb4f2e92a02ae1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.230749 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 6238740ce1894e899be9a40321f0ab73 (DEFAULT_TABLE table=default.Table_232f4e198e7746f8a49e0481f1166138 [id=53074cd97dec40c7b0e1444183d6707a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.231148 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6238740ce1894e899be9a40321f0ab73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.233035 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 369c750ff5bf449e989d95ffd89497d4 (DEFAULT_TABLE table=default.Table_b0b8582c6c054baab4688a30cfe5f767 [id=feefb00b97c34fa0a5ed8af66004830e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.233449 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 369c750ff5bf449e989d95ffd89497d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.234164 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 369c750ff5bf449e989d95ffd89497d4 (DEFAULT_TABLE table=default.Table_b0b8582c6c054baab4688a30cfe5f767 [id=feefb00b97c34fa0a5ed8af66004830e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.234558 27363 tablet_service.cc:1468] Processing CreateTablet for tablet bf2545037c214bb1b5060575e3cdf81a (DEFAULT_TABLE table=default.Table_983904dd0bb4486493150ce49e43a229 [id=42c554f2f532414a88b50818e2dbb458]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.234715 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 369c750ff5bf449e989d95ffd89497d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.234948 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf2545037c214bb1b5060575e3cdf81a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.239795 27358 tablet_service.cc:1468] Processing CreateTablet for tablet 9a76a17b64a74fbfbb4f2e92a02ae1cd (DEFAULT_TABLE table=default.Table_e822637ebd11495cbb3c65907c6a7c78 [id=297ca0b9100044d783d2c610c137f77b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.240245 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a76a17b64a74fbfbb4f2e92a02ae1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.243644 27362 tablet_service.cc:1468] Processing CreateTablet for tablet 0bdcbaa0169c4ab7ba6ec038b50482fe (DEFAULT_TABLE table=default.Table_80ad865643824ab88e2d63eb1b66d079 [id=e69321e60a3a4e71a463a0011c0c6d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.244110 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bdcbaa0169c4ab7ba6ec038b50482fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.246472 27685 ts_tablet_manager.cc:1918] T 0f4ef613b0ab4bca98294682b934f6ed P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:11.246623 27685 log.cc:1199] T 0f4ef613b0ab4bca98294682b934f6ed P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/0f4ef613b0ab4bca98294682b934f6ed
I20250903 20:05:11.247012 27685 ts_tablet_manager.cc:1939] T 0f4ef613b0ab4bca98294682b934f6ed P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:11.247663 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet deb84fc3465443d99c11097e53f5f575
I20250903 20:05:11.248354 27364 tablet_service.cc:1468] Processing CreateTablet for tablet ac1b0dc95fca49b4af6697c0a70e9730 (DEFAULT_TABLE table=default.Table_3f5617eea30e4038945f21ba8ca6be65 [id=6952319d78524829b3bc6c3de6425a67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.248751 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac1b0dc95fca49b4af6697c0a70e9730. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.251422 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 369c750ff5bf449e989d95ffd89497d4 (DEFAULT_TABLE table=default.Table_b0b8582c6c054baab4688a30cfe5f767 [id=feefb00b97c34fa0a5ed8af66004830e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.251869 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 369c750ff5bf449e989d95ffd89497d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.252346 27501 tablet_service.cc:1468] Processing CreateTablet for tablet 9a76a17b64a74fbfbb4f2e92a02ae1cd (DEFAULT_TABLE table=default.Table_e822637ebd11495cbb3c65907c6a7c78 [id=297ca0b9100044d783d2c610c137f77b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.252728 27501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a76a17b64a74fbfbb4f2e92a02ae1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.258656 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet deb84fc3465443d99c11097e53f5f575 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:11.259495 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet deb84fc3465443d99c11097e53f5f575 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet deb84fc3465443d99c11097e53f5f575 already in progress: creating tablet
I20250903 20:05:11.263085 27214 tablet_service.cc:1468] Processing CreateTablet for tablet ac1b0dc95fca49b4af6697c0a70e9730 (DEFAULT_TABLE table=default.Table_3f5617eea30e4038945f21ba8ca6be65 [id=6952319d78524829b3bc6c3de6425a67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.263609 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac1b0dc95fca49b4af6697c0a70e9730. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.264103 27223 tablet_service.cc:1468] Processing CreateTablet for tablet 0bdcbaa0169c4ab7ba6ec038b50482fe (DEFAULT_TABLE table=default.Table_80ad865643824ab88e2d63eb1b66d079 [id=e69321e60a3a4e71a463a0011c0c6d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.264573 27223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bdcbaa0169c4ab7ba6ec038b50482fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.267191 27695 tablet_bootstrap.cc:654] T be278233c7c34061988b267a44ee674f P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.267601 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_1e6e2159113540b7944963d2140ab460"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:11.268252 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e6e2159113540b7944963d2140ab460 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:11.270493 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_64c39ce0250747edacd69ba845d72455" } modify_external_catalogs: true
I20250903 20:05:11.270687 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_64c39ce0250747edacd69ba845d72455" } modify_external_catalogs: true
I20250903 20:05:11.271806 27500 tablet_service.cc:1468] Processing CreateTablet for tablet 5fa7cc0c03234223af382982ca737e27 (DEFAULT_TABLE table=default.Table_d8651e0216bf42e08446c9cdc2ed5b0a [id=66eb535587dc49e5b1ee72124cca0374]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.272295 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet deb84fc3465443d99c11097e53f5f575 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:11.272979 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_db7d825a12ef40739b70f11044170c72" } new_table_name: "default.Table_682ea2b65fb14ff3b2ce593e15cacba9" modify_external_catalogs: true
I20250903 20:05:11.273641 27684 ts_tablet_manager.cc:1918] T 0062307c6ee54aada74a08cf0d087f17 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:11.274001 27754 raft_consensus.cc:491] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:11.274154 27754 raft_consensus.cc:513] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.275179 27754 leader_election.cc:290] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:11.275920 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6b4713936134681aaef1732d46c6ab2" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:11.276166 27249 raft_consensus.cc:2466] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:11.276829 27321 leader_election.cc:304] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:11.277100 27754 raft_consensus.cc:2802] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:11.277241 27754 raft_consensus.cc:491] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:11.277341 27754 raft_consensus.cc:3058] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:11.277981 27499 tablet_service.cc:1468] Processing CreateTablet for tablet 6238740ce1894e899be9a40321f0ab73 (DEFAULT_TABLE table=default.Table_232f4e198e7746f8a49e0481f1166138 [id=53074cd97dec40c7b0e1444183d6707a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.279176 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 0f4ef613b0ab4bca98294682b934f6ed (table default.Table_1ade5bba3a18432c808ee7c4b910d247 [id=d20347a8196e44459448223a3af2dead]) successfully deleted
I20250903 20:05:11.279503 27685 tablet_replica.cc:331] T 12a7be47c9c2406596c9e00800074a7c P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:11.279726 27685 raft_consensus.cc:2241] T 12a7be47c9c2406596c9e00800074a7c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:11.279901 27685 raft_consensus.cc:2270] T 12a7be47c9c2406596c9e00800074a7c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:11.280483 27685 ts_tablet_manager.cc:1905] T 12a7be47c9c2406596c9e00800074a7c P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:11.280629 27754 raft_consensus.cc:513] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.281945 27754 leader_election.cc:290] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:11.282486 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6b4713936134681aaef1732d46c6ab2" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:11.283407 27359 tablet_service.cc:1468] Processing CreateTablet for tablet fbfabb6ff1714d2e921f45256b543759 (DEFAULT_TABLE table=default.Table_aab58ad5394041179974552994469305 [id=1889dee4122f4e69ac2e3d22fb4126a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.284119 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbfabb6ff1714d2e921f45256b543759. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.284358 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_6ce254828dc045d39e5463fe6a48fa82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:11.284884 27249 raft_consensus.cc:3058] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:11.285076 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ce254828dc045d39e5463fe6a48fa82 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:11.287508 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_1840897b7e7f4f1e8052d16ca5e77a4f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:11.288125 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1840897b7e7f4f1e8052d16ca5e77a4f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:11.288746 27249 raft_consensus.cc:2466] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:11.289304 27321 leader_election.cc:304] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:11.291997 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6d92c85ead9f4f58855e725b7823c449" } new_table_name: "default.Table_9ce3e6d31c314cd0bdbd59ea1260d46a" modify_external_catalogs: true
I20250903 20:05:11.294171 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_13dbf08231794aa085d17ca500361b14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:11.294981 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13dbf08231794aa085d17ca500361b14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:11.296932 27059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e822637ebd11495cbb3c65907c6a7c78" } modify_external_catalogs: true
I20250903 20:05:11.297176 27059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e822637ebd11495cbb3c65907c6a7c78" } modify_external_catalogs: true
I20250903 20:05:11.298923 27695 tablet_bootstrap.cc:492] T be278233c7c34061988b267a44ee674f P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:11.299098 27695 ts_tablet_manager.cc:1397] T be278233c7c34061988b267a44ee674f P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.094s user 0.001s sys 0.007s
I20250903 20:05:11.299414 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ffb4d9c1a2d945f1af90b41164c2875a" } new_table_name: "default.Table_f83028b9d34949b982a5bf7ffa812c27" modify_external_catalogs: true
I20250903 20:05:11.299957 27695 raft_consensus.cc:357] T be278233c7c34061988b267a44ee674f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.300159 27695 raft_consensus.cc:383] T be278233c7c34061988b267a44ee674f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.300246 27695 raft_consensus.cc:738] T be278233c7c34061988b267a44ee674f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.300558 27695 consensus_queue.cc:260] T be278233c7c34061988b267a44ee674f P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.302098 27695 ts_tablet_manager.cc:1428] T be278233c7c34061988b267a44ee674f P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:11.302487 27695 tablet_bootstrap.cc:492] T f3a33e90dc1f4540aee457de96f5b95b P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:11.304629 27695 tablet_bootstrap.cc:654] T f3a33e90dc1f4540aee457de96f5b95b P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.305974 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 262a3549867a473aaaa7dfcc0b5872cd
I20250903 20:05:11.307590 27060 catalog_manager.cc:6652] ReplaceTablet: tablet 262a3549867a473aaaa7dfcc0b5872cd deleted and replaced by tablet 7c398634d6304a12b35c0579fba4c813
I20250903 20:05:11.307857 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet 262a3549867a473aaaa7dfcc0b5872cd with delete_type TABLET_DATA_DELETED (replaced by tablet 7c398634d6304a12b35c0579fba4c813) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:11.309602 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet be278233c7c34061988b267a44ee674f with delete_type TABLET_DATA_DELETED (replaced by tablet 5bc3b9771a0042fd87e706204dcbb712) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:11.310601 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_43bea7dca99143f5ac13671739868cdb" } new_table_name: "default.Table_8e482428f5d8407795dcd3111aefc596" modify_external_catalogs: true
I20250903 20:05:11.311518 27685 ts_tablet_manager.cc:1918] T 12a7be47c9c2406596c9e00800074a7c P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:11.312341 27685 log.cc:1199] T 12a7be47c9c2406596c9e00800074a7c P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/12a7be47c9c2406596c9e00800074a7c
I20250903 20:05:11.313019 27754 raft_consensus.cc:2802] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:11.313580 27685 ts_tablet_manager.cc:1939] T 12a7be47c9c2406596c9e00800074a7c P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:11.313637 27695 tablet_bootstrap.cc:492] T f3a33e90dc1f4540aee457de96f5b95b P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:11.313607 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet 262a3549867a473aaaa7dfcc0b5872cd with delete_type TABLET_DATA_DELETED (replaced by tablet 7c398634d6304a12b35c0579fba4c813) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:11.313835 27695 ts_tablet_manager.cc:1397] T f3a33e90dc1f4540aee457de96f5b95b P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.006s
W20250903 20:05:11.314677 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 262a3549867a473aaaa7dfcc0b5872cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 262a3549867a473aaaa7dfcc0b5872cd already in progress: creating tablet
I20250903 20:05:11.315069 27695 raft_consensus.cc:357] T f3a33e90dc1f4540aee457de96f5b95b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.315274 27695 raft_consensus.cc:383] T f3a33e90dc1f4540aee457de96f5b95b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.315383 27695 raft_consensus.cc:738] T f3a33e90dc1f4540aee457de96f5b95b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.315629 27695 consensus_queue.cc:260] T f3a33e90dc1f4540aee457de96f5b95b P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.316260 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 12a7be47c9c2406596c9e00800074a7c (table default.Table_97be735f442746fe92867a2df71838a0 [id=fdc7899c8b334d309b834ae1daa14fa0]) successfully deleted
I20250903 20:05:11.317744 27695 ts_tablet_manager.cc:1428] T f3a33e90dc1f4540aee457de96f5b95b P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
W20250903 20:05:11.319304 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 710e3bc3c4194efbbeef6bc76ee64f24 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 710e3bc3c4194efbbeef6bc76ee64f24 already in progress: creating tablet
I20250903 20:05:11.318815 27695 tablet_bootstrap.cc:492] T 59e120e5e39e4de8a239f13545e4b1af P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
W20250903 20:05:11.321223 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 8945085bdc174e6db22a5a9477810297 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8945085bdc174e6db22a5a9477810297 already in progress: creating tablet
I20250903 20:05:11.321519 27503 tablet_service.cc:1468] Processing CreateTablet for tablet ac1b0dc95fca49b4af6697c0a70e9730 (DEFAULT_TABLE table=default.Table_3f5617eea30e4038945f21ba8ca6be65 [id=6952319d78524829b3bc6c3de6425a67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.321899 27692 tablet_bootstrap.cc:492] T 7e0810c96eb24b8ab4278038801be245 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:11.322039 27684 log.cc:1199] T 0062307c6ee54aada74a08cf0d087f17 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/0062307c6ee54aada74a08cf0d087f17
I20250903 20:05:11.322088 27692 ts_tablet_manager.cc:1397] T 7e0810c96eb24b8ab4278038801be245 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.224s user 0.010s sys 0.021s
I20250903 20:05:11.321846 27685 tablet_replica.cc:331] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:11.322690 27695 tablet_bootstrap.cc:654] T 59e120e5e39e4de8a239f13545e4b1af P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.322739 27685 raft_consensus.cc:2241] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:11.322842 27684 ts_tablet_manager.cc:1939] T 0062307c6ee54aada74a08cf0d087f17 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:11.323097 27685 raft_consensus.cc:2270] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:11.324468 27754 raft_consensus.cc:695] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:11.324884 27685 ts_tablet_manager.cc:1905] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:11.325520 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6b4713936134681aaef1732d46c6ab2" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:11.325583 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet deb84fc3465443d99c11097e53f5f575 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.325400 27754 consensus_queue.cc:237] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.325847 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fa7cc0c03234223af382982ca737e27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.326113 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac1b0dc95fca49b4af6697c0a70e9730. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.326032 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 0bdcbaa0169c4ab7ba6ec038b50482fe (DEFAULT_TABLE table=default.Table_80ad865643824ab88e2d63eb1b66d079 [id=e69321e60a3a4e71a463a0011c0c6d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.326439 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bdcbaa0169c4ab7ba6ec038b50482fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.327236 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6b4713936134681aaef1732d46c6ab2" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:11.327273 27692 raft_consensus.cc:357] T 7e0810c96eb24b8ab4278038801be245 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.330282 27692 raft_consensus.cc:383] T 7e0810c96eb24b8ab4278038801be245 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.330363 27527 raft_consensus.cc:2466] T b6b4713936134681aaef1732d46c6ab2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:11.330935 27695 tablet_bootstrap.cc:492] T 59e120e5e39e4de8a239f13545e4b1af P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:11.331108 27695 ts_tablet_manager.cc:1397] T 59e120e5e39e4de8a239f13545e4b1af P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20250903 20:05:11.331084 27692 raft_consensus.cc:738] T 7e0810c96eb24b8ab4278038801be245 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.331271 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6238740ce1894e899be9a40321f0ab73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.331396 27692 consensus_queue.cc:260] T 7e0810c96eb24b8ab4278038801be245 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.332132 27695 raft_consensus.cc:357] T 59e120e5e39e4de8a239f13545e4b1af P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.332327 27695 raft_consensus.cc:383] T 59e120e5e39e4de8a239f13545e4b1af P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.332401 27695 raft_consensus.cc:738] T 59e120e5e39e4de8a239f13545e4b1af P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.332651 27695 consensus_queue.cc:260] T 59e120e5e39e4de8a239f13545e4b1af P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.332939 27692 ts_tablet_manager.cc:1428] T 7e0810c96eb24b8ab4278038801be245 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.010s user 0.002s sys 0.001s
I20250903 20:05:11.334738 27695 ts_tablet_manager.cc:1428] T 59e120e5e39e4de8a239f13545e4b1af P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250903 20:05:11.335059 27695 tablet_bootstrap.cc:492] T a45604ad611e418e84dab793e04101a0 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:11.339099 27695 tablet_bootstrap.cc:654] T a45604ad611e418e84dab793e04101a0 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.344074 27685 ts_tablet_manager.cc:1918] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:11.344285 27685 log.cc:1199] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/fb76ba1d7cd54d6aa69d06a9a5618cd3
I20250903 20:05:11.345336 27692 tablet_bootstrap.cc:492] T 59e120e5e39e4de8a239f13545e4b1af P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:11.348066 27692 tablet_bootstrap.cc:654] T 59e120e5e39e4de8a239f13545e4b1af P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.349184 27685 ts_tablet_manager.cc:1939] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
W20250903 20:05:11.349305 27051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.125:44059, user_credentials={real_user=slave}} blocked reactor thread for 61152.8us
I20250903 20:05:11.365161 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 0062307c6ee54aada74a08cf0d087f17 (table default.Table_6d1b2694cd1a4963ac658f25026e9ef5 [id=f4bc244b05e840b9843b4d448d4f28aa]) successfully deleted
W20250903 20:05:11.365490 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 8945085bdc174e6db22a5a9477810297 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8945085bdc174e6db22a5a9477810297 already in progress: creating tablet
W20250903 20:05:11.365698 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet d8ed7b3e292241cb9525745a797ff13a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8ed7b3e292241cb9525745a797ff13a already in progress: creating tablet
W20250903 20:05:11.365891 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 12a7be47c9c2406596c9e00800074a7c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12a7be47c9c2406596c9e00800074a7c already in progress: creating tablet
I20250903 20:05:11.369758 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet fb76ba1d7cd54d6aa69d06a9a5618cd3 (table default.Table_a249558ab60e466ca7734ffeb24ac5fd [id=d1ed6b104f9541fbaf34c6b03b297f0f]) successfully deleted
W20250903 20:05:11.370513 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 29581a9dba46479884a0e6e01b7923de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29581a9dba46479884a0e6e01b7923de already in progress: creating tablet
W20250903 20:05:11.371340 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet d8ed7b3e292241cb9525745a797ff13a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8ed7b3e292241cb9525745a797ff13a already in progress: creating tablet
W20250903 20:05:11.372208 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 8b3121fce1c44d16a0b083313dc67b7a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b3121fce1c44d16a0b083313dc67b7a already in progress: creating tablet
W20250903 20:05:11.372920 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet deb84fc3465443d99c11097e53f5f575 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet deb84fc3465443d99c11097e53f5f575 already in progress: creating tablet
W20250903 20:05:11.373553 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 262a3549867a473aaaa7dfcc0b5872cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 262a3549867a473aaaa7dfcc0b5872cd already in progress: creating tablet
I20250903 20:05:11.373891 27685 tablet_replica.cc:331] T be278233c7c34061988b267a44ee674f P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:11.374151 27685 raft_consensus.cc:2241] T be278233c7c34061988b267a44ee674f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:11.374368 27685 raft_consensus.cc:2270] T be278233c7c34061988b267a44ee674f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:11.376153 27685 ts_tablet_manager.cc:1905] T be278233c7c34061988b267a44ee674f P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:11.379134 27831 mvcc.cc:204] Tried to move back new op lower bound from 7196384916121333760 to 7196384914485633024. Current Snapshot: MvccSnapshot[applied={T|T < 7196384916121333760}]
I20250903 20:05:11.381198 27823 tablet.cc:1722] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250903 20:05:11.381152 27844 tablet.cc:1722] T fd499b81344c4b58bf77f3622422763c P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20250903 20:05:11.385429 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 710e3bc3c4194efbbeef6bc76ee64f24 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 710e3bc3c4194efbbeef6bc76ee64f24 already in progress: creating tablet
W20250903 20:05:11.385747 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 29581a9dba46479884a0e6e01b7923de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29581a9dba46479884a0e6e01b7923de already in progress: creating tablet
I20250903 20:05:11.388586 27693 tablet_bootstrap.cc:700] T c8a293c6fa034e9fb29682439cb3bcc2 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent opening tablet: real 0.186s user 0.001s sys 0.000s
I20250903 20:05:11.388980 27693 tablet_bootstrap.cc:654] T c8a293c6fa034e9fb29682439cb3bcc2 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.391011 27692 tablet_bootstrap.cc:492] T 59e120e5e39e4de8a239f13545e4b1af P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:11.391364 27692 ts_tablet_manager.cc:1397] T 59e120e5e39e4de8a239f13545e4b1af P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.058s user 0.003s sys 0.017s
I20250903 20:05:11.392655 27692 raft_consensus.cc:357] T 59e120e5e39e4de8a239f13545e4b1af P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.393136 27692 raft_consensus.cc:383] T 59e120e5e39e4de8a239f13545e4b1af P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.393308 27692 raft_consensus.cc:738] T 59e120e5e39e4de8a239f13545e4b1af P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.393625 27692 consensus_queue.cc:260] T 59e120e5e39e4de8a239f13545e4b1af P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.395274 27822 tablet.cc:1722] T fd499b81344c4b58bf77f3622422763c P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250903 20:05:11.395677 27692 ts_tablet_manager.cc:1428] T 59e120e5e39e4de8a239f13545e4b1af P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:11.396044 27692 tablet_bootstrap.cc:492] T 12a7be47c9c2406596c9e00800074a7c P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:11.397987 27692 tablet_bootstrap.cc:654] T 12a7be47c9c2406596c9e00800074a7c P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.401924 27685 ts_tablet_manager.cc:1918] T be278233c7c34061988b267a44ee674f P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:11.402103 27685 log.cc:1199] T be278233c7c34061988b267a44ee674f P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/be278233c7c34061988b267a44ee674f
W20250903 20:05:11.402541 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 8b3121fce1c44d16a0b083313dc67b7a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b3121fce1c44d16a0b083313dc67b7a already in progress: creating tablet
I20250903 20:05:11.402582 27685 ts_tablet_manager.cc:1939] T be278233c7c34061988b267a44ee674f P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
W20250903 20:05:11.402752 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet deb84fc3465443d99c11097e53f5f575 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet deb84fc3465443d99c11097e53f5f575 already in progress: creating tablet
I20250903 20:05:11.403795 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet be278233c7c34061988b267a44ee674f (table default.Table_459903503b5148d09d3cb8f2f5519fab [id=3c69612aaf164dc899dcc252d85636f6]) successfully deleted
I20250903 20:05:11.403923 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 262a3549867a473aaaa7dfcc0b5872cd with delete_type TABLET_DATA_DELETED (replaced by tablet 7c398634d6304a12b35c0579fba4c813) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.405071 27692 tablet_bootstrap.cc:492] T 12a7be47c9c2406596c9e00800074a7c P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:11.405236 27692 ts_tablet_manager.cc:1397] T 12a7be47c9c2406596c9e00800074a7c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20250903 20:05:11.406021 27692 raft_consensus.cc:357] T 12a7be47c9c2406596c9e00800074a7c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.406255 27692 raft_consensus.cc:383] T 12a7be47c9c2406596c9e00800074a7c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.406364 27692 raft_consensus.cc:738] T 12a7be47c9c2406596c9e00800074a7c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.406639 27692 consensus_queue.cc:260] T 12a7be47c9c2406596c9e00800074a7c P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.410133 27692 ts_tablet_manager.cc:1428] T 12a7be47c9c2406596c9e00800074a7c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250903 20:05:11.410161 27693 tablet_bootstrap.cc:492] T c8a293c6fa034e9fb29682439cb3bcc2 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:11.410408 27693 ts_tablet_manager.cc:1397] T c8a293c6fa034e9fb29682439cb3bcc2 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.234s user 0.005s sys 0.000s
I20250903 20:05:11.412209 27693 raft_consensus.cc:357] T c8a293c6fa034e9fb29682439cb3bcc2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.412489 27693 raft_consensus.cc:383] T c8a293c6fa034e9fb29682439cb3bcc2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.412590 27693 raft_consensus.cc:738] T c8a293c6fa034e9fb29682439cb3bcc2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.412869 27693 consensus_queue.cc:260] T c8a293c6fa034e9fb29682439cb3bcc2 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.413358 27692 tablet_bootstrap.cc:492] T a45604ad611e418e84dab793e04101a0 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:11.415222 27693 ts_tablet_manager.cc:1428] T c8a293c6fa034e9fb29682439cb3bcc2 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250903 20:05:11.416607 27692 tablet_bootstrap.cc:654] T a45604ad611e418e84dab793e04101a0 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.416922 27693 tablet_bootstrap.cc:492] T d1e1add08013459ba0dab3b7e66f0136 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:11.419076 27695 tablet_bootstrap.cc:492] T a45604ad611e418e84dab793e04101a0 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:11.419359 27695 ts_tablet_manager.cc:1397] T a45604ad611e418e84dab793e04101a0 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.084s user 0.018s sys 0.021s
I20250903 20:05:11.420488 27695 raft_consensus.cc:357] T a45604ad611e418e84dab793e04101a0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.420830 27695 raft_consensus.cc:383] T a45604ad611e418e84dab793e04101a0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.420989 27695 raft_consensus.cc:738] T a45604ad611e418e84dab793e04101a0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.421396 27695 consensus_queue.cc:260] T a45604ad611e418e84dab793e04101a0 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.424242 27057 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet fd499b81344c4b58bf77f3622422763c (table default.Table_e914faf335d44d019f91bda71758c473 [id=374b9abdf6a54b38876a96d8ec4a0368]). Expected version 4 got 0
I20250903 20:05:11.424925 27695 ts_tablet_manager.cc:1428] T a45604ad611e418e84dab793e04101a0 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250903 20:05:11.426528 27693 tablet_bootstrap.cc:654] T d1e1add08013459ba0dab3b7e66f0136 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.428067 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet be278233c7c34061988b267a44ee674f with delete_type TABLET_DATA_DELETED (replaced by tablet 5bc3b9771a0042fd87e706204dcbb712) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:11.429237 27052 catalog_manager.cc:4895] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet be278233c7c34061988b267a44ee674f because the tablet was not found. No further retry: Not found: Tablet not found: be278233c7c34061988b267a44ee674f
W20250903 20:05:11.430153 27066 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e5136d5388bc4c2ba71e09be40e8f412 consensus state has no leader
I20250903 20:05:11.430291 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 691cb69759654fdc9223844049a06d02
I20250903 20:05:11.431372 27695 tablet_bootstrap.cc:492] T e1a49d0599654552bf1c72857031699b P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:11.431905 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_957f51b8fb684329a6e6fac205da5164"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:11.432677 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_957f51b8fb684329a6e6fac205da5164 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:11.434329 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet 691cb69759654fdc9223844049a06d02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:11.434286 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 691cb69759654fdc9223844049a06d02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:11.434613 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_a2fef2e8171c466e86ce74826ba6e281"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:11.435070 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 691cb69759654fdc9223844049a06d02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 691cb69759654fdc9223844049a06d02 already in progress: creating tablet
W20250903 20:05:11.435393 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 691cb69759654fdc9223844049a06d02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 691cb69759654fdc9223844049a06d02 already in progress: creating tablet
I20250903 20:05:11.430424 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1d5b22110e4b45c193eb0faaae131c67" } new_table_name: "default.Table_6d201f66980a4a7ab430058673f8d513" modify_external_catalogs: true
W20250903 20:05:11.435842 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8023fa7537234ce783c2bf90b575b00c consensus state has no leader
I20250903 20:05:11.436451 27695 tablet_bootstrap.cc:654] T e1a49d0599654552bf1c72857031699b P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.436307 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_53cc54536434460ba6637a083afaddb2" } modify_external_catalogs: true
I20250903 20:05:11.436612 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_53cc54536434460ba6637a083afaddb2" } modify_external_catalogs: true
W20250903 20:05:11.440681 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2fef2e8171c466e86ce74826ba6e281 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:11.441140 27692 tablet_bootstrap.cc:492] T a45604ad611e418e84dab793e04101a0 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:11.441327 27692 ts_tablet_manager.cc:1397] T a45604ad611e418e84dab793e04101a0 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.016s
I20250903 20:05:11.442170 27692 raft_consensus.cc:357] T a45604ad611e418e84dab793e04101a0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.442411 27692 raft_consensus.cc:383] T a45604ad611e418e84dab793e04101a0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.442489 27692 raft_consensus.cc:738] T a45604ad611e418e84dab793e04101a0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.442728 27057 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet fd499b81344c4b58bf77f3622422763c (table default.Table_e914faf335d44d019f91bda71758c473 [id=374b9abdf6a54b38876a96d8ec4a0368]). Expected version 4 got 0
I20250903 20:05:11.442857 27692 consensus_queue.cc:260] T a45604ad611e418e84dab793e04101a0 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.443513 27695 tablet_bootstrap.cc:492] T e1a49d0599654552bf1c72857031699b P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:11.443707 27695 ts_tablet_manager.cc:1397] T e1a49d0599654552bf1c72857031699b P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.001s
I20250903 20:05:11.444777 27057 master_service.cc:948] ReplaceTablet: received request to replace tablet e20fcd28c3ae4355a033d2b8429f84e2 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:11.444690 27695 raft_consensus.cc:357] T e1a49d0599654552bf1c72857031699b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.444948 27695 raft_consensus.cc:383] T e1a49d0599654552bf1c72857031699b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.445051 27695 raft_consensus.cc:738] T e1a49d0599654552bf1c72857031699b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.446151 27692 ts_tablet_manager.cc:1428] T a45604ad611e418e84dab793e04101a0 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250903 20:05:11.446650 27693 tablet_bootstrap.cc:492] T d1e1add08013459ba0dab3b7e66f0136 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:11.446800 27693 ts_tablet_manager.cc:1397] T d1e1add08013459ba0dab3b7e66f0136 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.003s
I20250903 20:05:11.447883 27693 raft_consensus.cc:357] T d1e1add08013459ba0dab3b7e66f0136 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.448127 27693 raft_consensus.cc:383] T d1e1add08013459ba0dab3b7e66f0136 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.448252 27693 raft_consensus.cc:738] T d1e1add08013459ba0dab3b7e66f0136 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.448571 27693 consensus_queue.cc:260] T d1e1add08013459ba0dab3b7e66f0136 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.448987 27692 tablet_bootstrap.cc:492] T 0f4ef613b0ab4bca98294682b934f6ed P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:11.449927 27693 ts_tablet_manager.cc:1428] T d1e1add08013459ba0dab3b7e66f0136 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250903 20:05:11.450695 27693 tablet_bootstrap.cc:492] T c12c957a36f643d9bfb60b755619cbbc P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:11.451472 27695 consensus_queue.cc:260] T e1a49d0599654552bf1c72857031699b P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.453666 27693 tablet_bootstrap.cc:654] T c12c957a36f643d9bfb60b755619cbbc P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.453953 27692 tablet_bootstrap.cc:654] T 0f4ef613b0ab4bca98294682b934f6ed P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:11.454634 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 262a3549867a473aaaa7dfcc0b5872cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 262a3549867a473aaaa7dfcc0b5872cd already in progress: creating tablet
I20250903 20:05:11.456492 27695 ts_tablet_manager.cc:1428] T e1a49d0599654552bf1c72857031699b P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.013s user 0.001s sys 0.002s
I20250903 20:05:11.456911 27695 tablet_bootstrap.cc:492] T 99a21a3e49934a79a66700c0b0d86307 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:11.456913 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 691cb69759654fdc9223844049a06d02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.461402 27695 tablet_bootstrap.cc:654] T 99a21a3e49934a79a66700c0b0d86307 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.467536 27692 tablet_bootstrap.cc:492] T 0f4ef613b0ab4bca98294682b934f6ed P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:11.467687 27692 ts_tablet_manager.cc:1397] T 0f4ef613b0ab4bca98294682b934f6ed P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.003s
I20250903 20:05:11.468529 27692 raft_consensus.cc:357] T 0f4ef613b0ab4bca98294682b934f6ed P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.468716 27692 raft_consensus.cc:383] T 0f4ef613b0ab4bca98294682b934f6ed P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.469022 27692 raft_consensus.cc:738] T 0f4ef613b0ab4bca98294682b934f6ed P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.469280 27692 consensus_queue.cc:260] T 0f4ef613b0ab4bca98294682b934f6ed P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.470736 27693 tablet_bootstrap.cc:492] T c12c957a36f643d9bfb60b755619cbbc P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:11.472388 27693 ts_tablet_manager.cc:1397] T c12c957a36f643d9bfb60b755619cbbc P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.001s
I20250903 20:05:11.472874 27692 ts_tablet_manager.cc:1428] T 0f4ef613b0ab4bca98294682b934f6ed P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250903 20:05:11.473485 27693 raft_consensus.cc:357] T c12c957a36f643d9bfb60b755619cbbc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.473781 27693 raft_consensus.cc:383] T c12c957a36f643d9bfb60b755619cbbc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.473863 27693 raft_consensus.cc:738] T c12c957a36f643d9bfb60b755619cbbc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.474653 27692 tablet_bootstrap.cc:492] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:11.474268 27693 consensus_queue.cc:260] T c12c957a36f643d9bfb60b755619cbbc P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.476933 27693 ts_tablet_manager.cc:1428] T c12c957a36f643d9bfb60b755619cbbc P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:11.477196 27065 master_service.cc:948] ReplaceTablet: received request to replace tablet 4bd2650e4d9e40f5ba5c09e367918224 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:11.477541 27692 tablet_bootstrap.cc:654] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.477739 27693 tablet_bootstrap.cc:492] T 43f88abc8f3640a48598c07c3d76755d P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:11.477759 27065 master_service.cc:948] ReplaceTablet: received request to replace tablet f7ab722c37264973b204cd4f1f95052d from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:11.478174 27065 master_service.cc:948] ReplaceTablet: received request to replace tablet 6ea37668d64d4feeb92a162711609c13 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:11.478678 27065 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1ade5bba3a18432c808ee7c4b910d247" } modify_external_catalogs: true
I20250903 20:05:11.478875 27065 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1ade5bba3a18432c808ee7c4b910d247" } modify_external_catalogs: true
W20250903 20:05:11.479664 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e8bc99de4ffe43d58351d170b7c61d8a consensus state has no leader
I20250903 20:05:11.480146 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 9a76a17b64a74fbfbb4f2e92a02ae1cd
I20250903 20:05:11.480265 27693 tablet_bootstrap.cc:654] T 43f88abc8f3640a48598c07c3d76755d P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:11.481109 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 44db427ee53a4d0a907dfd63282d6d4f consensus state has no leader
I20250903 20:05:11.482684 27695 tablet_bootstrap.cc:492] T 99a21a3e49934a79a66700c0b0d86307 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:11.482862 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a76a17b64a74fbfbb4f2e92a02ae1cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:11.482935 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a76a17b64a74fbfbb4f2e92a02ae1cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:11.483608 27695 ts_tablet_manager.cc:1397] T 99a21a3e49934a79a66700c0b0d86307 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.006s
W20250903 20:05:11.483637 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 9a76a17b64a74fbfbb4f2e92a02ae1cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a76a17b64a74fbfbb4f2e92a02ae1cd already in progress: creating tablet
W20250903 20:05:11.483935 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 9a76a17b64a74fbfbb4f2e92a02ae1cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a76a17b64a74fbfbb4f2e92a02ae1cd already in progress: creating tablet
I20250903 20:05:11.484815 27695 raft_consensus.cc:357] T 99a21a3e49934a79a66700c0b0d86307 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.485173 27695 raft_consensus.cc:383] T 99a21a3e49934a79a66700c0b0d86307 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.485430 27695 raft_consensus.cc:738] T 99a21a3e49934a79a66700c0b0d86307 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
W20250903 20:05:11.485520 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cfa8a1d9011e460681443a6b77b95958 consensus state has no leader
I20250903 20:05:11.485786 27695 consensus_queue.cc:260] T 99a21a3e49934a79a66700c0b0d86307 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.486135 27692 tablet_bootstrap.cc:492] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:11.487468 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet e20fcd28c3ae4355a033d2b8429f84e2
I20250903 20:05:11.488319 27695 ts_tablet_manager.cc:1428] T 99a21a3e49934a79a66700c0b0d86307 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250903 20:05:11.488792 27695 tablet_bootstrap.cc:492] T 7ba18e3f8cb148d484010cb4e72510b2 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:11.489573 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet e20fcd28c3ae4355a033d2b8429f84e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:11.489599 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet e20fcd28c3ae4355a033d2b8429f84e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:11.490339 27692 ts_tablet_manager.cc:1397] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.001s
W20250903 20:05:11.490381 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet e20fcd28c3ae4355a033d2b8429f84e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e20fcd28c3ae4355a033d2b8429f84e2 already in progress: creating tablet
W20250903 20:05:11.490706 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet e20fcd28c3ae4355a033d2b8429f84e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e20fcd28c3ae4355a033d2b8429f84e2 already in progress: creating tablet
I20250903 20:05:11.491247 27692 raft_consensus.cc:357] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.491456 27692 raft_consensus.cc:383] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.491533 27692 raft_consensus.cc:738] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.491827 27692 consensus_queue.cc:260] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.494095 27066 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_e914faf335d44d019f91bda71758c473 [id=374b9abdf6a54b38876a96d8ec4a0368] alter complete (version 4)
I20250903 20:05:11.495015 27695 tablet_bootstrap.cc:654] T 7ba18e3f8cb148d484010cb4e72510b2 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.496043 27692 ts_tablet_manager.cc:1428] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250903 20:05:11.496414 27213 tablet_service.cc:1515] Processing DeleteTablet for tablet 59e120e5e39e4de8a239f13545e4b1af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:11.496536 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet f3a33e90dc1f4540aee457de96f5b95b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:11.496784 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet a45604ad611e418e84dab793e04101a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:11.498065 27685 tablet_replica.cc:331] T 59e120e5e39e4de8a239f13545e4b1af P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:11.498373 27685 raft_consensus.cc:2241] T 59e120e5e39e4de8a239f13545e4b1af P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:11.498621 27685 raft_consensus.cc:2270] T 59e120e5e39e4de8a239f13545e4b1af P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:11.499636 27685 ts_tablet_manager.cc:1905] T 59e120e5e39e4de8a239f13545e4b1af P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:11.499715 27692 tablet_bootstrap.cc:492] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:11.501658 27692 tablet_bootstrap.cc:654] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.501799 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_835ced1f8b194192809c3b6fbf4d1d96" } modify_external_catalogs: true
I20250903 20:05:11.502002 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_835ced1f8b194192809c3b6fbf4d1d96" } modify_external_catalogs: true
W20250903 20:05:11.504334 27050 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 8023fa7537234ce783c2bf90b575b00c (table default.Table_9ce3e6d31c314cd0bdbd59ea1260d46a [id=7032fe3b789041d380f5db114f241e29], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8023fa7537234ce783c2bf90b575b00c consensus state has no leader
I20250903 20:05:11.508586 27685 ts_tablet_manager.cc:1918] T 59e120e5e39e4de8a239f13545e4b1af P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:11.508780 27685 log.cc:1199] T 59e120e5e39e4de8a239f13545e4b1af P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/59e120e5e39e4de8a239f13545e4b1af
I20250903 20:05:11.509295 27685 ts_tablet_manager.cc:1939] T 59e120e5e39e4de8a239f13545e4b1af P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
W20250903 20:05:11.509336 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 691cb69759654fdc9223844049a06d02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 691cb69759654fdc9223844049a06d02 already in progress: creating tablet
I20250903 20:05:11.511777 27693 tablet_bootstrap.cc:492] T 43f88abc8f3640a48598c07c3d76755d P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:11.511988 27693 ts_tablet_manager.cc:1397] T 43f88abc8f3640a48598c07c3d76755d P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.000s
I20250903 20:05:11.512027 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a76a17b64a74fbfbb4f2e92a02ae1cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.513127 27693 raft_consensus.cc:357] T 43f88abc8f3640a48598c07c3d76755d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.513527 27693 raft_consensus.cc:383] T 43f88abc8f3640a48598c07c3d76755d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.513643 27693 raft_consensus.cc:738] T 43f88abc8f3640a48598c07c3d76755d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.513993 27693 consensus_queue.cc:260] T 43f88abc8f3640a48598c07c3d76755d P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.514816 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 59e120e5e39e4de8a239f13545e4b1af (table default.Table_2dd5b5956b44471da9d0b89e782d469e [id=0bfcd1b8b71d45db8d9c370bd1231a38]) successfully deleted
I20250903 20:05:11.515054 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet e20fcd28c3ae4355a033d2b8429f84e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.516490 27693 ts_tablet_manager.cc:1428] T 43f88abc8f3640a48598c07c3d76755d P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250903 20:05:11.517701 27685 tablet_replica.cc:331] T f3a33e90dc1f4540aee457de96f5b95b P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:11.517992 27685 raft_consensus.cc:2241] T f3a33e90dc1f4540aee457de96f5b95b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:11.518745 27693 tablet_bootstrap.cc:492] T 0a266b1bbf1540ac86d49c56df2dd4ad P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:11.521184 27693 tablet_bootstrap.cc:654] T 0a266b1bbf1540ac86d49c56df2dd4ad P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.521786 27685 raft_consensus.cc:2270] T f3a33e90dc1f4540aee457de96f5b95b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:11.522933 27685 ts_tablet_manager.cc:1905] T f3a33e90dc1f4540aee457de96f5b95b P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:11.524626 27695 tablet_bootstrap.cc:492] T 7ba18e3f8cb148d484010cb4e72510b2 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:11.524837 27695 ts_tablet_manager.cc:1397] T 7ba18e3f8cb148d484010cb4e72510b2 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.000s
I20250903 20:05:11.525866 27695 raft_consensus.cc:357] T 7ba18e3f8cb148d484010cb4e72510b2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.526137 27695 raft_consensus.cc:383] T 7ba18e3f8cb148d484010cb4e72510b2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.526276 27695 raft_consensus.cc:738] T 7ba18e3f8cb148d484010cb4e72510b2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.526646 27695 consensus_queue.cc:260] T 7ba18e3f8cb148d484010cb4e72510b2 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.529184 27695 ts_tablet_manager.cc:1428] T 7ba18e3f8cb148d484010cb4e72510b2 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250903 20:05:11.529707 27695 tablet_bootstrap.cc:492] T 4bd2650e4d9e40f5ba5c09e367918224 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:11.529690 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e0810c96eb24b8ab4278038801be245 with delete_type TABLET_DATA_DELETED (replaced by tablet e5136d5388bc4c2ba71e09be40e8f412) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.529999 27684 tablet_replica.cc:331] T 7e0810c96eb24b8ab4278038801be245 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:11.530187 27684 raft_consensus.cc:2241] T 7e0810c96eb24b8ab4278038801be245 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:11.530429 27684 raft_consensus.cc:2270] T 7e0810c96eb24b8ab4278038801be245 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:11.531443 27684 ts_tablet_manager.cc:1905] T 7e0810c96eb24b8ab4278038801be245 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:11.535243 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 12a7be47c9c2406596c9e00800074a7c with delete_type TABLET_DATA_DELETED (replaced by tablet 912eee5b48ef467bbd6bbe34f8aa390c) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.536017 27695 tablet_bootstrap.cc:654] T 4bd2650e4d9e40f5ba5c09e367918224 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.541896 27684 ts_tablet_manager.cc:1918] T 7e0810c96eb24b8ab4278038801be245 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:11.542093 27684 log.cc:1199] T 7e0810c96eb24b8ab4278038801be245 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/7e0810c96eb24b8ab4278038801be245
I20250903 20:05:11.542577 27684 ts_tablet_manager.cc:1939] T 7e0810c96eb24b8ab4278038801be245 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:11.543392 27695 tablet_bootstrap.cc:492] T 4bd2650e4d9e40f5ba5c09e367918224 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:11.543583 27695 ts_tablet_manager.cc:1397] T 4bd2650e4d9e40f5ba5c09e367918224 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.001s
I20250903 20:05:11.544493 27695 raft_consensus.cc:357] T 4bd2650e4d9e40f5ba5c09e367918224 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.544670 27695 raft_consensus.cc:383] T 4bd2650e4d9e40f5ba5c09e367918224 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.544754 27695 raft_consensus.cc:738] T 4bd2650e4d9e40f5ba5c09e367918224 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.544989 27695 consensus_queue.cc:260] T 4bd2650e4d9e40f5ba5c09e367918224 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.546293 27058 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 99a21a3e49934a79a66700c0b0d86307 (table default.Table_7325977ff92e45bab0b59b79579ecae4 [id=428657b402d748c0a9e3942d13342b4e]). Expected version 1 got 0
I20250903 20:05:11.546715 27693 tablet_bootstrap.cc:492] T 0a266b1bbf1540ac86d49c56df2dd4ad P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:11.546927 27693 ts_tablet_manager.cc:1397] T 0a266b1bbf1540ac86d49c56df2dd4ad P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20250903 20:05:11.548087 27693 raft_consensus.cc:357] T 0a266b1bbf1540ac86d49c56df2dd4ad P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.548409 27693 raft_consensus.cc:383] T 0a266b1bbf1540ac86d49c56df2dd4ad P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.548518 27693 raft_consensus.cc:738] T 0a266b1bbf1540ac86d49c56df2dd4ad P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.548854 27693 consensus_queue.cc:260] T 0a266b1bbf1540ac86d49c56df2dd4ad P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.550181 27693 ts_tablet_manager.cc:1428] T 0a266b1bbf1540ac86d49c56df2dd4ad P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250903 20:05:11.550927 27061 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet c8a293c6fa034e9fb29682439cb3bcc2 (table default.Table_37e1d397cf99443d87211c3a8e1b9b82 [id=5cfc8827e4ca4f4a94424c48d6f71eb5]). Expected version 2 got 0
I20250903 20:05:11.551229 27061 catalog_manager.cc:5582] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66). New cstate: current_term: 1 leader_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:11.551671 27061 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet b6b4713936134681aaef1732d46c6ab2 (table default.Table_b6f25e1f7a484577a9cbff11e0deb857 [id=d0a75545b91c48bba47fb581bb0d1a96]). Expected version 1 got 0
I20250903 20:05:11.554016 27362 tablet_service.cc:1468] Processing CreateTablet for tablet 912eee5b48ef467bbd6bbe34f8aa390c (DEFAULT_TABLE table=default.Table_97be735f442746fe92867a2df71838a0 [id=fdc7899c8b334d309b834ae1daa14fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.554593 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 912eee5b48ef467bbd6bbe34f8aa390c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.555542 27693 tablet_bootstrap.cc:492] T 0062307c6ee54aada74a08cf0d087f17 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:11.558188 27693 tablet_bootstrap.cc:654] T 0062307c6ee54aada74a08cf0d087f17 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.559593 27695 ts_tablet_manager.cc:1428] T 4bd2650e4d9e40f5ba5c09e367918224 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:11.560124 27695 tablet_bootstrap.cc:492] T b5bdc27275cd4156afe767a1fb33a8bd P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:11.562299 27359 tablet_service.cc:1468] Processing CreateTablet for tablet 4be9bdc390de4a74a396003c9c3c2b71 (DEFAULT_TABLE table=default.Table_452fdb47678d4520828b9a3db6ddb237 [id=d945a830c86244bca66cb2652e64966d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.562760 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4be9bdc390de4a74a396003c9c3c2b71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.566591 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 639ecb9caa4a423899ab8eb35bb716d7 (DEFAULT_TABLE table=default.Table_646dcf6e43a24b31aaa79d2f93c58a14 [id=ad8b3a9bedef4c188b89640ab5e2473a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.567008 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 639ecb9caa4a423899ab8eb35bb716d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.567946 27693 tablet_bootstrap.cc:492] T 0062307c6ee54aada74a08cf0d087f17 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:11.568162 27693 ts_tablet_manager.cc:1397] T 0062307c6ee54aada74a08cf0d087f17 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.001s
I20250903 20:05:11.569362 27693 raft_consensus.cc:357] T 0062307c6ee54aada74a08cf0d087f17 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.569555 27693 raft_consensus.cc:383] T 0062307c6ee54aada74a08cf0d087f17 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.569768 27840 raft_consensus.cc:491] T a45604ad611e418e84dab793e04101a0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:11.570006 27840 raft_consensus.cc:513] T a45604ad611e418e84dab793e04101a0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.570019 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 912eee5b48ef467bbd6bbe34f8aa390c (DEFAULT_TABLE table=default.Table_97be735f442746fe92867a2df71838a0 [id=fdc7899c8b334d309b834ae1daa14fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.554016 27366 tablet_service.cc:1468] Processing CreateTablet for tablet f71d3dd9af2543299296f21381d1d563 (DEFAULT_TABLE table=default.Table_a0582ccb94a64603a5fbdd0e531a9edf [id=6f7fece7a8d54d4786f1bab10da9129c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.573550 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f71d3dd9af2543299296f21381d1d563. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.574487 27840 leader_election.cc:290] T a45604ad611e418e84dab793e04101a0 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:11.569638 27693 raft_consensus.cc:738] T 0062307c6ee54aada74a08cf0d087f17 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.576606 27693 consensus_queue.cc:260] T 0062307c6ee54aada74a08cf0d087f17 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.578267 27693 ts_tablet_manager.cc:1428] T 0062307c6ee54aada74a08cf0d087f17 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.010s user 0.002s sys 0.001s
I20250903 20:05:11.580137 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 912eee5b48ef467bbd6bbe34f8aa390c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.580363 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet e1a49d0599654552bf1c72857031699b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:11.583339 27685 ts_tablet_manager.cc:1918] T f3a33e90dc1f4540aee457de96f5b95b P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:11.583591 27685 log.cc:1199] T f3a33e90dc1f4540aee457de96f5b95b P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/f3a33e90dc1f4540aee457de96f5b95b
I20250903 20:05:11.584103 27685 ts_tablet_manager.cc:1939] T f3a33e90dc1f4540aee457de96f5b95b P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:11.585335 27685 tablet_replica.cc:331] T a45604ad611e418e84dab793e04101a0 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:11.585604 27685 raft_consensus.cc:2241] T a45604ad611e418e84dab793e04101a0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:11.585829 27685 raft_consensus.cc:2270] T a45604ad611e418e84dab793e04101a0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:11.586783 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 639ecb9caa4a423899ab8eb35bb716d7 (DEFAULT_TABLE table=default.Table_646dcf6e43a24b31aaa79d2f93c58a14 [id=ad8b3a9bedef4c188b89640ab5e2473a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.587248 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 639ecb9caa4a423899ab8eb35bb716d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.588845 27685 ts_tablet_manager.cc:1905] T a45604ad611e418e84dab793e04101a0 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:11.589843 27693 tablet_bootstrap.cc:492] T 12a7be47c9c2406596c9e00800074a7c P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:11.590610 27213 tablet_service.cc:1468] Processing CreateTablet for tablet f71d3dd9af2543299296f21381d1d563 (DEFAULT_TABLE table=default.Table_a0582ccb94a64603a5fbdd0e531a9edf [id=6f7fece7a8d54d4786f1bab10da9129c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.592594 27693 tablet_bootstrap.cc:654] T 12a7be47c9c2406596c9e00800074a7c P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:11.593117 27051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.125:44059, user_credentials={real_user=slave}} blocked reactor thread for 49103.6us
W20250903 20:05:11.593580 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 9a76a17b64a74fbfbb4f2e92a02ae1cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a76a17b64a74fbfbb4f2e92a02ae1cd already in progress: creating tablet
W20250903 20:05:11.594024 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet e20fcd28c3ae4355a033d2b8429f84e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e20fcd28c3ae4355a033d2b8429f84e2 already in progress: creating tablet
I20250903 20:05:11.594556 27223 tablet_service.cc:1468] Processing CreateTablet for tablet 4be9bdc390de4a74a396003c9c3c2b71 (DEFAULT_TABLE table=default.Table_452fdb47678d4520828b9a3db6ddb237 [id=d945a830c86244bca66cb2652e64966d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.596267 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f71d3dd9af2543299296f21381d1d563. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.596565 27223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4be9bdc390de4a74a396003c9c3c2b71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.596768 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ba18e3f8cb148d484010cb4e72510b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:11.542999 27692 tablet_bootstrap.cc:492] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:11.543334 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 59e120e5e39e4de8a239f13545e4b1af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.543676 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet a45604ad611e418e84dab793e04101a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.574241 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a45604ad611e418e84dab793e04101a0" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:11.598775 27692 ts_tablet_manager.cc:1397] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.099s user 0.013s sys 0.021s
I20250903 20:05:11.598881 27525 raft_consensus.cc:2466] T a45604ad611e418e84dab793e04101a0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:11.599860 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a45604ad611e418e84dab793e04101a0" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:11.599931 27692 raft_consensus.cc:357] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.600198 27692 raft_consensus.cc:383] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.600320 27692 raft_consensus.cc:738] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.600673 27692 consensus_queue.cc:260] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.601392 27692 ts_tablet_manager.cc:1428] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:11.601658 27183 leader_election.cc:304] T a45604ad611e418e84dab793e04101a0 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:11.601838 27692 tablet_bootstrap.cc:492] T f3a33e90dc1f4540aee457de96f5b95b P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
W20250903 20:05:11.601883 27183 raft_consensus.cc:2684] T a45604ad611e418e84dab793e04101a0 P 89f8f0ee68b940db9ce250c0604f934a: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250903 20:05:11.603588 27695 tablet_bootstrap.cc:654] T b5bdc27275cd4156afe767a1fb33a8bd P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.604918 27692 tablet_bootstrap.cc:654] T f3a33e90dc1f4540aee457de96f5b95b P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:11.606050 27184 leader_election.cc:343] T a45604ad611e418e84dab793e04101a0 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:11.606256 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet 4bd2650e4d9e40f5ba5c09e367918224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:11.601168 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet f3a33e90dc1f4540aee457de96f5b95b (table default.Table_f6203681100e4faeb016f0e6afc283f7 [id=91268ab201a4423faedd970b9a296152]) successfully deleted
I20250903 20:05:11.608232 27502 tablet_service.cc:1468] Processing CreateTablet for tablet 912eee5b48ef467bbd6bbe34f8aa390c (DEFAULT_TABLE table=default.Table_97be735f442746fe92867a2df71838a0 [id=fdc7899c8b334d309b834ae1daa14fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.608633 27692 tablet_bootstrap.cc:492] T f3a33e90dc1f4540aee457de96f5b95b P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:11.608894 27692 ts_tablet_manager.cc:1397] T f3a33e90dc1f4540aee457de96f5b95b P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20250903 20:05:11.608860 27499 tablet_service.cc:1468] Processing CreateTablet for tablet 639ecb9caa4a423899ab8eb35bb716d7 (DEFAULT_TABLE table=default.Table_646dcf6e43a24b31aaa79d2f93c58a14 [id=ad8b3a9bedef4c188b89640ab5e2473a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.608963 27503 tablet_service.cc:1468] Processing CreateTablet for tablet 4be9bdc390de4a74a396003c9c3c2b71 (DEFAULT_TABLE table=default.Table_452fdb47678d4520828b9a3db6ddb237 [id=d945a830c86244bca66cb2652e64966d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.608660 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 912eee5b48ef467bbd6bbe34f8aa390c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.609341 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 639ecb9caa4a423899ab8eb35bb716d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.609824 27692 raft_consensus.cc:357] T f3a33e90dc1f4540aee457de96f5b95b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.610047 27692 raft_consensus.cc:383] T f3a33e90dc1f4540aee457de96f5b95b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.610136 27692 raft_consensus.cc:738] T f3a33e90dc1f4540aee457de96f5b95b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.610425 27684 tablet_replica.cc:331] T 12a7be47c9c2406596c9e00800074a7c P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:11.610471 27692 consensus_queue.cc:260] T f3a33e90dc1f4540aee457de96f5b95b P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.610673 27684 raft_consensus.cc:2241] T 12a7be47c9c2406596c9e00800074a7c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:11.611039 27684 raft_consensus.cc:2270] T 12a7be47c9c2406596c9e00800074a7c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:11.611040 27692 ts_tablet_manager.cc:1428] T f3a33e90dc1f4540aee457de96f5b95b P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250903 20:05:11.611678 27692 tablet_bootstrap.cc:492] T 99a21a3e49934a79a66700c0b0d86307 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:11.612955 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e0e02fcc9f93461182e556c4623dbe7c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:11.613077 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 7e0810c96eb24b8ab4278038801be245 (table default.Table_682ea2b65fb14ff3b2ce593e15cacba9 [id=790f57164c7d4183a3df17ba7f230d24]) successfully deleted
W20250903 20:05:11.613629 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0e02fcc9f93461182e556c4623dbe7c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:11.614250 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ef30e0f336384b50bf8293fe525252cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:11.614768 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef30e0f336384b50bf8293fe525252cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:11.615103 27692 tablet_bootstrap.cc:654] T 99a21a3e49934a79a66700c0b0d86307 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.615087 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e3d549d1397a4167a21010a1041c6ab2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:11.615605 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3d549d1397a4167a21010a1041c6ab2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:11.616782 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_cc37561faad74a2093d52d313b9f1010"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:11.616932 27684 ts_tablet_manager.cc:1905] T 12a7be47c9c2406596c9e00800074a7c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:11.617274 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc37561faad74a2093d52d313b9f1010 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:11.620741 27693 tablet_bootstrap.cc:492] T 12a7be47c9c2406596c9e00800074a7c P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:11.620977 27693 ts_tablet_manager.cc:1397] T 12a7be47c9c2406596c9e00800074a7c P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20250903 20:05:11.622154 27693 raft_consensus.cc:357] T 12a7be47c9c2406596c9e00800074a7c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.622478 27693 raft_consensus.cc:383] T 12a7be47c9c2406596c9e00800074a7c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.622567 27693 raft_consensus.cc:738] T 12a7be47c9c2406596c9e00800074a7c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.622867 27693 consensus_queue.cc:260] T 12a7be47c9c2406596c9e00800074a7c P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.608882 27506 tablet_service.cc:1468] Processing CreateTablet for tablet f71d3dd9af2543299296f21381d1d563 (DEFAULT_TABLE table=default.Table_a0582ccb94a64603a5fbdd0e531a9edf [id=6f7fece7a8d54d4786f1bab10da9129c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.623865 27693 ts_tablet_manager.cc:1428] T 12a7be47c9c2406596c9e00800074a7c P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20250903 20:05:11.625185 27693 tablet_bootstrap.cc:492] T e722212b281a405fb08908df0a42d864 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:11.627977 27693 tablet_bootstrap.cc:654] T e722212b281a405fb08908df0a42d864 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.630965 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet d1e1add08013459ba0dab3b7e66f0136 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:11.631350 27687 tablet_replica.cc:331] T d1e1add08013459ba0dab3b7e66f0136 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:11.631614 27687 raft_consensus.cc:2241] T d1e1add08013459ba0dab3b7e66f0136 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:11.631844 27687 raft_consensus.cc:2270] T d1e1add08013459ba0dab3b7e66f0136 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:11.632848 27687 ts_tablet_manager.cc:1905] T d1e1add08013459ba0dab3b7e66f0136 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:11.636190 27800 raft_consensus.cc:491] T 59e120e5e39e4de8a239f13545e4b1af P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:11.636379 27800 raft_consensus.cc:513] T 59e120e5e39e4de8a239f13545e4b1af P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.637212 27800 leader_election.cc:290] T 59e120e5e39e4de8a239f13545e4b1af P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:11.637858 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59e120e5e39e4de8a239f13545e4b1af" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:11.638574 27462 leader_election.cc:343] T 59e120e5e39e4de8a239f13545e4b1af P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:11.639536 27800 raft_consensus.cc:491] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:11.639705 27800 raft_consensus.cc:513] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.640339 27800 leader_election.cc:290] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:11.640511 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet e0d1b29d7f1e422e8bb5b8bbc33423e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:11.641057 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eaa11d17303142ff935705c16f0b9690" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:11.644753 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f71d3dd9af2543299296f21381d1d563. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.645401 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4be9bdc390de4a74a396003c9c3c2b71. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:11.650847 27462 leader_election.cc:343] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:11.652756 27462 leader_election.cc:343] T 59e120e5e39e4de8a239f13545e4b1af P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 59e120e5e39e4de8a239f13545e4b1af
I20250903 20:05:11.652767 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet c12c957a36f643d9bfb60b755619cbbc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:08 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:11.652899 27462 leader_election.cc:304] T 59e120e5e39e4de8a239f13545e4b1af P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:11.653482 27800 raft_consensus.cc:2747] T 59e120e5e39e4de8a239f13545e4b1af P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:11.653913 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 0a266b1bbf1540ac86d49c56df2dd4ad with delete_type TABLET_DATA_DELETED (replaced by tablet 73f65e7515994b898eec2ea23903aff4) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:11.654973 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 43f88abc8f3640a48598c07c3d76755d with delete_type TABLET_DATA_DELETED (replaced by tablet 68cff0e416a54a5eb37071f5a12f07ab) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:11.655740 27184 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 51759.6us
I20250903 20:05:11.655885 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 12a7be47c9c2406596c9e00800074a7c with delete_type TABLET_DATA_DELETED (replaced by tablet 912eee5b48ef467bbd6bbe34f8aa390c) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:11.658565 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 0062307c6ee54aada74a08cf0d087f17 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:11.661585 27687 ts_tablet_manager.cc:1918] T d1e1add08013459ba0dab3b7e66f0136 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:11.658125 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eaa11d17303142ff935705c16f0b9690" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:11.662359 27248 raft_consensus.cc:2466] T eaa11d17303142ff935705c16f0b9690 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:11.662420 27687 log.cc:1199] T d1e1add08013459ba0dab3b7e66f0136 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/d1e1add08013459ba0dab3b7e66f0136
I20250903 20:05:11.662819 27687 ts_tablet_manager.cc:1939] T d1e1add08013459ba0dab3b7e66f0136 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:11.663354 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_19e44161daa5469da23656554f451dd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:11.664220 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19e44161daa5469da23656554f451dd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:11.665024 27462 leader_election.cc:304] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:11.665612 27800 raft_consensus.cc:2802] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:11.665741 27800 raft_consensus.cc:491] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:11.665845 27800 raft_consensus.cc:3058] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:11.666054 27687 tablet_replica.cc:331] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:11.666203 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet d1e1add08013459ba0dab3b7e66f0136 (table default.Table_4ace9616711f4778a94a0827ff79cc07 [id=e20fa37b0e0343ed8ab3a7e814358ae6]) successfully deleted
I20250903 20:05:11.666311 27687 raft_consensus.cc:2241] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:11.666519 27687 raft_consensus.cc:2270] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:11.667109 27687 ts_tablet_manager.cc:1905] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:11.667811 27692 tablet_bootstrap.cc:492] T 99a21a3e49934a79a66700c0b0d86307 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:11.667759 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_2f2c8693b96741448fbd96ff5eb9ad8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:11.667965 27692 ts_tablet_manager.cc:1397] T 99a21a3e49934a79a66700c0b0d86307 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.001s
W20250903 20:05:11.668352 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f2c8693b96741448fbd96ff5eb9ad8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:11.669103 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_db997f19dc99468facd46c7e27217f04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:11.669628 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db997f19dc99468facd46c7e27217f04 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:11.670799 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_9d09f11dcb4b4bf09c870934bf1b6acd" } new_table_name: "default.Table_a1ac671d35424891be575ac9621ce309" modify_external_catalogs: true
I20250903 20:05:11.668727 27800 raft_consensus.cc:513] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.672058 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eaa11d17303142ff935705c16f0b9690" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
W20250903 20:05:11.672629 27462 leader_election.cc:343] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:11.661501 27684 ts_tablet_manager.cc:1918] T 12a7be47c9c2406596c9e00800074a7c P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:11.673110 27684 log.cc:1199] T 12a7be47c9c2406596c9e00800074a7c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/12a7be47c9c2406596c9e00800074a7c
I20250903 20:05:11.673641 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d5626bea3c88471b846f37ae4ed79715" } modify_external_catalogs: true
I20250903 20:05:11.673844 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d5626bea3c88471b846f37ae4ed79715" } modify_external_catalogs: true
I20250903 20:05:11.674489 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_80ad865643824ab88e2d63eb1b66d079" } new_table_name: "default.Table_05d58d894486468a8a15aaa4b4c0cabb" modify_external_catalogs: true
I20250903 20:05:11.675473 27800 leader_election.cc:290] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:11.680707 27687 ts_tablet_manager.cc:1918] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:11.680989 27687 log.cc:1199] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/e0d1b29d7f1e422e8bb5b8bbc33423e8
I20250903 20:05:11.681011 27684 ts_tablet_manager.cc:1939] T 12a7be47c9c2406596c9e00800074a7c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:11.668866 27692 raft_consensus.cc:357] T 99a21a3e49934a79a66700c0b0d86307 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.681350 27692 raft_consensus.cc:383] T 99a21a3e49934a79a66700c0b0d86307 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.681452 27692 raft_consensus.cc:738] T 99a21a3e49934a79a66700c0b0d86307 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.681697 27692 consensus_queue.cc:260] T 99a21a3e49934a79a66700c0b0d86307 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.682308 27692 ts_tablet_manager.cc:1428] T 99a21a3e49934a79a66700c0b0d86307 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250903 20:05:11.682643 27692 tablet_bootstrap.cc:492] T e1a49d0599654552bf1c72857031699b P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
W20250903 20:05:11.683465 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2a5c74a281b44019872a6442d41bc373 consensus state has no leader
I20250903 20:05:11.683624 27687 ts_tablet_manager.cc:1939] T e0d1b29d7f1e422e8bb5b8bbc33423e8 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:11.685598 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet e0d1b29d7f1e422e8bb5b8bbc33423e8 (table default.Table_82f29a4a80194976bc7c8f864ec877a3 [id=49551de884e7488fb6db7249f834e12f]) successfully deleted
I20250903 20:05:11.686123 27687 tablet_replica.cc:331] T c12c957a36f643d9bfb60b755619cbbc P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:11.686452 27687 raft_consensus.cc:2241] T c12c957a36f643d9bfb60b755619cbbc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:11.686756 27687 raft_consensus.cc:2270] T c12c957a36f643d9bfb60b755619cbbc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:11.687882 27687 ts_tablet_manager.cc:1905] T c12c957a36f643d9bfb60b755619cbbc P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:11.691524 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eaa11d17303142ff935705c16f0b9690" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:11.691762 27248 raft_consensus.cc:3058] T eaa11d17303142ff935705c16f0b9690 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:11.697332 27687 ts_tablet_manager.cc:1918] T c12c957a36f643d9bfb60b755619cbbc P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:11.697628 27687 log.cc:1199] T c12c957a36f643d9bfb60b755619cbbc P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/c12c957a36f643d9bfb60b755619cbbc
I20250903 20:05:11.698287 27687 ts_tablet_manager.cc:1939] T c12c957a36f643d9bfb60b755619cbbc P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:11.699728 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet c12c957a36f643d9bfb60b755619cbbc (table default.Table_7d476d0d69e246e28664c7c5b84f3ba2 [id=7c51b3299c2948189100608937aa1ef9]) successfully deleted
I20250903 20:05:11.700249 27687 tablet_replica.cc:331] T 0a266b1bbf1540ac86d49c56df2dd4ad P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:11.700590 27687 raft_consensus.cc:2241] T 0a266b1bbf1540ac86d49c56df2dd4ad P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:11.700876 27687 raft_consensus.cc:2270] T 0a266b1bbf1540ac86d49c56df2dd4ad P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:11.701692 27687 ts_tablet_manager.cc:1905] T 0a266b1bbf1540ac86d49c56df2dd4ad P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:11.703502 27692 tablet_bootstrap.cc:654] T e1a49d0599654552bf1c72857031699b P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.703764 27840 raft_consensus.cc:491] T 99a21a3e49934a79a66700c0b0d86307 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:11.704186 27840 raft_consensus.cc:513] T 99a21a3e49934a79a66700c0b0d86307 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.704861 27840 leader_election.cc:290] T 99a21a3e49934a79a66700c0b0d86307 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:11.705590 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99a21a3e49934a79a66700c0b0d86307" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:11.705813 27525 raft_consensus.cc:2466] T 99a21a3e49934a79a66700c0b0d86307 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:11.707336 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99a21a3e49934a79a66700c0b0d86307" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:11.708858 27183 leader_election.cc:304] T 99a21a3e49934a79a66700c0b0d86307 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
W20250903 20:05:11.710054 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0bdcbaa0169c4ab7ba6ec038b50482fe consensus state has no leader
I20250903 20:05:11.713783 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_51d81329a9c84f5bb4a0973d5410d115"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:11.714520 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51d81329a9c84f5bb4a0973d5410d115 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:11.715144 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b6f25e1f7a484577a9cbff11e0deb857" } new_table_name: "default.Table_81f1c8f3063b4be686402371c42189fa" modify_external_catalogs: true
I20250903 20:05:11.714516 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6f5d3b1c0701490fb110f36cf852a96d" } new_table_name: "default.Table_08918e8b515e4487bf4aee0a33234bd5" modify_external_catalogs: true
I20250903 20:05:11.717031 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_509afdab983d4eed97a5d124ff19ab3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:11.717563 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_509afdab983d4eed97a5d124ff19ab3b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:11.717700 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_26a46b6cf89f459b99526bc640729603" } modify_external_catalogs: true
I20250903 20:05:11.717882 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_26a46b6cf89f459b99526bc640729603" } modify_external_catalogs: true
I20250903 20:05:11.718585 27840 raft_consensus.cc:2802] T 99a21a3e49934a79a66700c0b0d86307 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:11.718709 27840 raft_consensus.cc:491] T 99a21a3e49934a79a66700c0b0d86307 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:11.718823 27840 raft_consensus.cc:3058] T 99a21a3e49934a79a66700c0b0d86307 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:11.719089 27692 tablet_bootstrap.cc:492] T e1a49d0599654552bf1c72857031699b P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:11.719300 27692 ts_tablet_manager.cc:1397] T e1a49d0599654552bf1c72857031699b P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20250903 20:05:11.720175 27692 raft_consensus.cc:357] T e1a49d0599654552bf1c72857031699b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.720373 27692 raft_consensus.cc:383] T e1a49d0599654552bf1c72857031699b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.720474 27692 raft_consensus.cc:738] T e1a49d0599654552bf1c72857031699b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.720753 27692 consensus_queue.cc:260] T e1a49d0599654552bf1c72857031699b P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.721262 27692 ts_tablet_manager.cc:1428] T e1a49d0599654552bf1c72857031699b P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:11.721678 27692 tablet_bootstrap.cc:492] T be278233c7c34061988b267a44ee674f P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:11.721686 27840 raft_consensus.cc:513] T 99a21a3e49934a79a66700c0b0d86307 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.722677 27248 raft_consensus.cc:2466] T eaa11d17303142ff935705c16f0b9690 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
W20250903 20:05:11.723179 27184 leader_election.cc:343] T 99a21a3e49934a79a66700c0b0d86307 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:11.723203 27462 leader_election.cc:304] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:11.723498 27800 raft_consensus.cc:2802] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:11.723634 27800 raft_consensus.cc:695] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:11.723968 27800 consensus_queue.cc:237] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.732921 27692 tablet_bootstrap.cc:654] T be278233c7c34061988b267a44ee674f P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.744099 27685 ts_tablet_manager.cc:1918] T a45604ad611e418e84dab793e04101a0 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:11.744278 27685 log.cc:1199] T a45604ad611e418e84dab793e04101a0 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/a45604ad611e418e84dab793e04101a0
I20250903 20:05:11.744692 27685 ts_tablet_manager.cc:1939] T a45604ad611e418e84dab793e04101a0 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:11.745712 27685 tablet_replica.cc:331] T e1a49d0599654552bf1c72857031699b P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:11.745968 27685 raft_consensus.cc:2241] T e1a49d0599654552bf1c72857031699b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:11.746145 27685 raft_consensus.cc:2270] T e1a49d0599654552bf1c72857031699b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:11.747056 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_6f20b7d359084b4faef6f2eb3fdbab70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:11.747682 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f20b7d359084b4faef6f2eb3fdbab70 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:11.751840 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b88b7b0fe749402e90c3826d01e8cc8e" } modify_external_catalogs: true
I20250903 20:05:11.752017 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b88b7b0fe749402e90c3826d01e8cc8e" } modify_external_catalogs: true
I20250903 20:05:11.752696 27525 raft_consensus.cc:3058] T b6b4713936134681aaef1732d46c6ab2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:11.753937 27248 raft_consensus.cc:1273] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Refusing update from remote peer 51578fb7bceb4e119b3ab1e58c4f5528: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 20:05:11.754271 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_db997f19dc99468facd46c7e27217f04" } new_table_name: "default.Table_bffb6801f48c4124bfe24c49586d6e48" modify_external_catalogs: true
I20250903 20:05:11.754743 27754 consensus_queue.cc:1035] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:11.756114 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet a45604ad611e418e84dab793e04101a0 (table default.Table_7d8b7496fbc04f35bab6d6da573d4868 [id=5383eb1038e6480893e6eab6c7f7dd39]) successfully deleted
I20250903 20:05:11.758819 27687 ts_tablet_manager.cc:1918] T 0a266b1bbf1540ac86d49c56df2dd4ad P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:11.760339 27695 tablet_bootstrap.cc:492] T b5bdc27275cd4156afe767a1fb33a8bd P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:11.760532 27687 log.cc:1199] T 0a266b1bbf1540ac86d49c56df2dd4ad P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/0a266b1bbf1540ac86d49c56df2dd4ad
I20250903 20:05:11.760547 27695 ts_tablet_manager.cc:1397] T b5bdc27275cd4156afe767a1fb33a8bd P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.201s user 0.000s sys 0.004s
I20250903 20:05:11.761044 27687 ts_tablet_manager.cc:1939] T 0a266b1bbf1540ac86d49c56df2dd4ad P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:11.761418 27695 raft_consensus.cc:357] T b5bdc27275cd4156afe767a1fb33a8bd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.761624 27695 raft_consensus.cc:383] T b5bdc27275cd4156afe767a1fb33a8bd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.761708 27695 raft_consensus.cc:738] T b5bdc27275cd4156afe767a1fb33a8bd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.762270 27695 consensus_queue.cc:260] T b5bdc27275cd4156afe767a1fb33a8bd P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.762498 27685 ts_tablet_manager.cc:1905] T e1a49d0599654552bf1c72857031699b P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:11.766184 27687 tablet_replica.cc:331] T 43f88abc8f3640a48598c07c3d76755d P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:11.766844 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 0a266b1bbf1540ac86d49c56df2dd4ad (table default.Table_d5626bea3c88471b846f37ae4ed79715 [id=75e5cb0bab9b46a6b5caec7c99efc7b8]) successfully deleted
I20250903 20:05:11.767177 27687 raft_consensus.cc:2241] T 43f88abc8f3640a48598c07c3d76755d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:11.766418 27695 ts_tablet_manager.cc:1428] T b5bdc27275cd4156afe767a1fb33a8bd P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250903 20:05:11.768317 27695 tablet_bootstrap.cc:492] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:11.772821 27823 tablet.cc:1722] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:11.768811 27844 tablet.cc:1722] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:11.782936 27687 raft_consensus.cc:2270] T 43f88abc8f3640a48598c07c3d76755d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:11.784862 27823 tablet.cc:1717] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a: Skipping requested alter to schema version 2, tablet already version 2
I20250903 20:05:11.786733 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99a21a3e49934a79a66700c0b0d86307" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:11.786954 27526 raft_consensus.cc:3058] T 99a21a3e49934a79a66700c0b0d86307 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:11.787540 27840 leader_election.cc:290] T 99a21a3e49934a79a66700c0b0d86307 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:11.784637 27844 tablet.cc:1717] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528: Skipping requested alter to schema version 2, tablet already version 2
I20250903 20:05:11.788556 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99a21a3e49934a79a66700c0b0d86307" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
W20250903 20:05:11.790954 27184 leader_election.cc:343] T 99a21a3e49934a79a66700c0b0d86307 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:11.793509 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9367b3a22f624c4087848099de32c080 consensus state has no leader
I20250903 20:05:11.793408 27687 ts_tablet_manager.cc:1905] T 43f88abc8f3640a48598c07c3d76755d P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:11.797827 26890 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 57306.8us
I20250903 20:05:11.804708 27064 master_service.cc:948] ReplaceTablet: received request to replace tablet 5bc3b9771a0042fd87e706204dcbb712 from {username='slave'} at 127.0.0.1:45936
W20250903 20:05:11.808483 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7cc6cf579e2642e99fefed723b0ee900 has no consensus state
I20250903 20:05:11.808952 27058 master_service.cc:948] ReplaceTablet: received request to replace tablet a10b079e7d5a440580ac00195fe88c65 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:11.811796 27695 tablet_bootstrap.cc:654] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.812245 27687 ts_tablet_manager.cc:1918] T 43f88abc8f3640a48598c07c3d76755d P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:11.812424 27687 log.cc:1199] T 43f88abc8f3640a48598c07c3d76755d P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/43f88abc8f3640a48598c07c3d76755d
I20250903 20:05:11.812809 27687 ts_tablet_manager.cc:1939] T 43f88abc8f3640a48598c07c3d76755d P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:11.814074 27684 tablet_replica.cc:331] T 59e120e5e39e4de8a239f13545e4b1af P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:11.814297 27684 raft_consensus.cc:2241] T 59e120e5e39e4de8a239f13545e4b1af P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:11.814498 27684 raft_consensus.cc:2270] T 59e120e5e39e4de8a239f13545e4b1af P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:11.814841 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 12a7be47c9c2406596c9e00800074a7c (table default.Table_97be735f442746fe92867a2df71838a0 [id=fdc7899c8b334d309b834ae1daa14fa0]) successfully deleted
I20250903 20:05:11.815742 27684 ts_tablet_manager.cc:1905] T 59e120e5e39e4de8a239f13545e4b1af P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:11.815886 27685 ts_tablet_manager.cc:1918] T e1a49d0599654552bf1c72857031699b P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:11.816040 27685 log.cc:1199] T e1a49d0599654552bf1c72857031699b P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/e1a49d0599654552bf1c72857031699b
I20250903 20:05:11.816390 27685 ts_tablet_manager.cc:1939] T e1a49d0599654552bf1c72857031699b P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:11.817373 27685 tablet_replica.cc:331] T 7ba18e3f8cb148d484010cb4e72510b2 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:11.817606 27685 raft_consensus.cc:2241] T 7ba18e3f8cb148d484010cb4e72510b2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:11.817801 27685 raft_consensus.cc:2270] T 7ba18e3f8cb148d484010cb4e72510b2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:11.822517 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet e1a49d0599654552bf1c72857031699b (table default.Table_f9e41274ca534e46b7906480d7fc32fe [id=febabbd421d241948dd40f5480c052bb]) successfully deleted
I20250903 20:05:11.823035 27685 ts_tablet_manager.cc:1905] T 7ba18e3f8cb148d484010cb4e72510b2 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:11.825531 27525 raft_consensus.cc:1273] T b6b4713936134681aaef1732d46c6ab2 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Refusing update from remote peer 51578fb7bceb4e119b3ab1e58c4f5528: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 20:05:11.826308 27754 consensus_queue.cc:1035] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:11.828498 27526 raft_consensus.cc:2466] T 99a21a3e49934a79a66700c0b0d86307 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:11.828969 27183 leader_election.cc:304] T 99a21a3e49934a79a66700c0b0d86307 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:11.834106 27840 raft_consensus.cc:2802] T 99a21a3e49934a79a66700c0b0d86307 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:11.834547 27840 raft_consensus.cc:695] T 99a21a3e49934a79a66700c0b0d86307 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:11.834957 27840 consensus_queue.cc:237] T 99a21a3e49934a79a66700c0b0d86307 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.835786 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 43f88abc8f3640a48598c07c3d76755d (table default.Table_a5f8f31c12b742b597c8b35d662a607f [id=71d850a0cea14b0ba337a822ad99d513]) successfully deleted
I20250903 20:05:11.836719 27687 tablet_replica.cc:331] T 12a7be47c9c2406596c9e00800074a7c P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:11.837836 27687 raft_consensus.cc:2241] T 12a7be47c9c2406596c9e00800074a7c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:11.838555 27687 raft_consensus.cc:2270] T 12a7be47c9c2406596c9e00800074a7c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:11.840881 27684 ts_tablet_manager.cc:1918] T 59e120e5e39e4de8a239f13545e4b1af P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:11.841013 27684 log.cc:1199] T 59e120e5e39e4de8a239f13545e4b1af P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/59e120e5e39e4de8a239f13545e4b1af
I20250903 20:05:11.841534 27684 ts_tablet_manager.cc:1939] T 59e120e5e39e4de8a239f13545e4b1af P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:11.842768 27684 tablet_replica.cc:331] T a45604ad611e418e84dab793e04101a0 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:11.842975 27684 raft_consensus.cc:2241] T a45604ad611e418e84dab793e04101a0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:11.843151 27684 raft_consensus.cc:2270] T a45604ad611e418e84dab793e04101a0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:11.843554 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 59e120e5e39e4de8a239f13545e4b1af (table default.Table_2dd5b5956b44471da9d0b89e782d469e [id=0bfcd1b8b71d45db8d9c370bd1231a38]) successfully deleted
I20250903 20:05:11.845047 27695 tablet_bootstrap.cc:492] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:11.845396 27692 tablet_bootstrap.cc:492] T be278233c7c34061988b267a44ee674f P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:11.845546 27692 ts_tablet_manager.cc:1397] T be278233c7c34061988b267a44ee674f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.124s user 0.009s sys 0.027s
I20250903 20:05:11.845799 27684 ts_tablet_manager.cc:1905] T a45604ad611e418e84dab793e04101a0 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:11.845863 27695 ts_tablet_manager.cc:1397] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.078s user 0.003s sys 0.001s
I20250903 20:05:11.846438 27692 raft_consensus.cc:357] T be278233c7c34061988b267a44ee674f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.846627 27692 raft_consensus.cc:383] T be278233c7c34061988b267a44ee674f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.846725 27692 raft_consensus.cc:738] T be278233c7c34061988b267a44ee674f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.849673 27695 raft_consensus.cc:357] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.850661 27692 consensus_queue.cc:260] T be278233c7c34061988b267a44ee674f P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.850695 27695 raft_consensus.cc:383] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.851212 27695 raft_consensus.cc:738] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.852241 27692 ts_tablet_manager.cc:1428] T be278233c7c34061988b267a44ee674f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250903 20:05:11.852547 27692 tablet_bootstrap.cc:492] T 7ba18e3f8cb148d484010cb4e72510b2 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:11.852156 27695 consensus_queue.cc:260] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.854708 27822 tablet.cc:1722] T b6b4713936134681aaef1732d46c6ab2 P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:11.855829 27502 tablet_service.cc:1468] Processing CreateTablet for tablet cb4d99139516405bb65c7480ddf259f1 (DEFAULT_TABLE table=default.Table_a2fef2e8171c466e86ce74826ba6e281 [id=cbcfc7067aca4796b39e3eb195f38903]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.856269 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb4d99139516405bb65c7480ddf259f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.858126 27692 tablet_bootstrap.cc:654] T 7ba18e3f8cb148d484010cb4e72510b2 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.858587 27503 tablet_service.cc:1468] Processing CreateTablet for tablet fe902f1b58924be1ac3a5585413f7789 (DEFAULT_TABLE table=default.Table_1840897b7e7f4f1e8052d16ca5e77a4f [id=78785eeb2c464e6d9f0cc5c8792376ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.858650 27063 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet b6b4713936134681aaef1732d46c6ab2 (table default.Table_81f1c8f3063b4be686402371c42189fa [id=d0a75545b91c48bba47fb581bb0d1a96]). Expected version 2 got 0
I20250903 20:05:11.858951 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe902f1b58924be1ac3a5585413f7789. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.859586 27063 master_service.cc:948] ReplaceTablet: received request to replace tablet 0bdcbaa0169c4ab7ba6ec038b50482fe from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:11.860123 27806 mvcc.cc:204] Tried to move back new op lower bound from 7196384918529671168 to 7196384916797779968. Current Snapshot: MvccSnapshot[applied={T|T < 7196384918529671168}]
I20250903 20:05:11.861003 27822 tablet.cc:1717] T b6b4713936134681aaef1732d46c6ab2 P e72ef1dbcc5143cf9f04a808f101c80f: Skipping requested alter to schema version 2, tablet already version 2
I20250903 20:05:11.862553 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 516830fc42e84af693b3af52caee30ee (DEFAULT_TABLE table=default.Table_afa82ec7c58343adb623622c56033b9c [id=098bc9a9b0a24a4a9edb7859310ac9b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.862942 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 516830fc42e84af693b3af52caee30ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.865016 27685 ts_tablet_manager.cc:1918] T 7ba18e3f8cb148d484010cb4e72510b2 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:11.865152 27685 log.cc:1199] T 7ba18e3f8cb148d484010cb4e72510b2 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/7ba18e3f8cb148d484010cb4e72510b2
I20250903 20:05:11.865233 27695 ts_tablet_manager.cc:1428] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.019s user 0.003s sys 0.000s
I20250903 20:05:11.865571 27685 ts_tablet_manager.cc:1939] T 7ba18e3f8cb148d484010cb4e72510b2 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:11.865948 27499 tablet_service.cc:1468] Processing CreateTablet for tablet fa05f54ebc324b40855fa3288760e9e9 (DEFAULT_TABLE table=default.Table_dacd8238c8334c1c9f66e745f5036a06 [id=7ec413d945394dcfa36492f7b59aaaa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.866314 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa05f54ebc324b40855fa3288760e9e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.866814 27062 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet b6b4713936134681aaef1732d46c6ab2 (table default.Table_81f1c8f3063b4be686402371c42189fa [id=d0a75545b91c48bba47fb581bb0d1a96]). Expected version 2 got 0
I20250903 20:05:11.867549 27062 master_service.cc:948] ReplaceTablet: received request to replace tablet e20fcd28c3ae4355a033d2b8429f84e2 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:11.868064 27062 master_service.cc:948] ReplaceTablet: received request to replace tablet 04ccb3e4a8bd4789b9f19edd4a5b6145 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:11.869611 27693 tablet_bootstrap.cc:492] T e722212b281a405fb08908df0a42d864 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:11.869758 27693 ts_tablet_manager.cc:1397] T e722212b281a405fb08908df0a42d864 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.246s user 0.016s sys 0.044s
I20250903 20:05:11.869336 27687 ts_tablet_manager.cc:1905] T 12a7be47c9c2406596c9e00800074a7c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:11.870652 27693 raft_consensus.cc:357] T e722212b281a405fb08908df0a42d864 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.870841 27693 raft_consensus.cc:383] T e722212b281a405fb08908df0a42d864 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.870934 27693 raft_consensus.cc:738] T e722212b281a405fb08908df0a42d864 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.871197 27693 consensus_queue.cc:260] T e722212b281a405fb08908df0a42d864 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.865614 27695 tablet_bootstrap.cc:492] T 4692dd93b76c42c4883c7e22c6032250 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:11.874063 27695 tablet_bootstrap.cc:654] T 4692dd93b76c42c4883c7e22c6032250 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.875267 27693 ts_tablet_manager.cc:1428] T e722212b281a405fb08908df0a42d864 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250903 20:05:11.875653 27685 tablet_replica.cc:331] T 4bd2650e4d9e40f5ba5c09e367918224 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:11.876014 27362 tablet_service.cc:1468] Processing CreateTablet for tablet cb4d99139516405bb65c7480ddf259f1 (DEFAULT_TABLE table=default.Table_a2fef2e8171c466e86ce74826ba6e281 [id=cbcfc7067aca4796b39e3eb195f38903]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.876454 27359 tablet_service.cc:1468] Processing CreateTablet for tablet 516830fc42e84af693b3af52caee30ee (DEFAULT_TABLE table=default.Table_afa82ec7c58343adb623622c56033b9c [id=098bc9a9b0a24a4a9edb7859310ac9b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.876629 27360 tablet_service.cc:1468] Processing CreateTablet for tablet fe902f1b58924be1ac3a5585413f7789 (DEFAULT_TABLE table=default.Table_1840897b7e7f4f1e8052d16ca5e77a4f [id=78785eeb2c464e6d9f0cc5c8792376ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.876849 27685 raft_consensus.cc:2241] T 4bd2650e4d9e40f5ba5c09e367918224 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:11.876853 27366 tablet_service.cc:1468] Processing CreateTablet for tablet fa05f54ebc324b40855fa3288760e9e9 (DEFAULT_TABLE table=default.Table_dacd8238c8334c1c9f66e745f5036a06 [id=7ec413d945394dcfa36492f7b59aaaa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.877027 27685 raft_consensus.cc:2270] T 4bd2650e4d9e40f5ba5c09e367918224 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:11.876749 27693 tablet_bootstrap.cc:492] T 59e120e5e39e4de8a239f13545e4b1af P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:11.877233 27363 tablet_service.cc:1468] Processing CreateTablet for tablet cd92e3dafca6419f8eba7a4ef51a911e (DEFAULT_TABLE table=default.Table_13dbf08231794aa085d17ca500361b14 [id=b15b2b0164104454a23d766feea4e0de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.878412 27214 tablet_service.cc:1468] Processing CreateTablet for tablet cb4d99139516405bb65c7480ddf259f1 (DEFAULT_TABLE table=default.Table_a2fef2e8171c466e86ce74826ba6e281 [id=cbcfc7067aca4796b39e3eb195f38903]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.878804 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb4d99139516405bb65c7480ddf259f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.878995 27693 tablet_bootstrap.cc:654] T 59e120e5e39e4de8a239f13545e4b1af P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.879825 27500 tablet_service.cc:1468] Processing CreateTablet for tablet cd92e3dafca6419f8eba7a4ef51a911e (DEFAULT_TABLE table=default.Table_13dbf08231794aa085d17ca500361b14 [id=b15b2b0164104454a23d766feea4e0de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.880286 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd92e3dafca6419f8eba7a4ef51a911e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.880901 27685 ts_tablet_manager.cc:1905] T 4bd2650e4d9e40f5ba5c09e367918224 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:11.882411 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 7ba18e3f8cb148d484010cb4e72510b2 (table default.Table_9691d111178f4b9c9024a0a66fc3f03b [id=83d0f963c409436faa6c9a78bc0167cd]) successfully deleted
I20250903 20:05:11.882555 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 8605bc329627438998812ed028327f53 (DEFAULT_TABLE table=default.Table_8b0346420a89417b9ae2005be49e9f3a [id=b0b2e2d5798346708995a6b8dc028918]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.882566 27213 tablet_service.cc:1468] Processing CreateTablet for tablet fe902f1b58924be1ac3a5585413f7789 (DEFAULT_TABLE table=default.Table_1840897b7e7f4f1e8052d16ca5e77a4f [id=78785eeb2c464e6d9f0cc5c8792376ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.882968 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe902f1b58924be1ac3a5585413f7789. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.882978 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8605bc329627438998812ed028327f53. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.883157 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 8605bc329627438998812ed028327f53 (DEFAULT_TABLE table=default.Table_8b0346420a89417b9ae2005be49e9f3a [id=b0b2e2d5798346708995a6b8dc028918]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.883618 27358 tablet_service.cc:1468] Processing CreateTablet for tablet ac9b5d4f39db4f05a15316585040d142 (DEFAULT_TABLE table=default.Table_62f55f5f45434b01b7b7e2d6d1ddc6fd [id=74b402d4851f48c2bfd495e82917bb38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.883689 27840 raft_consensus.cc:491] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:11.883846 27840 raft_consensus.cc:513] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.883981 27357 tablet_service.cc:1468] Processing CreateTablet for tablet 8d2d64f103fc44d7b390559b92cbf4b4 (DEFAULT_TABLE table=default.Table_1e6e2159113540b7944963d2140ab460 [id=f0171ee670294ce58ab84483fd71ab7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.884549 27840 leader_election.cc:290] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:11.885001 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 885fff47fea14b84ac6b34a57b827b4b (DEFAULT_TABLE table=default.Table_957f51b8fb684329a6e6fac205da5164 [id=61a25739b03042f7bd5bb693b68656d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.886525 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 516830fc42e84af693b3af52caee30ee (DEFAULT_TABLE table=default.Table_afa82ec7c58343adb623622c56033b9c [id=098bc9a9b0a24a4a9edb7859310ac9b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.886682 27501 tablet_service.cc:1468] Processing CreateTablet for tablet ac9b5d4f39db4f05a15316585040d142 (DEFAULT_TABLE table=default.Table_62f55f5f45434b01b7b7e2d6d1ddc6fd [id=74b402d4851f48c2bfd495e82917bb38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.886927 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 516830fc42e84af693b3af52caee30ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.887122 27501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac9b5d4f39db4f05a15316585040d142. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.890251 27364 tablet_service.cc:1468] Processing CreateTablet for tablet 6f0cba70240f4f38a4fa5ccbcd99b3b1 (DEFAULT_TABLE table=default.Table_6ce254828dc045d39e5463fe6a48fa82 [id=7fb235b85bba4f7ab1b77bc5d2b9b58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.890581 27223 tablet_service.cc:1468] Processing CreateTablet for tablet fa05f54ebc324b40855fa3288760e9e9 (DEFAULT_TABLE table=default.Table_dacd8238c8334c1c9f66e745f5036a06 [id=7ec413d945394dcfa36492f7b59aaaa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.890947 27223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa05f54ebc324b40855fa3288760e9e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.894575 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 8d2d64f103fc44d7b390559b92cbf4b4 (DEFAULT_TABLE table=default.Table_1e6e2159113540b7944963d2140ab460 [id=f0171ee670294ce58ab84483fd71ab7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.894968 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d2d64f103fc44d7b390559b92cbf4b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.897599 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 885fff47fea14b84ac6b34a57b827b4b (DEFAULT_TABLE table=default.Table_957f51b8fb684329a6e6fac205da5164 [id=61a25739b03042f7bd5bb693b68656d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.898049 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 885fff47fea14b84ac6b34a57b827b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.898658 27356 tablet_service.cc:1468] Processing CreateTablet for tablet 7c398634d6304a12b35c0579fba4c813 (DEFAULT_TABLE table=default.Table_14af7742726b4562a02fd2c9b2895277 [id=6d72901e5102476890f466fd406a7127]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.899776 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 516830fc42e84af693b3af52caee30ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.902493 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe902f1b58924be1ac3a5585413f7789. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.902534 27385 raft_consensus.cc:3058] T c8a293c6fa034e9fb29682439cb3bcc2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:11.902606 27498 tablet_service.cc:1468] Processing CreateTablet for tablet 7c398634d6304a12b35c0579fba4c813 (DEFAULT_TABLE table=default.Table_14af7742726b4562a02fd2c9b2895277 [id=6d72901e5102476890f466fd406a7127]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.903126 27498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c398634d6304a12b35c0579fba4c813. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.910822 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 5bc3b9771a0042fd87e706204dcbb712
I20250903 20:05:11.910904 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 0bdcbaa0169c4ab7ba6ec038b50482fe
I20250903 20:05:11.911453 27063 catalog_manager.cc:6652] ReplaceTablet: tablet 0bdcbaa0169c4ab7ba6ec038b50482fe deleted and replaced by tablet 1bcb9d5f010f4297bab78c0183889238
I20250903 20:05:11.913407 27220 tablet_service.cc:1468] Processing CreateTablet for tablet 8605bc329627438998812ed028327f53 (DEFAULT_TABLE table=default.Table_8b0346420a89417b9ae2005be49e9f3a [id=b0b2e2d5798346708995a6b8dc028918]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.913882 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8605bc329627438998812ed028327f53. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.913923 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet a10b079e7d5a440580ac00195fe88c65
I20250903 20:05:11.914427 27058 catalog_manager.cc:6652] ReplaceTablet: tablet a10b079e7d5a440580ac00195fe88c65 deleted and replaced by tablet 3b3bfc63aa0b4e56bec189e834dc6a8d
I20250903 20:05:11.915329 27685 ts_tablet_manager.cc:1918] T 4bd2650e4d9e40f5ba5c09e367918224 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:11.915515 27685 log.cc:1199] T 4bd2650e4d9e40f5ba5c09e367918224 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/4bd2650e4d9e40f5ba5c09e367918224
I20250903 20:05:11.915992 27685 ts_tablet_manager.cc:1939] T 4bd2650e4d9e40f5ba5c09e367918224 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:11.915952 27058 catalog_manager.cc:5582] T 99a21a3e49934a79a66700c0b0d86307 P 89f8f0ee68b940db9ce250c0604f934a reported cstate change: term changed from 0 to 1, leader changed from <none> to 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65). New cstate: current_term: 1 leader_uuid: "89f8f0ee68b940db9ce250c0604f934a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: HEALTHY } } }
I20250903 20:05:11.916528 27058 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 99a21a3e49934a79a66700c0b0d86307 (table default.Table_7325977ff92e45bab0b59b79579ecae4 [id=428657b402d748c0a9e3942d13342b4e]). Expected version 1 got 0
I20250903 20:05:11.916884 27058 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 9367b3a22f624c4087848099de32c080 (table default.Table_08918e8b515e4487bf4aee0a33234bd5 [id=891b259c1c6b44e2a82dc982c24e6cb6]). Expected version 1 got 0
I20250903 20:05:11.917825 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 73f65e7515994b898eec2ea23903aff4
I20250903 20:05:11.918555 27226 tablet_service.cc:1468] Processing CreateTablet for tablet ac9b5d4f39db4f05a15316585040d142 (DEFAULT_TABLE table=default.Table_62f55f5f45434b01b7b7e2d6d1ddc6fd [id=74b402d4851f48c2bfd495e82917bb38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.918952 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac9b5d4f39db4f05a15316585040d142. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.918877 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_65e771add6404b41bd60f0bec5093c9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:11.919569 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65e771add6404b41bd60f0bec5093c9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:11.919900 27229 tablet_service.cc:1515] Processing DeleteTablet for tablet 0bdcbaa0169c4ab7ba6ec038b50482fe with delete_type TABLET_DATA_DELETED (replaced by tablet 1bcb9d5f010f4297bab78c0183889238) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:11.920143 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 5bc3b9771a0042fd87e706204dcbb712 with delete_type TABLET_DATA_DELETED (replaced by tablet 961ade36f7d14eba91bb16d5eae2e254) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:11.920534 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 7c398634d6304a12b35c0579fba4c813 (DEFAULT_TABLE table=default.Table_14af7742726b4562a02fd2c9b2895277 [id=6d72901e5102476890f466fd406a7127]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.920924 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c398634d6304a12b35c0579fba4c813. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.921068 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 4c543ac4df33429b9787c6d32bf1ebeb
I20250903 20:05:11.921921 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b325ef7aa1304c2a941f9d6b44c50fe6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:11.922572 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b325ef7aa1304c2a941f9d6b44c50fe6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:11.911449 27064 catalog_manager.cc:6652] ReplaceTablet: tablet 5bc3b9771a0042fd87e706204dcbb712 deleted and replaced by tablet 961ade36f7d14eba91bb16d5eae2e254
I20250903 20:05:11.924409 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_a2b477ce2adf4ea0bd1b8685c6483f66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:11.925024 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2b477ce2adf4ea0bd1b8685c6483f66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:11.926554 27224 tablet_service.cc:1468] Processing CreateTablet for tablet 6f0cba70240f4f38a4fa5ccbcd99b3b1 (DEFAULT_TABLE table=default.Table_6ce254828dc045d39e5463fe6a48fa82 [id=7fb235b85bba4f7ab1b77bc5d2b9b58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.926911 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f0cba70240f4f38a4fa5ccbcd99b3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.927624 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 885fff47fea14b84ac6b34a57b827b4b (DEFAULT_TABLE table=default.Table_957f51b8fb684329a6e6fac205da5164 [id=61a25739b03042f7bd5bb693b68656d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.926518 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 264127511a234e9fbf409f3aa2860ccc
I20250903 20:05:11.928009 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 885fff47fea14b84ac6b34a57b827b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.928457 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_61bb115ae97549f7ab9c44e52b75acf0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:11.929121 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61bb115ae97549f7ab9c44e52b75acf0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:11.927690 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_48f65d1e5e0b4836bb4df770bd94b4e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:11.930569 27221 tablet_service.cc:1468] Processing CreateTablet for tablet 8d2d64f103fc44d7b390559b92cbf4b4 (DEFAULT_TABLE table=default.Table_1e6e2159113540b7944963d2140ab460 [id=f0171ee670294ce58ab84483fd71ab7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.930935 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d2d64f103fc44d7b390559b92cbf4b4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:11.930915 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_48f65d1e5e0b4836bb4df770bd94b4e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:11.935091 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 4bd2650e4d9e40f5ba5c09e367918224 (table default.Table_2ace21da933742e69b8ec85acbd919c2 [id=dcc845689a884dc69c7217043de59d32]) successfully deleted
W20250903 20:05:11.935361 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 0bdcbaa0169c4ab7ba6ec038b50482fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bdcbaa0169c4ab7ba6ec038b50482fe already in progress: creating tablet
W20250903 20:05:11.935566 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 5bc3b9771a0042fd87e706204dcbb712 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bc3b9771a0042fd87e706204dcbb712 already in progress: creating tablet
I20250903 20:05:11.937981 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c543ac4df33429b9787c6d32bf1ebeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:11.940624 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 4c543ac4df33429b9787c6d32bf1ebeb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c543ac4df33429b9787c6d32bf1ebeb already in progress: creating tablet
I20250903 20:05:11.941123 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet 264127511a234e9fbf409f3aa2860ccc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:11.941596 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 46b5d491888147bbb101963641fd05f8
W20250903 20:05:11.942497 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 264127511a234e9fbf409f3aa2860ccc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 264127511a234e9fbf409f3aa2860ccc already in progress: creating tablet
I20250903 20:05:11.942615 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6ce254828dc045d39e5463fe6a48fa82" } new_table_name: "default.Table_5ca5478ef07f4acdbbedb24af75f9e04" modify_external_catalogs: true
I20250903 20:05:11.943053 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet 46b5d491888147bbb101963641fd05f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:11.943737 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 46b5d491888147bbb101963641fd05f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46b5d491888147bbb101963641fd05f8 already in progress: creating tablet
I20250903 20:05:11.950662 27505 tablet_service.cc:1515] Processing DeleteTablet for tablet a10b079e7d5a440580ac00195fe88c65 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b3bfc63aa0b4e56bec189e834dc6a8d) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.950904 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c543ac4df33429b9787c6d32bf1ebeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.951025 27498 tablet_service.cc:1515] Processing DeleteTablet for tablet 73f65e7515994b898eec2ea23903aff4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.951251 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet 264127511a234e9fbf409f3aa2860ccc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.951335 27496 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f4ef613b0ab4bca98294682b934f6ed with delete_type TABLET_DATA_DELETED (replaced by tablet 264127511a234e9fbf409f3aa2860ccc) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.951593 27498 tablet_service.cc:1515] Processing DeleteTablet for tablet 46b5d491888147bbb101963641fd05f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.951812 27505 tablet_service.cc:1515] Processing DeleteTablet for tablet e722212b281a405fb08908df0a42d864 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.952498 27499 tablet_service.cc:1468] Processing CreateTablet for tablet 6f0cba70240f4f38a4fa5ccbcd99b3b1 (DEFAULT_TABLE table=default.Table_6ce254828dc045d39e5463fe6a48fa82 [id=7fb235b85bba4f7ab1b77bc5d2b9b58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.953032 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f0cba70240f4f38a4fa5ccbcd99b3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.953343 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet 5bc3b9771a0042fd87e706204dcbb712 with delete_type TABLET_DATA_DELETED (replaced by tablet 961ade36f7d14eba91bb16d5eae2e254) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.953624 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 0bdcbaa0169c4ab7ba6ec038b50482fe with delete_type TABLET_DATA_DELETED (replaced by tablet 1bcb9d5f010f4297bab78c0183889238) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.959190 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa05f54ebc324b40855fa3288760e9e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.960099 27355 tablet_service.cc:1515] Processing DeleteTablet for tablet 0bdcbaa0169c4ab7ba6ec038b50482fe with delete_type TABLET_DATA_DELETED (replaced by tablet 1bcb9d5f010f4297bab78c0183889238) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:11.960227 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9367b3a22f624c4087848099de32c080" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:11.961189 27183 leader_election.cc:343] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:11.961963 27221 tablet_service.cc:1515] Processing DeleteTablet for tablet 73f65e7515994b898eec2ea23903aff4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:11.961995 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 3f78a951bb024cf290116b0fe438ddb4
W20250903 20:05:11.962792 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 73f65e7515994b898eec2ea23903aff4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73f65e7515994b898eec2ea23903aff4 already in progress: creating tablet
I20250903 20:05:11.963601 27217 tablet_service.cc:1468] Processing CreateTablet for tablet cd92e3dafca6419f8eba7a4ef51a911e (DEFAULT_TABLE table=default.Table_13dbf08231794aa085d17ca500361b14 [id=b15b2b0164104454a23d766feea4e0de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:11.964046 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd92e3dafca6419f8eba7a4ef51a911e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.965600 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d2d64f103fc44d7b390559b92cbf4b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.965894 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac9b5d4f39db4f05a15316585040d142. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.966401 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 885fff47fea14b84ac6b34a57b827b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.968568 27221 tablet_service.cc:1515] Processing DeleteTablet for tablet 3f78a951bb024cf290116b0fe438ddb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:11.969406 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 3f78a951bb024cf290116b0fe438ddb4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f78a951bb024cf290116b0fe438ddb4 already in progress: creating tablet
I20250903 20:05:11.969765 27224 tablet_service.cc:1515] Processing DeleteTablet for tablet a10b079e7d5a440580ac00195fe88c65 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b3bfc63aa0b4e56bec189e834dc6a8d) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:11.970465 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8605bc329627438998812ed028327f53. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:11.970465 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet a10b079e7d5a440580ac00195fe88c65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a10b079e7d5a440580ac00195fe88c65 already in progress: creating tablet
I20250903 20:05:11.975455 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f0cba70240f4f38a4fa5ccbcd99b3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.975961 27693 tablet_bootstrap.cc:492] T 59e120e5e39e4de8a239f13545e4b1af P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:11.976200 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd92e3dafca6419f8eba7a4ef51a911e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:11.976197 27501 tablet_service.cc:1515] Processing DeleteTablet for tablet 3f78a951bb024cf290116b0fe438ddb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.964299 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet a10b079e7d5a440580ac00195fe88c65 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b3bfc63aa0b4e56bec189e834dc6a8d) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:11.976194 27693 ts_tablet_manager.cc:1397] T 59e120e5e39e4de8a239f13545e4b1af P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.099s user 0.004s sys 0.023s
I20250903 20:05:11.977190 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 04ccb3e4a8bd4789b9f19edd4a5b6145
I20250903 20:05:11.977814 27062 catalog_manager.cc:6652] ReplaceTablet: tablet 04ccb3e4a8bd4789b9f19edd4a5b6145 deleted and replaced by tablet a4c582b441f6467c99ba0b090abcf4cc
I20250903 20:05:11.978334 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_5b44669e91a8427c8e7d96b866fdb3b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:11.979260 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b44669e91a8427c8e7d96b866fdb3b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:11.980428 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 264127511a234e9fbf409f3aa2860ccc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:11.980667 27354 tablet_service.cc:1515] Processing DeleteTablet for tablet 5bc3b9771a0042fd87e706204dcbb712 with delete_type TABLET_DATA_DELETED (replaced by tablet 961ade36f7d14eba91bb16d5eae2e254) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:11.983511 27501 tablet_service.cc:1515] Processing DeleteTablet for tablet 04ccb3e4a8bd4789b9f19edd4a5b6145 with delete_type TABLET_DATA_DELETED (replaced by tablet a4c582b441f6467c99ba0b090abcf4cc) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:11.983843 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_7fb8db1fe4064eb2be0b9bba8b5ec73b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:11.984208 27066 catalog_manager.cc:5582] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f reported cstate change: term changed from 0 to 1, leader changed from <none> to e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67). New cstate: current_term: 1 leader_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:11.984884 27066 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 99a21a3e49934a79a66700c0b0d86307 (table default.Table_7325977ff92e45bab0b59b79579ecae4 [id=428657b402d748c0a9e3942d13342b4e]). Expected version 1 got 0
I20250903 20:05:11.985682 27350 tablet_service.cc:1515] Processing DeleteTablet for tablet 3f78a951bb024cf290116b0fe438ddb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:11.986477 27692 tablet_bootstrap.cc:492] T 7ba18e3f8cb148d484010cb4e72510b2 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:11.986650 27692 ts_tablet_manager.cc:1397] T 7ba18e3f8cb148d484010cb4e72510b2 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.134s user 0.001s sys 0.003s
I20250903 20:05:11.986809 27063 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_81f1c8f3063b4be686402371c42189fa [id=d0a75545b91c48bba47fb581bb0d1a96] alter complete (version 2)
I20250903 20:05:11.987390 27692 raft_consensus.cc:357] T 7ba18e3f8cb148d484010cb4e72510b2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.987568 27692 raft_consensus.cc:383] T 7ba18e3f8cb148d484010cb4e72510b2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:11.987624 27692 raft_consensus.cc:738] T 7ba18e3f8cb148d484010cb4e72510b2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:11.987332 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a5f8f31c12b742b597c8b35d662a607f" } new_table_name: "default.Table_1e3f794ae5ec4ad6a9b4351b4bda7b15" modify_external_catalogs: true
I20250903 20:05:11.987844 27692 consensus_queue.cc:260] T 7ba18e3f8cb148d484010cb4e72510b2 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:11.988288 27692 ts_tablet_manager.cc:1428] T 7ba18e3f8cb148d484010cb4e72510b2 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:11.988638 27692 tablet_bootstrap.cc:492] T b5bdc27275cd4156afe767a1fb33a8bd P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:11.989116 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 04ccb3e4a8bd4789b9f19edd4a5b6145 with delete_type TABLET_DATA_DELETED (replaced by tablet a4c582b441f6467c99ba0b090abcf4cc) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:11.984478 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fb8db1fe4064eb2be0b9bba8b5ec73b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:11.990224 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 04ccb3e4a8bd4789b9f19edd4a5b6145 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04ccb3e4a8bd4789b9f19edd4a5b6145 already in progress: creating tablet
I20250903 20:05:11.990396 27692 tablet_bootstrap.cc:654] T b5bdc27275cd4156afe767a1fb33a8bd P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:11.993445 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_81f1c8f3063b4be686402371c42189fa" } new_table_name: "default.Table_d4f82c96b77d430caad8a6b1bdec1a2f" modify_external_catalogs: true
I20250903 20:05:11.995056 27355 tablet_service.cc:1515] Processing DeleteTablet for tablet 73f65e7515994b898eec2ea23903aff4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:11.996541 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9367b3a22f624c4087848099de32c080" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:11.996783 27354 tablet_service.cc:1515] Processing DeleteTablet for tablet 04ccb3e4a8bd4789b9f19edd4a5b6145 with delete_type TABLET_DATA_DELETED (replaced by tablet a4c582b441f6467c99ba0b090abcf4cc) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:11.997879 27184 leader_election.cc:343] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:11.998052 27184 leader_election.cc:304] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:11.998593 27840 raft_consensus.cc:2747] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:11.997356 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet b5bdc27275cd4156afe767a1fb33a8bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:12.000389 27685 tablet_replica.cc:331] T b5bdc27275cd4156afe767a1fb33a8bd P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:12.000437 27351 tablet_service.cc:1515] Processing DeleteTablet for tablet 46b5d491888147bbb101963641fd05f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:12.000615 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb4d99139516405bb65c7480ddf259f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.000618 27685 raft_consensus.cc:2241] T b5bdc27275cd4156afe767a1fb33a8bd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:12.000855 27685 raft_consensus.cc:2270] T b5bdc27275cd4156afe767a1fb33a8bd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:12.001505 27352 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c543ac4df33429b9787c6d32bf1ebeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:12.001597 27685 ts_tablet_manager.cc:1905] T b5bdc27275cd4156afe767a1fb33a8bd P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:12.001849 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c398634d6304a12b35c0579fba4c813. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.005821 27693 raft_consensus.cc:357] T 59e120e5e39e4de8a239f13545e4b1af P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.007663 27693 raft_consensus.cc:383] T 59e120e5e39e4de8a239f13545e4b1af P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:12.007790 27693 raft_consensus.cc:738] T 59e120e5e39e4de8a239f13545e4b1af P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:12.008489 27693 consensus_queue.cc:260] T 59e120e5e39e4de8a239f13545e4b1af P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.012336 27693 ts_tablet_manager.cc:1428] T 59e120e5e39e4de8a239f13545e4b1af P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.032s user 0.001s sys 0.001s
I20250903 20:05:12.010226 27685 ts_tablet_manager.cc:1918] T b5bdc27275cd4156afe767a1fb33a8bd P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:12.012918 27685 log.cc:1199] T b5bdc27275cd4156afe767a1fb33a8bd P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/b5bdc27275cd4156afe767a1fb33a8bd
I20250903 20:05:12.014241 27685 ts_tablet_manager.cc:1939] T b5bdc27275cd4156afe767a1fb33a8bd P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
W20250903 20:05:12.015075 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6f0cba70240f4f38a4fa5ccbcd99b3b1 consensus state has no leader
I20250903 20:05:12.016160 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet b5bdc27275cd4156afe767a1fb33a8bd (table default.Table_c557d48ec717410fb860dfdc7ce9957b [id=31f68383b3734a8691dfe0317fc60345]) successfully deleted
I20250903 20:05:12.018535 27693 tablet_bootstrap.cc:492] T 0f4ef613b0ab4bca98294682b934f6ed P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:12.019134 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b0a0a2dfe31e4961871f334372541d05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:12.019778 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0a0a2dfe31e4961871f334372541d05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:12.021200 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c79b97a3bb294caf96bb5fa0d144628d" } new_table_name: "default.Table_cdb258f01fd94ff48ddd75c368e68d6c" modify_external_catalogs: true
I20250903 20:05:12.021718 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_0cd65e00bd5a40bd9dc9db12fd0bda4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:12.022190 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0cd65e00bd5a40bd9dc9db12fd0bda4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:12.023422 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_594fcf5c34de4f7287d7b7f605271d6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:12.023877 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_594fcf5c34de4f7287d7b7f605271d6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:12.027094 27692 tablet_bootstrap.cc:492] T b5bdc27275cd4156afe767a1fb33a8bd P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:12.027307 27692 ts_tablet_manager.cc:1397] T b5bdc27275cd4156afe767a1fb33a8bd P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.039s user 0.001s sys 0.007s
I20250903 20:05:12.027673 27684 ts_tablet_manager.cc:1918] T a45604ad611e418e84dab793e04101a0 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:12.027848 27684 log.cc:1199] T a45604ad611e418e84dab793e04101a0 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/a45604ad611e418e84dab793e04101a0
I20250903 20:05:12.028090 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_682ea2b65fb14ff3b2ce593e15cacba9" } new_table_name: "default.Table_9b80dba44fe8402999adc810b230d87a" modify_external_catalogs: true
I20250903 20:05:12.028251 27684 ts_tablet_manager.cc:1939] T a45604ad611e418e84dab793e04101a0 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:12.028525 27692 raft_consensus.cc:357] T b5bdc27275cd4156afe767a1fb33a8bd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.028766 27692 raft_consensus.cc:383] T b5bdc27275cd4156afe767a1fb33a8bd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:12.028875 27692 raft_consensus.cc:738] T b5bdc27275cd4156afe767a1fb33a8bd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:12.029186 27692 consensus_queue.cc:260] T b5bdc27275cd4156afe767a1fb33a8bd P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.030105 27692 ts_tablet_manager.cc:1428] T b5bdc27275cd4156afe767a1fb33a8bd P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250903 20:05:12.030521 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet a45604ad611e418e84dab793e04101a0 (table default.Table_7d8b7496fbc04f35bab6d6da573d4868 [id=5383eb1038e6480893e6eab6c7f7dd39]) successfully deleted
W20250903 20:05:12.030961 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet a10b079e7d5a440580ac00195fe88c65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a10b079e7d5a440580ac00195fe88c65 already in progress: creating tablet
I20250903 20:05:12.038012 27692 tablet_bootstrap.cc:492] T 4bd2650e4d9e40f5ba5c09e367918224 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:12.040230 27800 raft_consensus.cc:491] T be278233c7c34061988b267a44ee674f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:12.041481 27800 raft_consensus.cc:513] T be278233c7c34061988b267a44ee674f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.042255 27800 leader_election.cc:290] T be278233c7c34061988b267a44ee674f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:12.043788 27462 leader_election.cc:343] T be278233c7c34061988b267a44ee674f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: be278233c7c34061988b267a44ee674f
I20250903 20:05:12.044633 27692 tablet_bootstrap.cc:654] T 4bd2650e4d9e40f5ba5c09e367918224 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:12.044842 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be278233c7c34061988b267a44ee674f" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:12.045320 27800 raft_consensus.cc:491] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:12.045487 27800 raft_consensus.cc:513] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:12.045718 27462 leader_election.cc:343] T be278233c7c34061988b267a44ee674f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:12.045832 27462 leader_election.cc:304] T be278233c7c34061988b267a44ee674f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:12.046108 27800 leader_election.cc:290] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:12.046268 27800 raft_consensus.cc:2747] T be278233c7c34061988b267a44ee674f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:12.047063 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fb76ba1d7cd54d6aa69d06a9a5618cd3" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:12.047124 27462 leader_election.cc:343] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: fb76ba1d7cd54d6aa69d06a9a5618cd3
W20250903 20:05:12.047935 27462 leader_election.cc:343] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:12.048064 27462 leader_election.cc:304] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:12.048488 27800 raft_consensus.cc:2747] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:12.051087 27684 tablet_replica.cc:331] T 0f4ef613b0ab4bca98294682b934f6ed P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:12.052285 27684 raft_consensus.cc:2241] T 0f4ef613b0ab4bca98294682b934f6ed P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:12.053020 27684 raft_consensus.cc:2270] T 0f4ef613b0ab4bca98294682b934f6ed P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:12.057185 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 68cff0e416a54a5eb37071f5a12f07ab consensus state has no leader
I20250903 20:05:12.065443 27844 tablet.cc:1722] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250903 20:05:12.067152 27693 tablet_bootstrap.cc:654] T 0f4ef613b0ab4bca98294682b934f6ed P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:12.069543 27800 raft_consensus.cc:491] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:12.070703 27800 raft_consensus.cc:513] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.071033 27822 tablet.cc:1722] T b6b4713936134681aaef1732d46c6ab2 P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250903 20:05:12.071353 27800 leader_election.cc:290] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:12.072113 27684 ts_tablet_manager.cc:1905] T 0f4ef613b0ab4bca98294682b934f6ed P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:12.082247 27692 tablet_bootstrap.cc:492] T 4bd2650e4d9e40f5ba5c09e367918224 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:12.082626 27692 ts_tablet_manager.cc:1397] T 4bd2650e4d9e40f5ba5c09e367918224 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.045s user 0.006s sys 0.000s
I20250903 20:05:12.083679 27692 raft_consensus.cc:357] T 4bd2650e4d9e40f5ba5c09e367918224 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.083976 27692 raft_consensus.cc:383] T 4bd2650e4d9e40f5ba5c09e367918224 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:12.084105 27692 raft_consensus.cc:738] T 4bd2650e4d9e40f5ba5c09e367918224 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:12.084288 27823 tablet.cc:1722] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250903 20:05:12.084560 27692 consensus_queue.cc:260] T 4bd2650e4d9e40f5ba5c09e367918224 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.085355 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e722212b281a405fb08908df0a42d864" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:12.085575 27386 raft_consensus.cc:2466] T e722212b281a405fb08908df0a42d864 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
W20250903 20:05:12.085558 27462 leader_election.cc:343] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: e722212b281a405fb08908df0a42d864
I20250903 20:05:12.089756 27800 raft_consensus.cc:491] T 4bd2650e4d9e40f5ba5c09e367918224 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:12.089895 27800 raft_consensus.cc:513] T 4bd2650e4d9e40f5ba5c09e367918224 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.090639 27800 leader_election.cc:290] T 4bd2650e4d9e40f5ba5c09e367918224 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:12.091216 27462 leader_election.cc:304] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
W20250903 20:05:12.092658 27462 leader_election.cc:343] T 4bd2650e4d9e40f5ba5c09e367918224 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 4bd2650e4d9e40f5ba5c09e367918224
I20250903 20:05:12.092947 27692 ts_tablet_manager.cc:1428] T 4bd2650e4d9e40f5ba5c09e367918224 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250903 20:05:12.093340 27800 raft_consensus.cc:2802] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:12.093465 27800 raft_consensus.cc:491] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:12.093407 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4bd2650e4d9e40f5ba5c09e367918224" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:12.093562 27800 raft_consensus.cc:3058] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:12.096216 27692 tablet_bootstrap.cc:492] T 1396d4f8a93b412ca8397368fb98834e P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:12.097837 27800 raft_consensus.cc:513] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:12.102782 27051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.125:44059, user_credentials={real_user=slave}} blocked reactor thread for 55669.7us
I20250903 20:05:12.105624 27687 ts_tablet_manager.cc:1918] T 12a7be47c9c2406596c9e00800074a7c P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:12.107926 27695 tablet_bootstrap.cc:492] T 4692dd93b76c42c4883c7e22c6032250 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:12.108127 27695 ts_tablet_manager.cc:1397] T 4692dd93b76c42c4883c7e22c6032250 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.243s user 0.005s sys 0.000s
I20250903 20:05:12.109052 27695 raft_consensus.cc:357] T 4692dd93b76c42c4883c7e22c6032250 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.109306 27695 raft_consensus.cc:383] T 4692dd93b76c42c4883c7e22c6032250 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:12.109404 27695 raft_consensus.cc:738] T 4692dd93b76c42c4883c7e22c6032250 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:12.109716 27695 consensus_queue.cc:260] T 4692dd93b76c42c4883c7e22c6032250 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.109766 27800 leader_election.cc:290] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:12.099444 27462 leader_election.cc:343] T 4bd2650e4d9e40f5ba5c09e367918224 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:12.110041 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet e1a49d0599654552bf1c72857031699b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:12.110082 27462 leader_election.cc:304] T 4bd2650e4d9e40f5ba5c09e367918224 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:12.110821 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet be278233c7c34061988b267a44ee674f with delete_type TABLET_DATA_DELETED (replaced by tablet 5bc3b9771a0042fd87e706204dcbb712) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:12.111680 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet f3a33e90dc1f4540aee457de96f5b95b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:12.112389 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet fb76ba1d7cd54d6aa69d06a9a5618cd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:12.113677 27695 ts_tablet_manager.cc:1428] T 4692dd93b76c42c4883c7e22c6032250 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250903 20:05:12.102514 27692 tablet_bootstrap.cc:654] T 1396d4f8a93b412ca8397368fb98834e P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:12.115412 27462 leader_election.cc:343] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: e722212b281a405fb08908df0a42d864
I20250903 20:05:12.115986 27800 raft_consensus.cc:2747] T 4bd2650e4d9e40f5ba5c09e367918224 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:12.117756 27695 tablet_bootstrap.cc:492] T 29581a9dba46479884a0e6e01b7923de P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
W20250903 20:05:12.120551 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 4c543ac4df33429b9787c6d32bf1ebeb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c543ac4df33429b9787c6d32bf1ebeb already in progress: creating tablet
W20250903 20:05:12.120868 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 73f65e7515994b898eec2ea23903aff4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73f65e7515994b898eec2ea23903aff4 already in progress: creating tablet
W20250903 20:05:12.121104 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 264127511a234e9fbf409f3aa2860ccc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 264127511a234e9fbf409f3aa2860ccc already in progress: creating tablet
I20250903 20:05:12.111812 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e722212b281a405fb08908df0a42d864" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:12.123704 27687 log.cc:1199] T 12a7be47c9c2406596c9e00800074a7c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/12a7be47c9c2406596c9e00800074a7c
I20250903 20:05:12.123737 27386 raft_consensus.cc:3058] T e722212b281a405fb08908df0a42d864 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:12.126655 27386 raft_consensus.cc:2466] T e722212b281a405fb08908df0a42d864 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:12.127326 27462 leader_election.cc:304] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:12.117291 27844 tablet.cc:1722] T c8a293c6fa034e9fb29682439cb3bcc2 P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250903 20:05:12.128881 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4268316c28e8471a9f28807d20dbb395 consensus state has no leader
I20250903 20:05:12.130766 27064 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet c8a293c6fa034e9fb29682439cb3bcc2 (table default.Table_37e1d397cf99443d87211c3a8e1b9b82 [id=5cfc8827e4ca4f4a94424c48d6f71eb5]). Expected version 2 got 0
I20250903 20:05:12.131683 27060 master_service.cc:948] ReplaceTablet: received request to replace tablet 262a3549867a473aaaa7dfcc0b5872cd from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:12.132565 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet e722212b281a405fb08908df0a42d864 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:12.133056 27800 raft_consensus.cc:2802] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:12.133239 27800 raft_consensus.cc:695] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:12.133795 27800 consensus_queue.cc:237] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.139496 27695 tablet_bootstrap.cc:654] T 29581a9dba46479884a0e6e01b7923de P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:12.140442 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_8e482428f5d8407795dcd3111aefc596" } modify_external_catalogs: true
I20250903 20:05:12.140748 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_8e482428f5d8407795dcd3111aefc596" } modify_external_catalogs: true
I20250903 20:05:12.141822 27064 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_65e771add6404b41bd60f0bec5093c9b" } modify_external_catalogs: true
I20250903 20:05:12.142006 27064 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_65e771add6404b41bd60f0bec5093c9b" } modify_external_catalogs: true
I20250903 20:05:12.147085 27684 ts_tablet_manager.cc:1918] T 0f4ef613b0ab4bca98294682b934f6ed P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:12.148492 27684 log.cc:1199] T 0f4ef613b0ab4bca98294682b934f6ed P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/0f4ef613b0ab4bca98294682b934f6ed
I20250903 20:05:12.149190 27684 ts_tablet_manager.cc:1939] T 0f4ef613b0ab4bca98294682b934f6ed P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:12.151345 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 0f4ef613b0ab4bca98294682b934f6ed (table default.Table_1ade5bba3a18432c808ee7c4b910d247 [id=d20347a8196e44459448223a3af2dead]) successfully deleted
I20250903 20:05:12.151444 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_54bf0e4b392545b08001021008e93985" } modify_external_catalogs: true
I20250903 20:05:12.151604 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_54bf0e4b392545b08001021008e93985" } modify_external_catalogs: true
I20250903 20:05:12.152388 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_452fdb47678d4520828b9a3db6ddb237" } modify_external_catalogs: true
I20250903 20:05:12.152556 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_452fdb47678d4520828b9a3db6ddb237" } modify_external_catalogs: true
I20250903 20:05:12.148341 27692 tablet_bootstrap.cc:492] T 1396d4f8a93b412ca8397368fb98834e P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:12.153046 27692 ts_tablet_manager.cc:1397] T 1396d4f8a93b412ca8397368fb98834e P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.057s user 0.006s sys 0.000s
I20250903 20:05:12.153892 27692 raft_consensus.cc:357] T 1396d4f8a93b412ca8397368fb98834e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.155436 27692 raft_consensus.cc:383] T 1396d4f8a93b412ca8397368fb98834e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:12.155575 27692 raft_consensus.cc:738] T 1396d4f8a93b412ca8397368fb98834e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
W20250903 20:05:12.158452 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e5136d5388bc4c2ba71e09be40e8f412 consensus state has no leader
I20250903 20:05:12.158756 27066 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_d4f82c96b77d430caad8a6b1bdec1a2f [id=d0a75545b91c48bba47fb581bb0d1a96] alter complete (version 3)
I20250903 20:05:12.162744 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet b5bdc27275cd4156afe767a1fb33a8bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:12.163419 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet 4bd2650e4d9e40f5ba5c09e367918224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:12.163981 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ba18e3f8cb148d484010cb4e72510b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:12.165298 27499 tablet_service.cc:1468] Processing CreateTablet for tablet 45d9b6ff520348628d19671127b9426c (DEFAULT_TABLE table=default.Table_509afdab983d4eed97a5d124ff19ab3b [id=7a023e4098354d038d92f90d46fc5125]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.165746 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45d9b6ff520348628d19671127b9426c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.169586 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet 59e120e5e39e4de8a239f13545e4b1af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:12.170475 27364 tablet_service.cc:1468] Processing CreateTablet for tablet e78e6d212daf4589a6b4b65711e356cb (DEFAULT_TABLE table=default.Table_51d81329a9c84f5bb4a0973d5410d115 [id=6b3f6e948024464fa28fe985534395ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.170560 27361 tablet_service.cc:1468] Processing CreateTablet for tablet dbfdab4332234b2a93fd3c901a9cb645 (DEFAULT_TABLE table=default.Table_6f20b7d359084b4faef6f2eb3fdbab70 [id=67cd1b33fab14606babd6ba4524a6966]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.170538 27359 tablet_service.cc:1468] Processing CreateTablet for tablet 0ad64c699b294679b6fec37b5ad23528 (DEFAULT_TABLE table=default.Table_2f2c8693b96741448fbd96ff5eb9ad8b [id=0904dc5d0c594bc3bbf32de2508c6195]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.170840 27366 tablet_service.cc:1468] Processing CreateTablet for tablet 45d9b6ff520348628d19671127b9426c (DEFAULT_TABLE table=default.Table_509afdab983d4eed97a5d124ff19ab3b [id=7a023e4098354d038d92f90d46fc5125]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.170964 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e78e6d212daf4589a6b4b65711e356cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.171053 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 7cc6cf579e2642e99fefed723b0ee900 (DEFAULT_TABLE table=default.Table_bffb6801f48c4124bfe24c49586d6e48 [id=aa020b54da65482ba59a71768c2bd03b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.171180 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45d9b6ff520348628d19671127b9426c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.171415 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ad64c699b294679b6fec37b5ad23528. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.171700 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbfdab4332234b2a93fd3c901a9cb645. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.171732 27362 tablet_service.cc:1468] Processing CreateTablet for tablet c6d67cd0c0304c77ba571b414878dad2 (DEFAULT_TABLE table=default.Table_e0e02fcc9f93461182e556c4623dbe7c [id=5fe055478b724744a65fcaa2f25bef0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.172133 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6d67cd0c0304c77ba571b414878dad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.173328 27356 tablet_service.cc:1468] Processing CreateTablet for tablet cd6d6bb514134bdf8e3bc0aaad180732 (DEFAULT_TABLE table=default.Table_ef30e0f336384b50bf8293fe525252cf [id=b2f5b8646204428993ce29cf7b2e4caa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.173745 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd6d6bb514134bdf8e3bc0aaad180732. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.174885 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cc6cf579e2642e99fefed723b0ee900. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.177088 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 45d9b6ff520348628d19671127b9426c (DEFAULT_TABLE table=default.Table_509afdab983d4eed97a5d124ff19ab3b [id=7a023e4098354d038d92f90d46fc5125]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.177376 27224 tablet_service.cc:1468] Processing CreateTablet for tablet d859c0222c3647dab3dddd105e4cf0fc (DEFAULT_TABLE table=default.Table_e3d549d1397a4167a21010a1041c6ab2 [id=7add4b636f1b4bffae663fdf432de05f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.177577 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45d9b6ff520348628d19671127b9426c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.177889 27221 tablet_service.cc:1468] Processing CreateTablet for tablet e78e6d212daf4589a6b4b65711e356cb (DEFAULT_TABLE table=default.Table_51d81329a9c84f5bb4a0973d5410d115 [id=6b3f6e948024464fa28fe985534395ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.177090 27220 tablet_service.cc:1468] Processing CreateTablet for tablet c6d67cd0c0304c77ba571b414878dad2 (DEFAULT_TABLE table=default.Table_e0e02fcc9f93461182e556c4623dbe7c [id=5fe055478b724744a65fcaa2f25bef0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.178475 27213 tablet_service.cc:1468] Processing CreateTablet for tablet cd6d6bb514134bdf8e3bc0aaad180732 (DEFAULT_TABLE table=default.Table_ef30e0f336384b50bf8293fe525252cf [id=b2f5b8646204428993ce29cf7b2e4caa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.178958 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 0ad64c699b294679b6fec37b5ad23528 (DEFAULT_TABLE table=default.Table_2f2c8693b96741448fbd96ff5eb9ad8b [id=0904dc5d0c594bc3bbf32de2508c6195]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.179486 27223 tablet_service.cc:1468] Processing CreateTablet for tablet 4157633525034a019fd6a80185892cec (DEFAULT_TABLE table=default.Table_cc37561faad74a2093d52d313b9f1010 [id=4b42213db8534ac3b3b97318f5a38e61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.179980 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 4692dd93b76c42c4883c7e22c6032250 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:12.180011 27225 tablet_service.cc:1468] Processing CreateTablet for tablet dbfdab4332234b2a93fd3c901a9cb645 (DEFAULT_TABLE table=default.Table_6f20b7d359084b4faef6f2eb3fdbab70 [id=67cd1b33fab14606babd6ba4524a6966]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.180518 27226 tablet_service.cc:1468] Processing CreateTablet for tablet 283a3afed7414a958b2dc4176f00bb01 (DEFAULT_TABLE table=default.Table_19e44161daa5469da23656554f451dd0 [id=2ed5c84cc5a3466ba8a8cfeda62b643e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.180969 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 7cc6cf579e2642e99fefed723b0ee900 (DEFAULT_TABLE table=default.Table_bffb6801f48c4124bfe24c49586d6e48 [id=aa020b54da65482ba59a71768c2bd03b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.181430 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e78e6d212daf4589a6b4b65711e356cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.182451 27685 tablet_replica.cc:331] T 4692dd93b76c42c4883c7e22c6032250 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:12.182705 27685 raft_consensus.cc:2241] T 4692dd93b76c42c4883c7e22c6032250 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:12.182894 27685 raft_consensus.cc:2270] T 4692dd93b76c42c4883c7e22c6032250 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:12.183146 27684 tablet_replica.cc:331] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:12.183363 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6d67cd0c0304c77ba571b414878dad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.183372 27684 raft_consensus.cc:2241] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:12.183629 27684 pending_rounds.cc:70] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f: Trying to abort 1 pending ops.
I20250903 20:05:12.183709 27684 pending_rounds.cc:77] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384920104681472 op_type: NO_OP noop_request { }
I20250903 20:05:12.183813 27684 raft_consensus.cc:2887] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:12.183921 27684 raft_consensus.cc:2270] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:12.184110 27692 consensus_queue.cc:260] T 1396d4f8a93b412ca8397368fb98834e P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.185045 27692 ts_tablet_manager.cc:1428] T 1396d4f8a93b412ca8397368fb98834e P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
I20250903 20:05:12.186357 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd6d6bb514134bdf8e3bc0aaad180732. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.186431 27695 tablet_bootstrap.cc:492] T 29581a9dba46479884a0e6e01b7923de P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:12.186586 27695 ts_tablet_manager.cc:1397] T 29581a9dba46479884a0e6e01b7923de P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.069s user 0.005s sys 0.000s
I20250903 20:05:12.187518 27695 raft_consensus.cc:357] T 29581a9dba46479884a0e6e01b7923de P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.187726 27695 raft_consensus.cc:383] T 29581a9dba46479884a0e6e01b7923de P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:12.187840 27695 raft_consensus.cc:738] T 29581a9dba46479884a0e6e01b7923de P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:12.188124 27695 consensus_queue.cc:260] T 29581a9dba46479884a0e6e01b7923de P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.188266 27685 ts_tablet_manager.cc:1905] T 4692dd93b76c42c4883c7e22c6032250 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:12.190696 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 283a3afed7414a958b2dc4176f00bb01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.191051 27692 tablet_bootstrap.cc:492] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:12.192734 27692 tablet_bootstrap.cc:654] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:12.193691 27695 ts_tablet_manager.cc:1428] T 29581a9dba46479884a0e6e01b7923de P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250903 20:05:12.194653 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cc6cf579e2642e99fefed723b0ee900. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.195832 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbfdab4332234b2a93fd3c901a9cb645. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.196570 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 29581a9dba46479884a0e6e01b7923de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:12.196667 27695 tablet_bootstrap.cc:492] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:12.198165 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ad64c699b294679b6fec37b5ad23528. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.198688 27223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4157633525034a019fd6a80185892cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.202078 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d859c0222c3647dab3dddd105e4cf0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.209479 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 4157633525034a019fd6a80185892cec (DEFAULT_TABLE table=default.Table_cc37561faad74a2093d52d313b9f1010 [id=4b42213db8534ac3b3b97318f5a38e61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.210577 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4157633525034a019fd6a80185892cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.211630 27363 tablet_service.cc:1468] Processing CreateTablet for tablet 283a3afed7414a958b2dc4176f00bb01 (DEFAULT_TABLE table=default.Table_19e44161daa5469da23656554f451dd0 [id=2ed5c84cc5a3466ba8a8cfeda62b643e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.215222 27357 tablet_service.cc:1468] Processing CreateTablet for tablet d859c0222c3647dab3dddd105e4cf0fc (DEFAULT_TABLE table=default.Table_e3d549d1397a4167a21010a1041c6ab2 [id=7add4b636f1b4bffae663fdf432de05f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.215811 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 283a3afed7414a958b2dc4176f00bb01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.217232 27685 ts_tablet_manager.cc:1918] T 4692dd93b76c42c4883c7e22c6032250 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:12.226687 27499 tablet_service.cc:1468] Processing CreateTablet for tablet c6d67cd0c0304c77ba571b414878dad2 (DEFAULT_TABLE table=default.Table_e0e02fcc9f93461182e556c4623dbe7c [id=5fe055478b724744a65fcaa2f25bef0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.226886 27684 ts_tablet_manager.cc:1905] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:12.227063 27695 tablet_bootstrap.cc:654] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:12.227700 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6d67cd0c0304c77ba571b414878dad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.228744 27685 log.cc:1199] T 4692dd93b76c42c4883c7e22c6032250 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/4692dd93b76c42c4883c7e22c6032250
I20250903 20:05:12.230098 27501 tablet_service.cc:1468] Processing CreateTablet for tablet d859c0222c3647dab3dddd105e4cf0fc (DEFAULT_TABLE table=default.Table_e3d549d1397a4167a21010a1041c6ab2 [id=7add4b636f1b4bffae663fdf432de05f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.230541 27501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d859c0222c3647dab3dddd105e4cf0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.242591 27684 ts_tablet_manager.cc:1918] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:12.242832 27684 log.cc:1199] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/e722212b281a405fb08908df0a42d864
I20250903 20:05:12.243309 27684 ts_tablet_manager.cc:1939] T e722212b281a405fb08908df0a42d864 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:12.245914 27684 tablet_replica.cc:331] T e1a49d0599654552bf1c72857031699b P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:12.246237 27684 raft_consensus.cc:2241] T e1a49d0599654552bf1c72857031699b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:12.246492 27684 raft_consensus.cc:2270] T e1a49d0599654552bf1c72857031699b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:12.247571 27684 ts_tablet_manager.cc:1905] T e1a49d0599654552bf1c72857031699b P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:12.255682 27053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.126:39015, user_credentials={real_user=slave}} blocked reactor thread for 85678.4us
I20250903 20:05:12.256633 27684 ts_tablet_manager.cc:1918] T e1a49d0599654552bf1c72857031699b P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:12.258069 27685 ts_tablet_manager.cc:1939] T 4692dd93b76c42c4883c7e22c6032250 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:12.258858 27685 tablet_replica.cc:331] T 29581a9dba46479884a0e6e01b7923de P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:12.259057 27685 raft_consensus.cc:2241] T 29581a9dba46479884a0e6e01b7923de P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:12.259373 27685 raft_consensus.cc:2270] T 29581a9dba46479884a0e6e01b7923de P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:12.261348 27685 ts_tablet_manager.cc:1905] T 29581a9dba46479884a0e6e01b7923de P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:12.261667 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d859c0222c3647dab3dddd105e4cf0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.258822 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 4692dd93b76c42c4883c7e22c6032250 (table default.Table_7dfd9faa02434381b40a56369804413b [id=f11bdb54ed5c4b1ca1e1306d39aaf25f]) successfully deleted
I20250903 20:05:12.268378 27693 tablet_bootstrap.cc:492] T 0f4ef613b0ab4bca98294682b934f6ed P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:12.268584 27693 ts_tablet_manager.cc:1397] T 0f4ef613b0ab4bca98294682b934f6ed P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.255s user 0.004s sys 0.007s
I20250903 20:05:12.268772 27685 ts_tablet_manager.cc:1918] T 29581a9dba46479884a0e6e01b7923de P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:12.268921 27685 log.cc:1199] T 29581a9dba46479884a0e6e01b7923de P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/29581a9dba46479884a0e6e01b7923de
I20250903 20:05:12.269376 27685 ts_tablet_manager.cc:1939] T 29581a9dba46479884a0e6e01b7923de P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:12.269531 27693 raft_consensus.cc:357] T 0f4ef613b0ab4bca98294682b934f6ed P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.269739 27693 raft_consensus.cc:383] T 0f4ef613b0ab4bca98294682b934f6ed P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:12.269841 27693 raft_consensus.cc:738] T 0f4ef613b0ab4bca98294682b934f6ed P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:12.270119 27693 consensus_queue.cc:260] T 0f4ef613b0ab4bca98294682b934f6ed P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.270167 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 29581a9dba46479884a0e6e01b7923de (table default.Table_7e75298df3b54e3794ab39c8b0fbffc7 [id=cf5354e1f11f4a228ea25ec2baa569a4]) successfully deleted
I20250903 20:05:12.271319 27693 ts_tablet_manager.cc:1428] T 0f4ef613b0ab4bca98294682b934f6ed P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:12.271870 27693 tablet_bootstrap.cc:492] T a45604ad611e418e84dab793e04101a0 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:12.274240 27693 tablet_bootstrap.cc:654] T a45604ad611e418e84dab793e04101a0 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:12.275749 27687 ts_tablet_manager.cc:1939] T 12a7be47c9c2406596c9e00800074a7c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:12.278133 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 12a7be47c9c2406596c9e00800074a7c (table default.Table_97be735f442746fe92867a2df71838a0 [id=fdc7899c8b334d309b834ae1daa14fa0]) successfully deleted
I20250903 20:05:12.279093 27064 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 09cc12bc08b946e7a2ff6775d69f902d
I20250903 20:05:12.281131 27687 tablet_replica.cc:331] T 0062307c6ee54aada74a08cf0d087f17 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:12.281422 27687 raft_consensus.cc:2241] T 0062307c6ee54aada74a08cf0d087f17 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:12.281673 27687 raft_consensus.cc:2270] T 0062307c6ee54aada74a08cf0d087f17 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:12.282490 27687 ts_tablet_manager.cc:1905] T 0062307c6ee54aada74a08cf0d087f17 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:12.278957 27500 tablet_service.cc:1468] Processing CreateTablet for tablet cd6d6bb514134bdf8e3bc0aaad180732 (DEFAULT_TABLE table=default.Table_ef30e0f336384b50bf8293fe525252cf [id=b2f5b8646204428993ce29cf7b2e4caa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.284921 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd6d6bb514134bdf8e3bc0aaad180732. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:12.280704 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 46b5d491888147bbb101963641fd05f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46b5d491888147bbb101963641fd05f8 already in progress: creating tablet
I20250903 20:05:12.286304 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet e722212b281a405fb08908df0a42d864 (table default.Table_5f1e055d858e4d31bce98ac57741cc30 [id=7fd6a8bf57a846f78b1bf70fa3ea5175]) successfully deleted
W20250903 20:05:12.286592 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 5bc3b9771a0042fd87e706204dcbb712 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bc3b9771a0042fd87e706204dcbb712 already in progress: creating tablet
W20250903 20:05:12.286789 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 0bdcbaa0169c4ab7ba6ec038b50482fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bdcbaa0169c4ab7ba6ec038b50482fe already in progress: creating tablet
W20250903 20:05:12.286984 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 3f78a951bb024cf290116b0fe438ddb4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f78a951bb024cf290116b0fe438ddb4 already in progress: creating tablet
W20250903 20:05:12.287185 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 04ccb3e4a8bd4789b9f19edd4a5b6145 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04ccb3e4a8bd4789b9f19edd4a5b6145 already in progress: creating tablet
I20250903 20:05:12.279078 27800 raft_consensus.cc:491] T b5bdc27275cd4156afe767a1fb33a8bd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:12.279273 27507 tablet_service.cc:1468] Processing CreateTablet for tablet dbfdab4332234b2a93fd3c901a9cb645 (DEFAULT_TABLE table=default.Table_6f20b7d359084b4faef6f2eb3fdbab70 [id=67cd1b33fab14606babd6ba4524a6966]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.289496 27800 raft_consensus.cc:513] T b5bdc27275cd4156afe767a1fb33a8bd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.289789 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbfdab4332234b2a93fd3c901a9cb645. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.290921 27800 leader_election.cc:290] T b5bdc27275cd4156afe767a1fb33a8bd P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:12.279443 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 0ad64c699b294679b6fec37b5ad23528 (DEFAULT_TABLE table=default.Table_2f2c8693b96741448fbd96ff5eb9ad8b [id=0904dc5d0c594bc3bbf32de2508c6195]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.292721 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b5bdc27275cd4156afe767a1fb33a8bd" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:12.292932 27695 tablet_bootstrap.cc:492] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:12.279752 27502 tablet_service.cc:1468] Processing CreateTablet for tablet 7cc6cf579e2642e99fefed723b0ee900 (DEFAULT_TABLE table=default.Table_bffb6801f48c4124bfe24c49586d6e48 [id=aa020b54da65482ba59a71768c2bd03b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.293121 27695 ts_tablet_manager.cc:1397] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.097s user 0.009s sys 0.007s
I20250903 20:05:12.279888 27503 tablet_service.cc:1468] Processing CreateTablet for tablet 283a3afed7414a958b2dc4176f00bb01 (DEFAULT_TABLE table=default.Table_19e44161daa5469da23656554f451dd0 [id=2ed5c84cc5a3466ba8a8cfeda62b643e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.278954 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 4157633525034a019fd6a80185892cec (DEFAULT_TABLE table=default.Table_cc37561faad74a2093d52d313b9f1010 [id=4b42213db8534ac3b3b97318f5a38e61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.294008 27695 raft_consensus.cc:357] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.294293 27695 raft_consensus.cc:383] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:12.294389 27695 raft_consensus.cc:738] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
W20250903 20:05:12.294518 27462 leader_election.cc:343] T b5bdc27275cd4156afe767a1fb33a8bd P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: b5bdc27275cd4156afe767a1fb33a8bd
W20250903 20:05:12.294780 27462 leader_election.cc:343] T b5bdc27275cd4156afe767a1fb33a8bd P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:12.294706 27695 consensus_queue.cc:260] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.294848 27064 master_service.cc:948] ReplaceTablet: received request to replace tablet d8ed7b3e292241cb9525745a797ff13a from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:12.280323 27501 tablet_service.cc:1468] Processing CreateTablet for tablet e78e6d212daf4589a6b4b65711e356cb (DEFAULT_TABLE table=default.Table_51d81329a9c84f5bb4a0973d5410d115 [id=6b3f6e948024464fa28fe985534395ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.297256 27058 master_service.cc:948] ReplaceTablet: received request to replace tablet e20fcd28c3ae4355a033d2b8429f84e2 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:12.297922 27695 ts_tablet_manager.cc:1428] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20250903 20:05:12.298103 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_14af7742726b4562a02fd2c9b2895277" } modify_external_catalogs: true
I20250903 20:05:12.298327 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_14af7742726b4562a02fd2c9b2895277" } modify_external_catalogs: true
I20250903 20:05:12.300344 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_be001b7aec594b93a1b45e9313b772ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:12.300963 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be001b7aec594b93a1b45e9313b772ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:12.303234 27695 tablet_bootstrap.cc:492] T c0343d2226e345669d42375de2545986 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:12.304662 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_d342b3c06e1c4b98bc7f07e88844cdc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:12.305284 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d342b3c06e1c4b98bc7f07e88844cdc9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:12.305877 27695 tablet_bootstrap.cc:654] T c0343d2226e345669d42375de2545986 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:12.310400 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_58997b732a0f46e69409a76f193c699b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:12.311151 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58997b732a0f46e69409a76f193c699b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:12.314532 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_646dcf6e43a24b31aaa79d2f93c58a14" } new_table_name: "default.Table_3da0f03ae8114e348baf44301196b2e5" modify_external_catalogs: true
I20250903 20:05:12.314823 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_60d0c49d795c4cad954d2b7327485eb9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:12.315604 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60d0c49d795c4cad954d2b7327485eb9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:12.316294 27695 tablet_bootstrap.cc:492] T c0343d2226e345669d42375de2545986 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:12.316892 27695 ts_tablet_manager.cc:1397] T c0343d2226e345669d42375de2545986 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.006s
I20250903 20:05:12.317909 27695 raft_consensus.cc:357] T c0343d2226e345669d42375de2545986 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.318194 27695 raft_consensus.cc:383] T c0343d2226e345669d42375de2545986 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:12.318385 27695 raft_consensus.cc:738] T c0343d2226e345669d42375de2545986 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:12.318828 27695 consensus_queue.cc:260] T c0343d2226e345669d42375de2545986 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.322460 27695 ts_tablet_manager.cc:1428] T c0343d2226e345669d42375de2545986 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250903 20:05:12.325881 27695 tablet_bootstrap.cc:492] T 46b5d491888147bbb101963641fd05f8 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:12.328881 27687 ts_tablet_manager.cc:1918] T 0062307c6ee54aada74a08cf0d087f17 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:12.329187 27695 tablet_bootstrap.cc:654] T 46b5d491888147bbb101963641fd05f8 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:12.331697 27462 leader_election.cc:304] T b5bdc27275cd4156afe767a1fb33a8bd P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:12.332094 27800 raft_consensus.cc:2747] T b5bdc27275cd4156afe767a1fb33a8bd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:12.335098 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cc6cf579e2642e99fefed723b0ee900. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.336397 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 283a3afed7414a958b2dc4176f00bb01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.336907 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 44db427ee53a4d0a907dfd63282d6d4f
I20250903 20:05:12.337106 27501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e78e6d212daf4589a6b4b65711e356cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.339290 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_72b5f55c55bc481d85de016fa2276ffe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:12.339435 27505 tablet_service.cc:1515] Processing DeleteTablet for tablet e722212b281a405fb08908df0a42d864 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:12.339447 27684 log.cc:1199] T e1a49d0599654552bf1c72857031699b P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/e1a49d0599654552bf1c72857031699b
I20250903 20:05:12.340463 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 4be9bdc390de4a74a396003c9c3c2b71
I20250903 20:05:12.341185 27063 master_service.cc:948] ReplaceTablet: received request to replace tablet 9509364ba3ec4e9bad09db09406a978a from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:12.342762 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet ecd339ec12e84e88acec427630cc8668
I20250903 20:05:12.343457 27057 master_service.cc:948] ReplaceTablet: received request to replace tablet 925c0ab4873041da82702cd9377ec7bc from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:12.344139 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_923868d510f34889be990c9cd4180adc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:12.344834 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_923868d510f34889be990c9cd4180adc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:12.346926 27684 ts_tablet_manager.cc:1939] T e1a49d0599654552bf1c72857031699b P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
W20250903 20:05:12.339998 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_72b5f55c55bc481d85de016fa2276ffe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:12.352012 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet 4be9bdc390de4a74a396003c9c3c2b71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:12.350831 27224 tablet_service.cc:1515] Processing DeleteTablet for tablet 44db427ee53a4d0a907dfd63282d6d4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:12.352680 27223 tablet_service.cc:1515] Processing DeleteTablet for tablet ecd339ec12e84e88acec427630cc8668 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:12.350750 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ad64c699b294679b6fec37b5ad23528. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:12.354252 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 4be9bdc390de4a74a396003c9c3c2b71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4be9bdc390de4a74a396003c9c3c2b71 already in progress: creating tablet
W20250903 20:05:12.354550 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 44db427ee53a4d0a907dfd63282d6d4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44db427ee53a4d0a907dfd63282d6d4f already in progress: creating tablet
W20250903 20:05:12.354790 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet ecd339ec12e84e88acec427630cc8668 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecd339ec12e84e88acec427630cc8668 already in progress: creating tablet
I20250903 20:05:12.355221 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4157633525034a019fd6a80185892cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.360983 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 44db427ee53a4d0a907dfd63282d6d4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:12.361991 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 4be9bdc390de4a74a396003c9c3c2b71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:12.364293 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet ecd339ec12e84e88acec427630cc8668 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:12.378233 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 7c398634d6304a12b35c0579fba4c813
I20250903 20:05:12.379623 27684 tablet_replica.cc:331] T be278233c7c34061988b267a44ee674f P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:12.379828 27684 raft_consensus.cc:2241] T be278233c7c34061988b267a44ee674f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:12.380066 27684 raft_consensus.cc:2270] T be278233c7c34061988b267a44ee674f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:12.380031 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_10864de211fa4474949fecdfbe4c1e9e" } new_table_name: "default.Table_118af17580524e4d86fc930e787e92c5" modify_external_catalogs: true
I20250903 20:05:12.380621 27684 ts_tablet_manager.cc:1905] T be278233c7c34061988b267a44ee674f P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:12.381752 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_c21ce98311d847f2aaae2ea2c0549856"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:12.382359 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c21ce98311d847f2aaae2ea2c0549856 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:12.383911 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e0e02fcc9f93461182e556c4623dbe7c" } new_table_name: "default.Table_a540eea45dfb46ec92b02a3f9a2b3aed" modify_external_catalogs: true
W20250903 20:05:12.385241 27059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 639ecb9caa4a423899ab8eb35bb716d7 consensus state has no leader
I20250903 20:05:12.385615 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_1480d4baa8e149bb80b62f09d4cf3f90"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:12.386138 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1480d4baa8e149bb80b62f09d4cf3f90 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:12.387604 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a1ac671d35424891be575ac9621ce309" } new_table_name: "default.Table_5c08431ef8e04a6aab9b96ab78615d2f" modify_external_catalogs: true
I20250903 20:05:12.390636 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 9509364ba3ec4e9bad09db09406a978a
I20250903 20:05:12.391038 27063 catalog_manager.cc:6652] ReplaceTablet: tablet 9509364ba3ec4e9bad09db09406a978a deleted and replaced by tablet 1fbe54dc40dd47daa109ceeb07546b0a
I20250903 20:05:12.391487 27684 ts_tablet_manager.cc:1918] T be278233c7c34061988b267a44ee674f P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:12.391417 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e5def563dd414101b9ee8fbaf7156e29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:12.391609 27684 log.cc:1199] T be278233c7c34061988b267a44ee674f P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/be278233c7c34061988b267a44ee674f
I20250903 20:05:12.391990 27684 ts_tablet_manager.cc:1939] T be278233c7c34061988b267a44ee674f P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
W20250903 20:05:12.392041 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e5def563dd414101b9ee8fbaf7156e29 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:12.393868 27692 tablet_bootstrap.cc:492] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:12.394039 27692 ts_tablet_manager.cc:1397] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.203s user 0.033s sys 0.013s
I20250903 20:05:12.394948 27692 raft_consensus.cc:357] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.395193 27692 raft_consensus.cc:383] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:12.395274 27692 raft_consensus.cc:738] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:12.395588 27692 consensus_queue.cc:260] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.400520 27692 ts_tablet_manager.cc:1428] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20250903 20:05:12.402619 27057 master_service.cc:948] ReplaceTablet: received request to replace tablet 885fff47fea14b84ac6b34a57b827b4b from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:12.403465 27223 tablet_service.cc:1515] Processing DeleteTablet for tablet 7c398634d6304a12b35c0579fba4c813 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:12.403771 27224 tablet_service.cc:1515] Processing DeleteTablet for tablet 9509364ba3ec4e9bad09db09406a978a with delete_type TABLET_DATA_DELETED (replaced by tablet 1fbe54dc40dd47daa109ceeb07546b0a) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:12.404873 27692 tablet_bootstrap.cc:492] T 29581a9dba46479884a0e6e01b7923de P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
W20250903 20:05:12.404983 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 7c398634d6304a12b35c0579fba4c813 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c398634d6304a12b35c0579fba4c813 already in progress: creating tablet
W20250903 20:05:12.405295 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 9509364ba3ec4e9bad09db09406a978a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9509364ba3ec4e9bad09db09406a978a already in progress: creating tablet
I20250903 20:05:12.406167 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_781d39d6b06f49c28fa93c3692c7b5f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:12.406412 27684 tablet_replica.cc:331] T f3a33e90dc1f4540aee457de96f5b95b P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:12.406627 27684 raft_consensus.cc:2241] T f3a33e90dc1f4540aee457de96f5b95b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:12.406807 27684 raft_consensus.cc:2270] T f3a33e90dc1f4540aee457de96f5b95b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:12.407397 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_781d39d6b06f49c28fa93c3692c7b5f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:12.407428 27684 ts_tablet_manager.cc:1905] T f3a33e90dc1f4540aee457de96f5b95b P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:12.408149 27692 tablet_bootstrap.cc:654] T 29581a9dba46479884a0e6e01b7923de P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:12.415234 27695 tablet_bootstrap.cc:492] T 46b5d491888147bbb101963641fd05f8 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:12.415558 27695 ts_tablet_manager.cc:1397] T 46b5d491888147bbb101963641fd05f8 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.090s user 0.005s sys 0.002s
I20250903 20:05:12.416530 27695 raft_consensus.cc:357] T 46b5d491888147bbb101963641fd05f8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.416905 27695 raft_consensus.cc:383] T 46b5d491888147bbb101963641fd05f8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:12.417047 27695 raft_consensus.cc:738] T 46b5d491888147bbb101963641fd05f8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:12.417439 27695 consensus_queue.cc:260] T 46b5d491888147bbb101963641fd05f8 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:12.417699 27051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.125:44059, user_credentials={real_user=slave}} blocked reactor thread for 45105.6us
I20250903 20:05:12.421397 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 7c398634d6304a12b35c0579fba4c813 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:12.330734 27693 tablet_bootstrap.cc:492] T a45604ad611e418e84dab793e04101a0 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:12.348719 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet 4be9bdc390de4a74a396003c9c3c2b71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:12.349862 27363 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f4ef613b0ab4bca98294682b934f6ed with delete_type TABLET_DATA_DELETED (replaced by tablet 264127511a234e9fbf409f3aa2860ccc) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:12.352311 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet ecd339ec12e84e88acec427630cc8668 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:12.356710 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet 44db427ee53a4d0a907dfd63282d6d4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:12.390183 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet 7c398634d6304a12b35c0579fba4c813 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:12.393304 27754 raft_consensus.cc:491] T 0f4ef613b0ab4bca98294682b934f6ed P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:12.422189 27754 raft_consensus.cc:513] T 0f4ef613b0ab4bca98294682b934f6ed P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.422622 27695 ts_tablet_manager.cc:1428] T 46b5d491888147bbb101963641fd05f8 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250903 20:05:12.422978 27754 leader_election.cc:290] T 0f4ef613b0ab4bca98294682b934f6ed P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:12.394354 27366 tablet_service.cc:1515] Processing DeleteTablet for tablet 9509364ba3ec4e9bad09db09406a978a with delete_type TABLET_DATA_DELETED (replaced by tablet 1fbe54dc40dd47daa109ceeb07546b0a) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:12.424273 27320 leader_election.cc:343] T 0f4ef613b0ab4bca98294682b934f6ed P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 0f4ef613b0ab4bca98294682b934f6ed
I20250903 20:05:12.424721 27693 ts_tablet_manager.cc:1397] T a45604ad611e418e84dab793e04101a0 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.153s user 0.004s sys 0.000s
W20250903 20:05:12.425236 27321 leader_election.cc:343] T 0f4ef613b0ab4bca98294682b934f6ed P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 0f4ef613b0ab4bca98294682b934f6ed
I20250903 20:05:12.425374 27321 leader_election.cc:304] T 0f4ef613b0ab4bca98294682b934f6ed P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:12.425772 27754 raft_consensus.cc:2747] T 0f4ef613b0ab4bca98294682b934f6ed P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:12.426551 27693 raft_consensus.cc:357] T a45604ad611e418e84dab793e04101a0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.426764 27693 raft_consensus.cc:383] T a45604ad611e418e84dab793e04101a0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:12.426853 27693 raft_consensus.cc:738] T a45604ad611e418e84dab793e04101a0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:12.427071 27695 tablet_bootstrap.cc:492] T d8ed7b3e292241cb9525745a797ff13a P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:12.427248 27693 consensus_queue.cc:260] T a45604ad611e418e84dab793e04101a0 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.428471 27693 ts_tablet_manager.cc:1428] T a45604ad611e418e84dab793e04101a0 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:12.429008 27693 tablet_bootstrap.cc:492] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:12.429154 27695 tablet_bootstrap.cc:654] T d8ed7b3e292241cb9525745a797ff13a P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:12.431268 27693 tablet_bootstrap.cc:654] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:12.431332 27692 tablet_bootstrap.cc:492] T 29581a9dba46479884a0e6e01b7923de P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:12.431497 27692 ts_tablet_manager.cc:1397] T 29581a9dba46479884a0e6e01b7923de P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20250903 20:05:12.432390 27692 raft_consensus.cc:357] T 29581a9dba46479884a0e6e01b7923de P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.432615 27692 raft_consensus.cc:383] T 29581a9dba46479884a0e6e01b7923de P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:12.432722 27692 raft_consensus.cc:738] T 29581a9dba46479884a0e6e01b7923de P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:12.433053 27692 consensus_queue.cc:260] T 29581a9dba46479884a0e6e01b7923de P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.434864 27687 log.cc:1199] T 0062307c6ee54aada74a08cf0d087f17 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/0062307c6ee54aada74a08cf0d087f17
I20250903 20:05:12.436221 27687 ts_tablet_manager.cc:1939] T 0062307c6ee54aada74a08cf0d087f17 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:12.437816 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 9509364ba3ec4e9bad09db09406a978a with delete_type TABLET_DATA_DELETED (replaced by tablet 1fbe54dc40dd47daa109ceeb07546b0a) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:12.438707 27692 ts_tablet_manager.cc:1428] T 29581a9dba46479884a0e6e01b7923de P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250903 20:05:12.444707 27692 tablet_bootstrap.cc:492] T 4692dd93b76c42c4883c7e22c6032250 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:12.446681 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet e1a49d0599654552bf1c72857031699b (table default.Table_f9e41274ca534e46b7906480d7fc32fe [id=febabbd421d241948dd40f5480c052bb]) successfully deleted
I20250903 20:05:12.447252 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet be278233c7c34061988b267a44ee674f (table default.Table_459903503b5148d09d3cb8f2f5519fab [id=3c69612aaf164dc899dcc252d85636f6]) successfully deleted
I20250903 20:05:12.448474 27930 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.65.126:39015 every 8 connection(s)
I20250903 20:05:12.450282 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 0062307c6ee54aada74a08cf0d087f17 (table default.Table_6d1b2694cd1a4963ac658f25026e9ef5 [id=f4bc244b05e840b9843b4d448d4f28aa]) successfully deleted
I20250903 20:05:12.450322 27684 ts_tablet_manager.cc:1918] T f3a33e90dc1f4540aee457de96f5b95b P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:12.450474 27684 log.cc:1199] T f3a33e90dc1f4540aee457de96f5b95b P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/f3a33e90dc1f4540aee457de96f5b95b
I20250903 20:05:12.451128 27684 ts_tablet_manager.cc:1939] T f3a33e90dc1f4540aee457de96f5b95b P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
W20250903 20:05:12.451129 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 0bdcbaa0169c4ab7ba6ec038b50482fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bdcbaa0169c4ab7ba6ec038b50482fe already in progress: creating tablet
W20250903 20:05:12.451815 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet a10b079e7d5a440580ac00195fe88c65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a10b079e7d5a440580ac00195fe88c65 already in progress: creating tablet
I20250903 20:05:12.452405 27695 tablet_bootstrap.cc:492] T d8ed7b3e292241cb9525745a797ff13a P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:12.452653 27695 ts_tablet_manager.cc:1397] T d8ed7b3e292241cb9525745a797ff13a P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20250903 20:05:12.453807 27695 raft_consensus.cc:357] T d8ed7b3e292241cb9525745a797ff13a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.454102 27695 raft_consensus.cc:383] T d8ed7b3e292241cb9525745a797ff13a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:12.454278 27695 raft_consensus.cc:738] T d8ed7b3e292241cb9525745a797ff13a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:12.454682 27695 consensus_queue.cc:260] T d8ed7b3e292241cb9525745a797ff13a P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:12.454875 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet de8ac9f984be46ec8f021928de12a090 consensus state has no leader
I20250903 20:05:12.448499 27717 rpc_server.cc:307] RPC server started. Bound to: 127.26.65.126:39015
W20250903 20:05:12.456813 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c6d67cd0c0304c77ba571b414878dad2 consensus state has no leader
I20250903 20:05:12.458096 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b325ef7aa1304c2a941f9d6b44c50fe6" } new_table_name: "default.Table_7a6b693ddf594ff6821359a4a7fbee01" modify_external_catalogs: true
I20250903 20:05:12.458241 27695 ts_tablet_manager.cc:1428] T d8ed7b3e292241cb9525745a797ff13a P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250903 20:05:12.450747 27692 tablet_bootstrap.cc:654] T 4692dd93b76c42c4883c7e22c6032250 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:12.463449 27695 tablet_bootstrap.cc:492] T 76742d0df44c43729197f7e09f96cad8 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:12.465340 27695 tablet_bootstrap.cc:654] T 76742d0df44c43729197f7e09f96cad8 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:12.466246 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 264127511a234e9fbf409f3aa2860ccc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 264127511a234e9fbf409f3aa2860ccc already in progress: creating tablet
I20250903 20:05:12.470481 27684 tablet_replica.cc:331] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:12.470781 27684 raft_consensus.cc:2241] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:12.471062 27684 raft_consensus.cc:2270] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:12.471264 27717 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-0/data/info.pb
I20250903 20:05:12.471693 27684 ts_tablet_manager.cc:1905] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:12.476435 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet f3a33e90dc1f4540aee457de96f5b95b (table default.Table_f6203681100e4faeb016f0e6afc283f7 [id=91268ab201a4423faedd970b9a296152]) successfully deleted
W20250903 20:05:12.477149 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 5bc3b9771a0042fd87e706204dcbb712 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bc3b9771a0042fd87e706204dcbb712 already in progress: creating tablet
W20250903 20:05:12.477927 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 73f65e7515994b898eec2ea23903aff4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73f65e7515994b898eec2ea23903aff4 already in progress: creating tablet
I20250903 20:05:12.479022 26885 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskH7eH49/build/asan/bin/kudu as pid 27717
W20250903 20:05:12.479578 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 3f78a951bb024cf290116b0fe438ddb4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f78a951bb024cf290116b0fe438ddb4 already in progress: creating tablet
W20250903 20:05:12.480069 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 04ccb3e4a8bd4789b9f19edd4a5b6145 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04ccb3e4a8bd4789b9f19edd4a5b6145 already in progress: creating tablet
W20250903 20:05:12.482200 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2a5c74a281b44019872a6442d41bc373 consensus state has no leader
I20250903 20:05:12.486840 27692 tablet_bootstrap.cc:492] T 4692dd93b76c42c4883c7e22c6032250 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:12.487044 27692 ts_tablet_manager.cc:1397] T 4692dd93b76c42c4883c7e22c6032250 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.042s user 0.001s sys 0.002s
I20250903 20:05:12.487882 27692 raft_consensus.cc:357] T 4692dd93b76c42c4883c7e22c6032250 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.488111 27692 raft_consensus.cc:383] T 4692dd93b76c42c4883c7e22c6032250 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:12.488211 27692 raft_consensus.cc:738] T 4692dd93b76c42c4883c7e22c6032250 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:12.488483 27692 consensus_queue.cc:260] T 4692dd93b76c42c4883c7e22c6032250 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.489221 27692 ts_tablet_manager.cc:1428] T 4692dd93b76c42c4883c7e22c6032250 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250903 20:05:12.490448 27692 tablet_bootstrap.cc:492] T c0343d2226e345669d42375de2545986 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:12.492233 27692 tablet_bootstrap.cc:654] T c0343d2226e345669d42375de2545986 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:12.500380 27366 tablet_service.cc:1515] Processing DeleteTablet for tablet a45604ad611e418e84dab793e04101a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:12.502393 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 46b5d491888147bbb101963641fd05f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46b5d491888147bbb101963641fd05f8 already in progress: creating tablet
I20250903 20:05:12.503453 27695 tablet_bootstrap.cc:492] T 76742d0df44c43729197f7e09f96cad8 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:12.503674 27695 ts_tablet_manager.cc:1397] T 76742d0df44c43729197f7e09f96cad8 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.040s user 0.001s sys 0.002s
I20250903 20:05:12.504607 27693 tablet_bootstrap.cc:492] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:12.504655 27695 raft_consensus.cc:357] T 76742d0df44c43729197f7e09f96cad8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.504863 27695 raft_consensus.cc:383] T 76742d0df44c43729197f7e09f96cad8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:12.504902 27693 ts_tablet_manager.cc:1397] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.076s user 0.006s sys 0.000s
I20250903 20:05:12.504949 27695 raft_consensus.cc:738] T 76742d0df44c43729197f7e09f96cad8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:12.505252 27695 consensus_queue.cc:260] T 76742d0df44c43729197f7e09f96cad8 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.505914 27693 raft_consensus.cc:357] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.506155 27693 raft_consensus.cc:383] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:12.506317 27693 raft_consensus.cc:738] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:12.507043 27695 ts_tablet_manager.cc:1428] T 76742d0df44c43729197f7e09f96cad8 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250903 20:05:12.507375 27695 tablet_bootstrap.cc:492] T 46e3a65118b744298b5536f88c22f1db P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:12.506642 27693 consensus_queue.cc:260] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.509196 27693 ts_tablet_manager.cc:1428] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250903 20:05:12.510388 27695 tablet_bootstrap.cc:654] T 46e3a65118b744298b5536f88c22f1db P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:12.511864 27937 sys_catalog.cc:263] Verifying existing consensus state
I20250903 20:05:12.514254 27693 tablet_bootstrap.cc:492] T eaa11d17303142ff935705c16f0b9690 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:12.514928 27692 tablet_bootstrap.cc:492] T c0343d2226e345669d42375de2545986 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:12.515112 27692 ts_tablet_manager.cc:1397] T c0343d2226e345669d42375de2545986 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
W20250903 20:05:12.516481 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 4c543ac4df33429b9787c6d32bf1ebeb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c543ac4df33429b9787c6d32bf1ebeb already in progress: creating tablet
W20250903 20:05:12.516798 27051 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 639ecb9caa4a423899ab8eb35bb716d7 (table default.Table_3da0f03ae8114e348baf44301196b2e5 [id=ad8b3a9bedef4c188b89640ab5e2473a], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 639ecb9caa4a423899ab8eb35bb716d7 consensus state has no leader
I20250903 20:05:12.517982 27687 tablet_replica.cc:331] T e722212b281a405fb08908df0a42d864 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:12.519213 27687 raft_consensus.cc:2241] T e722212b281a405fb08908df0a42d864 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:12.519469 27687 raft_consensus.cc:2270] T e722212b281a405fb08908df0a42d864 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:12.519896 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b0b8582c6c054baab4688a30cfe5f767" } modify_external_catalogs: true
I20250903 20:05:12.520123 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b0b8582c6c054baab4688a30cfe5f767" } modify_external_catalogs: true
I20250903 20:05:12.522186 27692 raft_consensus.cc:357] T c0343d2226e345669d42375de2545986 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.523613 27692 raft_consensus.cc:383] T c0343d2226e345669d42375de2545986 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:12.523782 27692 raft_consensus.cc:738] T c0343d2226e345669d42375de2545986 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:12.524080 27692 consensus_queue.cc:260] T c0343d2226e345669d42375de2545986 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.496438 27684 ts_tablet_manager.cc:1918] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:12.524449 27684 log.cc:1199] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/fb76ba1d7cd54d6aa69d06a9a5618cd3
I20250903 20:05:12.525487 27684 ts_tablet_manager.cc:1939] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:12.525010 27692 ts_tablet_manager.cc:1428] T c0343d2226e345669d42375de2545986 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20250903 20:05:12.526820 27692 tablet_bootstrap.cc:492] T 46b5d491888147bbb101963641fd05f8 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:12.527871 27684 tablet_replica.cc:331] T b5bdc27275cd4156afe767a1fb33a8bd P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:12.529410 27684 raft_consensus.cc:2241] T b5bdc27275cd4156afe767a1fb33a8bd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:12.529623 27684 raft_consensus.cc:2270] T b5bdc27275cd4156afe767a1fb33a8bd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:12.529187 27692 tablet_bootstrap.cc:654] T 46b5d491888147bbb101963641fd05f8 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:12.530359 27684 ts_tablet_manager.cc:1905] T b5bdc27275cd4156afe767a1fb33a8bd P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:12.532526 27573 heartbeater.cc:344] Connected to a master server at 127.26.65.126:39015
I20250903 20:05:12.532732 27573 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
I20250903 20:05:12.534281 27295 heartbeater.cc:344] Connected to a master server at 127.26.65.126:39015
I20250903 20:05:12.534507 27295 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
I20250903 20:05:12.535025 27695 tablet_bootstrap.cc:492] T 46e3a65118b744298b5536f88c22f1db P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:12.535192 27695 ts_tablet_manager.cc:1397] T 46e3a65118b744298b5536f88c22f1db P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.003s
I20250903 20:05:12.536182 27695 raft_consensus.cc:357] T 46e3a65118b744298b5536f88c22f1db P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.536377 27695 raft_consensus.cc:383] T 46e3a65118b744298b5536f88c22f1db P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:12.536494 27695 raft_consensus.cc:738] T 46e3a65118b744298b5536f88c22f1db P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:12.537002 27695 consensus_queue.cc:260] T 46e3a65118b744298b5536f88c22f1db P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.541675 27684 ts_tablet_manager.cc:1918] T b5bdc27275cd4156afe767a1fb33a8bd P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:12.542834 27684 log.cc:1199] T b5bdc27275cd4156afe767a1fb33a8bd P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/b5bdc27275cd4156afe767a1fb33a8bd
I20250903 20:05:12.543447 27684 ts_tablet_manager.cc:1939] T b5bdc27275cd4156afe767a1fb33a8bd P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
W20250903 20:05:12.545677 27462 consensus_peers.cc:489] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:12.546089 27684 tablet_replica.cc:331] T 4bd2650e4d9e40f5ba5c09e367918224 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:12.546561 27684 raft_consensus.cc:2241] T 4bd2650e4d9e40f5ba5c09e367918224 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:12.547458 27684 raft_consensus.cc:2270] T 4bd2650e4d9e40f5ba5c09e367918224 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:12.548610 27684 ts_tablet_manager.cc:1905] T 4bd2650e4d9e40f5ba5c09e367918224 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:12.555433 27224 tablet_service.cc:1468] Processing CreateTablet for tablet f2cc35f4244a42eeb4a65de749824c11 (DEFAULT_TABLE table=default.Table_594fcf5c34de4f7287d7b7f605271d6c [id=fb43441ca8f2415d99a628e81ef1642f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.556092 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2cc35f4244a42eeb4a65de749824c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.562079 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet d8ed7b3e292241cb9525745a797ff13a with delete_type TABLET_DATA_DELETED (replaced by tablet 639ecb9caa4a423899ab8eb35bb716d7) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:12.562352 27223 tablet_service.cc:1515] Processing DeleteTablet for tablet 76742d0df44c43729197f7e09f96cad8 with delete_type TABLET_DATA_DELETED (replaced by tablet 0940159eaea740f99be2b67d5abacd2d) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:12.562562 27685 tablet_replica.cc:331] T d8ed7b3e292241cb9525745a797ff13a P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:12.562814 27685 raft_consensus.cc:2241] T d8ed7b3e292241cb9525745a797ff13a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:12.563017 27685 raft_consensus.cc:2270] T d8ed7b3e292241cb9525745a797ff13a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:12.562572 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet 46b5d491888147bbb101963641fd05f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:12.564260 27685 ts_tablet_manager.cc:1905] T d8ed7b3e292241cb9525745a797ff13a P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:12.564726 27840 raft_consensus.cc:491] T 46b5d491888147bbb101963641fd05f8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:12.564877 27840 raft_consensus.cc:513] T 46b5d491888147bbb101963641fd05f8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.565572 27840 leader_election.cc:290] T 46b5d491888147bbb101963641fd05f8 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:12.567625 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46b5d491888147bbb101963641fd05f8" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:12.568706 27183 leader_election.cc:343] T 46b5d491888147bbb101963641fd05f8 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:12.571573 27685 ts_tablet_manager.cc:1918] T d8ed7b3e292241cb9525745a797ff13a P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:12.571842 27685 log.cc:1199] T d8ed7b3e292241cb9525745a797ff13a P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/d8ed7b3e292241cb9525745a797ff13a
I20250903 20:05:12.572269 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 9685cba2f73b412382a69dab2c7f50c1 (DEFAULT_TABLE table=default.Table_0cd65e00bd5a40bd9dc9db12fd0bda4d [id=15369700921549be8faa450d2ca98949]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.572666 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9685cba2f73b412382a69dab2c7f50c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.573637 27684 ts_tablet_manager.cc:1918] T 4bd2650e4d9e40f5ba5c09e367918224 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:12.573657 27223 tablet_service.cc:1468] Processing CreateTablet for tablet 514e382479014fa7bf1f404efff11c86 (DEFAULT_TABLE table=default.Table_b0a0a2dfe31e4961871f334372541d05 [id=30918e7a5f1d43aab5eaddef7c485cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.573776 27684 log.cc:1199] T 4bd2650e4d9e40f5ba5c09e367918224 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/4bd2650e4d9e40f5ba5c09e367918224
I20250903 20:05:12.574098 27223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 514e382479014fa7bf1f404efff11c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.576416 27684 ts_tablet_manager.cc:1939] T 4bd2650e4d9e40f5ba5c09e367918224 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:12.579066 27684 tablet_replica.cc:331] T 7ba18e3f8cb148d484010cb4e72510b2 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:12.579638 27684 raft_consensus.cc:2241] T 7ba18e3f8cb148d484010cb4e72510b2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:12.580205 27684 raft_consensus.cc:2270] T 7ba18e3f8cb148d484010cb4e72510b2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:12.581140 27684 ts_tablet_manager.cc:1905] T 7ba18e3f8cb148d484010cb4e72510b2 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:12.581234 27218 tablet_service.cc:1468] Processing CreateTablet for tablet a0293a69f1c246d08d414e95729b4ab4 (DEFAULT_TABLE table=default.Table_61bb115ae97549f7ab9c44e52b75acf0 [id=4ce7195efb6e432e9b232b8a4749179f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.581746 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0293a69f1c246d08d414e95729b4ab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.580384 27573 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
I20250903 20:05:12.572360 27685 ts_tablet_manager.cc:1939] T d8ed7b3e292241cb9525745a797ff13a P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:12.585204 27695 ts_tablet_manager.cc:1428] T 46e3a65118b744298b5536f88c22f1db P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.050s user 0.000s sys 0.002s
I20250903 20:05:12.585579 27695 tablet_bootstrap.cc:492] T 9509364ba3ec4e9bad09db09406a978a P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:12.588160 27937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2: Bootstrap starting.
I20250903 20:05:12.588673 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet d8ed7b3e292241cb9525745a797ff13a (table default.Table_3da0f03ae8114e348baf44301196b2e5 [id=ad8b3a9bedef4c188b89640ab5e2473a]) successfully deleted
I20250903 20:05:12.589596 27685 tablet_replica.cc:331] T 76742d0df44c43729197f7e09f96cad8 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:12.589884 27685 raft_consensus.cc:2241] T 76742d0df44c43729197f7e09f96cad8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:12.590126 27685 raft_consensus.cc:2270] T 76742d0df44c43729197f7e09f96cad8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:12.590852 27573 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
I20250903 20:05:12.591842 27695 tablet_bootstrap.cc:654] T 9509364ba3ec4e9bad09db09406a978a P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:12.592789 27685 ts_tablet_manager.cc:1905] T 76742d0df44c43729197f7e09f96cad8 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:12.597169 27295 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
I20250903 20:05:12.606184 27224 tablet_service.cc:1468] Processing CreateTablet for tablet 6788b1683f984949816edfc5b68de8db (DEFAULT_TABLE table=default.Table_48f65d1e5e0b4836bb4df770bd94b4e0 [id=45702f222451484c80915421a0cdeb68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.606782 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6788b1683f984949816edfc5b68de8db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.608630 27214 tablet_service.cc:1468] Processing CreateTablet for tablet c9e90abef9814338b70a3550ab5686e0 (DEFAULT_TABLE table=default.Table_5b44669e91a8427c8e7d96b866fdb3b5 [id=96167b0d9f9b47168a9fc59635b7136d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.609028 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9e90abef9814338b70a3550ab5686e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.612612 27684 ts_tablet_manager.cc:1918] T 7ba18e3f8cb148d484010cb4e72510b2 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:12.613682 27226 tablet_service.cc:1468] Processing CreateTablet for tablet 1bcb9d5f010f4297bab78c0183889238 (DEFAULT_TABLE table=default.Table_05d58d894486468a8a15aaa4b4c0cabb [id=e69321e60a3a4e71a463a0011c0c6d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.617774 27684 log.cc:1199] T 7ba18e3f8cb148d484010cb4e72510b2 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/7ba18e3f8cb148d484010cb4e72510b2
I20250903 20:05:12.618312 27684 ts_tablet_manager.cc:1939] T 7ba18e3f8cb148d484010cb4e72510b2 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:12.524389 27432 heartbeater.cc:344] Connected to a master server at 127.26.65.126:39015
I20250903 20:05:12.557788 27366 tablet_service.cc:1468] Processing CreateTablet for tablet f2cc35f4244a42eeb4a65de749824c11 (DEFAULT_TABLE table=default.Table_594fcf5c34de4f7287d7b7f605271d6c [id=fb43441ca8f2415d99a628e81ef1642f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.618824 27432 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
I20250903 20:05:12.619016 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2cc35f4244a42eeb4a65de749824c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.619818 27937 log.cc:826] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2: Log is configured to *not* fsync() on all Append() calls
I20250903 20:05:12.566855 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46b5d491888147bbb101963641fd05f8" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:12.571590 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 9685cba2f73b412382a69dab2c7f50c1 (DEFAULT_TABLE table=default.Table_0cd65e00bd5a40bd9dc9db12fd0bda4d [id=15369700921549be8faa450d2ca98949]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.626974 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9685cba2f73b412382a69dab2c7f50c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.579684 27364 tablet_service.cc:1468] Processing CreateTablet for tablet 514e382479014fa7bf1f404efff11c86 (DEFAULT_TABLE table=default.Table_b0a0a2dfe31e4961871f334372541d05 [id=30918e7a5f1d43aab5eaddef7c485cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.630944 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 514e382479014fa7bf1f404efff11c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.632769 27432 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
I20250903 20:05:12.604318 27359 tablet_service.cc:1468] Processing CreateTablet for tablet a0293a69f1c246d08d414e95729b4ab4 (DEFAULT_TABLE table=default.Table_61bb115ae97549f7ab9c44e52b75acf0 [id=4ce7195efb6e432e9b232b8a4749179f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.638779 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0293a69f1c246d08d414e95729b4ab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.606812 27363 tablet_service.cc:1468] Processing CreateTablet for tablet 6788b1683f984949816edfc5b68de8db (DEFAULT_TABLE table=default.Table_48f65d1e5e0b4836bb4df770bd94b4e0 [id=45702f222451484c80915421a0cdeb68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.609405 27357 tablet_service.cc:1468] Processing CreateTablet for tablet c9e90abef9814338b70a3550ab5686e0 (DEFAULT_TABLE table=default.Table_5b44669e91a8427c8e7d96b866fdb3b5 [id=96167b0d9f9b47168a9fc59635b7136d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.611707 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 1bcb9d5f010f4297bab78c0183889238 (DEFAULT_TABLE table=default.Table_05d58d894486468a8a15aaa4b4c0cabb [id=e69321e60a3a4e71a463a0011c0c6d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.645510 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6788b1683f984949816edfc5b68de8db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.646303 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bcb9d5f010f4297bab78c0183889238. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.646600 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9e90abef9814338b70a3550ab5686e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.665189 27754 raft_consensus.cc:491] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:12.665472 27754 raft_consensus.cc:513] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.665990 27687 ts_tablet_manager.cc:1905] T e722212b281a405fb08908df0a42d864 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:12.666347 27754 leader_election.cc:290] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:12.668735 27320 leader_election.cc:343] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: fb76ba1d7cd54d6aa69d06a9a5618cd3
I20250903 20:05:12.671456 27506 tablet_service.cc:1468] Processing CreateTablet for tablet f2cc35f4244a42eeb4a65de749824c11 (DEFAULT_TABLE table=default.Table_594fcf5c34de4f7287d7b7f605271d6c [id=fb43441ca8f2415d99a628e81ef1642f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.671927 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2cc35f4244a42eeb4a65de749824c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.671808 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_6b435f3c61214c74a632c94e955abcdf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:12.672452 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b435f3c61214c74a632c94e955abcdf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:12.673933 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet fb76ba1d7cd54d6aa69d06a9a5618cd3 (table default.Table_a249558ab60e466ca7734ffeb24ac5fd [id=d1ed6b104f9541fbaf34c6b03b297f0f]) successfully deleted
I20250903 20:05:12.674162 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet b5bdc27275cd4156afe767a1fb33a8bd (table default.Table_c557d48ec717410fb860dfdc7ce9957b [id=31f68383b3734a8691dfe0317fc60345]) successfully deleted
I20250903 20:05:12.674350 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 4bd2650e4d9e40f5ba5c09e367918224 (table default.Table_2ace21da933742e69b8ec85acbd919c2 [id=dcc845689a884dc69c7217043de59d32]) successfully deleted
I20250903 20:05:12.674481 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_72b5f55c55bc481d85de016fa2276ffe" } modify_external_catalogs: true
I20250903 20:05:12.674561 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 7ba18e3f8cb148d484010cb4e72510b2 (table default.Table_9691d111178f4b9c9024a0a66fc3f03b [id=83d0f963c409436faa6c9a78bc0167cd]) successfully deleted
I20250903 20:05:12.674677 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_72b5f55c55bc481d85de016fa2276ffe" } modify_external_catalogs: true
I20250903 20:05:12.674631 27502 tablet_service.cc:1468] Processing CreateTablet for tablet 9685cba2f73b412382a69dab2c7f50c1 (DEFAULT_TABLE table=default.Table_0cd65e00bd5a40bd9dc9db12fd0bda4d [id=15369700921549be8faa450d2ca98949]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:12.674784 27051 catalog_manager.cc:4895] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet e722212b281a405fb08908df0a42d864 because the tablet was not found. No further retry: Not found: Tablet not found: e722212b281a405fb08908df0a42d864
W20250903 20:05:12.675022 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 44db427ee53a4d0a907dfd63282d6d4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44db427ee53a4d0a907dfd63282d6d4f already in progress: creating tablet
I20250903 20:05:12.675051 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9685cba2f73b412382a69dab2c7f50c1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:12.675212 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 4be9bdc390de4a74a396003c9c3c2b71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4be9bdc390de4a74a396003c9c3c2b71 already in progress: creating tablet
W20250903 20:05:12.675388 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet ecd339ec12e84e88acec427630cc8668 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecd339ec12e84e88acec427630cc8668 already in progress: creating tablet
W20250903 20:05:12.675549 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 7c398634d6304a12b35c0579fba4c813 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c398634d6304a12b35c0579fba4c813 already in progress: creating tablet
W20250903 20:05:12.675761 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 9509364ba3ec4e9bad09db09406a978a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9509364ba3ec4e9bad09db09406a978a already in progress: creating tablet
I20250903 20:05:12.675884 27062 master_service.cc:948] ReplaceTablet: received request to replace tablet cfa8a1d9011e460681443a6b77b95958 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:12.676373 27693 tablet_bootstrap.cc:700] T eaa11d17303142ff935705c16f0b9690 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent opening tablet: real 0.161s user 0.000s sys 0.001s
I20250903 20:05:12.676695 27693 tablet_bootstrap.cc:654] T eaa11d17303142ff935705c16f0b9690 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:12.677188 27059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_61bb115ae97549f7ab9c44e52b75acf0" } modify_external_catalogs: true
I20250903 20:05:12.677376 27059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_61bb115ae97549f7ab9c44e52b75acf0" } modify_external_catalogs: true
I20250903 20:05:12.678575 27065 master_service.cc:948] ReplaceTablet: received request to replace tablet 8945085bdc174e6db22a5a9477810297 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:12.678588 27503 tablet_service.cc:1468] Processing CreateTablet for tablet 514e382479014fa7bf1f404efff11c86 (DEFAULT_TABLE table=default.Table_b0a0a2dfe31e4961871f334372541d05 [id=30918e7a5f1d43aab5eaddef7c485cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.679102 27504 tablet_service.cc:1468] Processing CreateTablet for tablet a0293a69f1c246d08d414e95729b4ab4 (DEFAULT_TABLE table=default.Table_61bb115ae97549f7ab9c44e52b75acf0 [id=4ce7195efb6e432e9b232b8a4749179f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.679271 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7fb8db1fe4064eb2be0b9bba8b5ec73b" } new_table_name: "default.Table_94e9e32ef1134f07ade1cdf8da69ead6" modify_external_catalogs: true
I20250903 20:05:12.679625 27501 tablet_service.cc:1468] Processing CreateTablet for tablet 6788b1683f984949816edfc5b68de8db (DEFAULT_TABLE table=default.Table_48f65d1e5e0b4836bb4df770bd94b4e0 [id=45702f222451484c80915421a0cdeb68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.680043 27507 tablet_service.cc:1468] Processing CreateTablet for tablet c9e90abef9814338b70a3550ab5686e0 (DEFAULT_TABLE table=default.Table_5b44669e91a8427c8e7d96b866fdb3b5 [id=96167b0d9f9b47168a9fc59635b7136d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.680462 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 1bcb9d5f010f4297bab78c0183889238 (DEFAULT_TABLE table=default.Table_05d58d894486468a8a15aaa4b4c0cabb [id=e69321e60a3a4e71a463a0011c0c6d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.681799 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 514e382479014fa7bf1f404efff11c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.682433 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9e90abef9814338b70a3550ab5686e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.682967 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bcb9d5f010f4297bab78c0183889238. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.683418 27501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6788b1683f984949816edfc5b68de8db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.683910 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0293a69f1c246d08d414e95729b4ab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.687156 27687 ts_tablet_manager.cc:1918] T e722212b281a405fb08908df0a42d864 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:12.687371 27687 log.cc:1199] T e722212b281a405fb08908df0a42d864 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/e722212b281a405fb08908df0a42d864
I20250903 20:05:12.687760 27687 ts_tablet_manager.cc:1939] T e722212b281a405fb08908df0a42d864 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:12.688374 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b4634182d45b490eb24ae0950859ab2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:12.689078 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4634182d45b490eb24ae0950859ab2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:12.689258 27687 tablet_replica.cc:331] T 59e120e5e39e4de8a239f13545e4b1af P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:12.689551 27687 raft_consensus.cc:2241] T 59e120e5e39e4de8a239f13545e4b1af P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:12.689787 27687 raft_consensus.cc:2270] T 59e120e5e39e4de8a239f13545e4b1af P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:12.689901 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b520cd881f58424dadf3fa4a013812ee" } modify_external_catalogs: true
I20250903 20:05:12.690472 27687 ts_tablet_manager.cc:1905] T 59e120e5e39e4de8a239f13545e4b1af P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:12.690675 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet fb76ba1d7cd54d6aa69d06a9a5618cd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:09 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:12.692231 27364 tablet_service.cc:1468] Processing CreateTablet for tablet 3b3bfc63aa0b4e56bec189e834dc6a8d (DEFAULT_TABLE table=default.Table_56d2ca07338b4ce4ad732c5b2401812b [id=6d6f5e8cadf9445d8695c075cfabb2ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.692664 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet e722212b281a405fb08908df0a42d864 (table default.Table_5f1e055d858e4d31bce98ac57741cc30 [id=7fd6a8bf57a846f78b1bf70fa3ea5175]) successfully deleted
I20250903 20:05:12.692782 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b3bfc63aa0b4e56bec189e834dc6a8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.693748 27366 tablet_service.cc:1468] Processing CreateTablet for tablet 961ade36f7d14eba91bb16d5eae2e254 (DEFAULT_TABLE table=default.Table_459903503b5148d09d3cb8f2f5519fab [id=3c69612aaf164dc899dcc252d85636f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.694005 27356 tablet_service.cc:1468] Processing CreateTablet for tablet a4c582b441f6467c99ba0b090abcf4cc (DEFAULT_TABLE table=default.Table_edfc268f076142a2b7c2ef149edf7f5c [id=b287b55c1542426fb4fd0372ef6bcd05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.695585 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4c582b441f6467c99ba0b090abcf4cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.698253 27357 tablet_service.cc:1468] Processing CreateTablet for tablet b0d79c78e0024b7ba3bb3356be7c8ceb (DEFAULT_TABLE table=default.Table_7fb8db1fe4064eb2be0b9bba8b5ec73b [id=e248f9c4b9c9406395aaea7ba3feb9ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.698393 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 3b3bfc63aa0b4e56bec189e834dc6a8d (DEFAULT_TABLE table=default.Table_56d2ca07338b4ce4ad732c5b2401812b [id=6d6f5e8cadf9445d8695c075cfabb2ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.699409 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0d79c78e0024b7ba3bb3356be7c8ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.702687 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bcb9d5f010f4297bab78c0183889238. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.701679 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 33fb6b357bda43e7abeaed5634183ef6 (DEFAULT_TABLE table=default.Table_b325ef7aa1304c2a941f9d6b44c50fe6 [id=fb19a730ea6343fbba98571261d7a894]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.705425 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33fb6b357bda43e7abeaed5634183ef6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:12.707702 27321 leader_election.cc:343] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: fb76ba1d7cd54d6aa69d06a9a5618cd3
I20250903 20:05:12.707882 27321 leader_election.cc:304] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:12.708000 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 961ade36f7d14eba91bb16d5eae2e254. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.708606 27754 raft_consensus.cc:2747] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:12.713749 27500 tablet_service.cc:1468] Processing CreateTablet for tablet 0f5b87c1c4a24f8abdbfd6194f3b2c4a (DEFAULT_TABLE table=default.Table_a2b477ce2adf4ea0bd1b8685c6483f66 [id=524a27cbe1ad4c10aa2fd4f4074105c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.714201 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f5b87c1c4a24f8abdbfd6194f3b2c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.715780 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b520cd881f58424dadf3fa4a013812ee" } modify_external_catalogs: true
I20250903 20:05:12.718581 27359 tablet_service.cc:1468] Processing CreateTablet for tablet 0f5b87c1c4a24f8abdbfd6194f3b2c4a (DEFAULT_TABLE table=default.Table_a2b477ce2adf4ea0bd1b8685c6483f66 [id=524a27cbe1ad4c10aa2fd4f4074105c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.718683 27499 tablet_service.cc:1468] Processing CreateTablet for tablet b0d79c78e0024b7ba3bb3356be7c8ceb (DEFAULT_TABLE table=default.Table_7fb8db1fe4064eb2be0b9bba8b5ec73b [id=e248f9c4b9c9406395aaea7ba3feb9ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.719031 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f5b87c1c4a24f8abdbfd6194f3b2c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.719066 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0d79c78e0024b7ba3bb3356be7c8ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.722604 27498 tablet_service.cc:1468] Processing CreateTablet for tablet a4c582b441f6467c99ba0b090abcf4cc (DEFAULT_TABLE table=default.Table_edfc268f076142a2b7c2ef149edf7f5c [id=b287b55c1542426fb4fd0372ef6bcd05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.723016 27498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4c582b441f6467c99ba0b090abcf4cc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:12.739099 27066 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 33fb6b357bda43e7abeaed5634183ef6 consensus state has no leader
I20250903 20:05:12.741297 27066 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 9367b3a22f624c4087848099de32c080 (table default.Table_08918e8b515e4487bf4aee0a33234bd5 [id=891b259c1c6b44e2a82dc982c24e6cb6]). Expected version 1 got 0
I20250903 20:05:12.742476 27800 raft_consensus.cc:491] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:12.742650 27800 raft_consensus.cc:513] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.743440 27800 leader_election.cc:290] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:12.743911 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 885fff47fea14b84ac6b34a57b827b4b
I20250903 20:05:12.744422 27057 catalog_manager.cc:6652] ReplaceTablet: tablet 885fff47fea14b84ac6b34a57b827b4b deleted and replaced by tablet 5f27f525fdab407280aa23df1ae71b43
I20250903 20:05:12.744836 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_48f65d1e5e0b4836bb4df770bd94b4e0" } modify_external_catalogs: true
I20250903 20:05:12.745051 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_48f65d1e5e0b4836bb4df770bd94b4e0" } modify_external_catalogs: true
I20250903 20:05:12.747191 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9367b3a22f624c4087848099de32c080" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:12.748411 27462 leader_election.cc:343] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:12.749362 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 369c750ff5bf449e989d95ffd89497d4
I20250903 20:05:12.750118 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_44d61e5bea3c4bbab999158701853479"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:12.750636 27217 tablet_service.cc:1468] Processing CreateTablet for tablet b0d79c78e0024b7ba3bb3356be7c8ceb (DEFAULT_TABLE table=default.Table_7fb8db1fe4064eb2be0b9bba8b5ec73b [id=e248f9c4b9c9406395aaea7ba3feb9ba]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:12.750785 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44d61e5bea3c4bbab999158701853479 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:12.750784 27501 tablet_service.cc:1468] Processing CreateTablet for tablet 33fb6b357bda43e7abeaed5634183ef6 (DEFAULT_TABLE table=default.Table_b325ef7aa1304c2a941f9d6b44c50fe6 [id=fb19a730ea6343fbba98571261d7a894]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.751191 27501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33fb6b357bda43e7abeaed5634183ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.754557 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 3b3bfc63aa0b4e56bec189e834dc6a8d (DEFAULT_TABLE table=default.Table_56d2ca07338b4ce4ad732c5b2401812b [id=6d6f5e8cadf9445d8695c075cfabb2ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.754700 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b3bfc63aa0b4e56bec189e834dc6a8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.755122 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b3bfc63aa0b4e56bec189e834dc6a8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.755476 27220 tablet_service.cc:1468] Processing CreateTablet for tablet 0f5b87c1c4a24f8abdbfd6194f3b2c4a (DEFAULT_TABLE table=default.Table_a2b477ce2adf4ea0bd1b8685c6483f66 [id=524a27cbe1ad4c10aa2fd4f4074105c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.756168 27695 tablet_bootstrap.cc:492] T 9509364ba3ec4e9bad09db09406a978a P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:12.757054 27695 ts_tablet_manager.cc:1397] T 9509364ba3ec4e9bad09db09406a978a P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.171s user 0.001s sys 0.006s
I20250903 20:05:12.757926 27695 raft_consensus.cc:357] T 9509364ba3ec4e9bad09db09406a978a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.758386 27215 tablet_service.cc:1468] Processing CreateTablet for tablet a4c582b441f6467c99ba0b090abcf4cc (DEFAULT_TABLE table=default.Table_edfc268f076142a2b7c2ef149edf7f5c [id=b287b55c1542426fb4fd0372ef6bcd05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.759305 27695 raft_consensus.cc:383] T 9509364ba3ec4e9bad09db09406a978a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:12.759404 27695 raft_consensus.cc:738] T 9509364ba3ec4e9bad09db09406a978a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
W20250903 20:05:12.759750 27184 leader_election.cc:343] T 46b5d491888147bbb101963641fd05f8 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:12.760309 27213 tablet_service.cc:1468] Processing CreateTablet for tablet 33fb6b357bda43e7abeaed5634183ef6 (DEFAULT_TABLE table=default.Table_b325ef7aa1304c2a941f9d6b44c50fe6 [id=fb19a730ea6343fbba98571261d7a894]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.760623 27221 tablet_service.cc:1468] Processing CreateTablet for tablet 961ade36f7d14eba91bb16d5eae2e254 (DEFAULT_TABLE table=default.Table_459903503b5148d09d3cb8f2f5519fab [id=3c69612aaf164dc899dcc252d85636f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.760664 27687 ts_tablet_manager.cc:1918] T 59e120e5e39e4de8a239f13545e4b1af P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:12.760802 27502 tablet_service.cc:1468] Processing CreateTablet for tablet 961ade36f7d14eba91bb16d5eae2e254 (DEFAULT_TABLE table=default.Table_459903503b5148d09d3cb8f2f5519fab [id=3c69612aaf164dc899dcc252d85636f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:12.761173 27695 consensus_queue.cc:260] T 9509364ba3ec4e9bad09db09406a978a P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.761358 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 961ade36f7d14eba91bb16d5eae2e254. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.761046 27687 log.cc:1199] T 59e120e5e39e4de8a239f13545e4b1af P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/59e120e5e39e4de8a239f13545e4b1af
I20250903 20:05:12.763335 27687 ts_tablet_manager.cc:1939] T 59e120e5e39e4de8a239f13545e4b1af P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:12.766553 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9367b3a22f624c4087848099de32c080" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:12.766844 27248 raft_consensus.cc:2466] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:12.766851 27695 ts_tablet_manager.cc:1428] T 9509364ba3ec4e9bad09db09406a978a P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250903 20:05:12.768128 27462 leader_election.cc:304] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:12.768735 27800 raft_consensus.cc:2802] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:12.768870 27800 raft_consensus.cc:491] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:12.768988 27800 raft_consensus.cc:3058] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:12.769404 27687 tablet_replica.cc:331] T 0f4ef613b0ab4bca98294682b934f6ed P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:12.770005 27695 tablet_bootstrap.cc:492] T 710e3bc3c4194efbbeef6bc76ee64f24 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:12.770138 27184 leader_election.cc:304] T 46b5d491888147bbb101963641fd05f8 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:12.770820 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0d79c78e0024b7ba3bb3356be7c8ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.771844 27800 raft_consensus.cc:513] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.772387 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33fb6b357bda43e7abeaed5634183ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.772259 27687 raft_consensus.cc:2241] T 0f4ef613b0ab4bca98294682b934f6ed P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:12.773075 27800 leader_election.cc:290] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:12.773028 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9367b3a22f624c4087848099de32c080" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:12.773670 27687 raft_consensus.cc:2270] T 0f4ef613b0ab4bca98294682b934f6ed P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:12.773761 27184 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 67809.5us
I20250903 20:05:12.774013 27937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=23 ignored=0} mutations{seen=20 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250903 20:05:12.774029 27840 raft_consensus.cc:2747] T 46b5d491888147bbb101963641fd05f8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:12.776674 27687 ts_tablet_manager.cc:1905] T 0f4ef613b0ab4bca98294682b934f6ed P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:12.776813 27692 tablet_bootstrap.cc:492] T 46b5d491888147bbb101963641fd05f8 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:12.776979 27692 ts_tablet_manager.cc:1397] T 46b5d491888147bbb101963641fd05f8 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.250s user 0.038s sys 0.035s
I20250903 20:05:12.777801 27692 raft_consensus.cc:357] T 46b5d491888147bbb101963641fd05f8 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.778016 27692 raft_consensus.cc:383] T 46b5d491888147bbb101963641fd05f8 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:12.778105 27692 raft_consensus.cc:738] T 46b5d491888147bbb101963641fd05f8 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:12.778403 27692 consensus_queue.cc:260] T 46b5d491888147bbb101963641fd05f8 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.780534 27573 heartbeater.cc:344] Connected to a master server at 127.26.65.126:39015
I20250903 20:05:12.780758 27573 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
W20250903 20:05:12.785580 27462 leader_election.cc:343] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:12.788661 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f5b87c1c4a24f8abdbfd6194f3b2c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.789456 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9367b3a22f624c4087848099de32c080" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:12.789675 27248 raft_consensus.cc:3058] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:12.789734 27937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2: Bootstrap complete.
I20250903 20:05:12.781575 27692 ts_tablet_manager.cc:1428] T 46b5d491888147bbb101963641fd05f8 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250903 20:05:12.791270 27573 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
I20250903 20:05:12.792340 27687 ts_tablet_manager.cc:1918] T 0f4ef613b0ab4bca98294682b934f6ed P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:12.792672 27687 log.cc:1199] T 0f4ef613b0ab4bca98294682b934f6ed P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/0f4ef613b0ab4bca98294682b934f6ed
I20250903 20:05:12.793419 27687 ts_tablet_manager.cc:1939] T 0f4ef613b0ab4bca98294682b934f6ed P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:12.795135 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_0064d6a69e9a474bb17ad72deb3fbbc8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:12.797107 27692 tablet_bootstrap.cc:492] T 925c0ab4873041da82702cd9377ec7bc P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:12.798238 27937 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } }
I20250903 20:05:12.799849 27687 tablet_replica.cc:331] T a45604ad611e418e84dab793e04101a0 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:12.800208 27687 raft_consensus.cc:2241] T a45604ad611e418e84dab793e04101a0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:12.801393 27687 raft_consensus.cc:2270] T a45604ad611e418e84dab793e04101a0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:12.803105 27687 ts_tablet_manager.cc:1905] T a45604ad611e418e84dab793e04101a0 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:12.803450 27937 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8857b5d4ea1e4f45aafb8e029837cca2, State: Initialized, Role: FOLLOWER
I20250903 20:05:12.808403 27840 consensus_queue.cc:1035] T 99a21a3e49934a79a66700c0b0d86307 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20250903 20:05:12.807610 27937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } }
I20250903 20:05:12.810734 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4c582b441f6467c99ba0b090abcf4cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.812155 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 961ade36f7d14eba91bb16d5eae2e254. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:12.815166 27692 tablet_bootstrap.cc:654] T 925c0ab4873041da82702cd9377ec7bc P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:12.816288 27840 raft_consensus.cc:491] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:12.818670 27840 raft_consensus.cc:513] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.821630 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1396d4f8a93b412ca8397368fb98834e" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:12.820715 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0064d6a69e9a474bb17ad72deb3fbbc8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:12.822727 27687 ts_tablet_manager.cc:1918] T a45604ad611e418e84dab793e04101a0 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:12.822906 27687 log.cc:1199] T a45604ad611e418e84dab793e04101a0 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/a45604ad611e418e84dab793e04101a0
I20250903 20:05:12.823330 27687 ts_tablet_manager.cc:1939] T a45604ad611e418e84dab793e04101a0 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:12.823385 27949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:12.823724 27949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 20:05:12.824484 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1396d4f8a93b412ca8397368fb98834e" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:12.824719 27525 raft_consensus.cc:2466] T 1396d4f8a93b412ca8397368fb98834e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:12.825985 27937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 20:05:12.826577 27687 tablet_replica.cc:331] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:12.826797 27687 raft_consensus.cc:2241] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:12.826982 27687 raft_consensus.cc:2270] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:12.828655 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_36a3bbdf545442dcacd498d928fa996b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:12.828840 27840 leader_election.cc:290] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
W20250903 20:05:12.829283 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36a3bbdf545442dcacd498d928fa996b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:12.829284 27184 leader_election.cc:343] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:12.831827 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 885fff47fea14b84ac6b34a57b827b4b with delete_type TABLET_DATA_DELETED (replaced by tablet 5f27f525fdab407280aa23df1ae71b43) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:12.832085 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet 369c750ff5bf449e989d95ffd89497d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:12.832197 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 59e120e5e39e4de8a239f13545e4b1af (table default.Table_2dd5b5956b44471da9d0b89e782d469e [id=0bfcd1b8b71d45db8d9c370bd1231a38]) successfully deleted
I20250903 20:05:12.832471 27695 tablet_bootstrap.cc:654] T 710e3bc3c4194efbbeef6bc76ee64f24 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:12.832485 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 4be9bdc390de4a74a396003c9c3c2b71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4be9bdc390de4a74a396003c9c3c2b71 already in progress: creating tablet
I20250903 20:05:12.832772 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 0f4ef613b0ab4bca98294682b934f6ed (table default.Table_1ade5bba3a18432c808ee7c4b910d247 [id=d20347a8196e44459448223a3af2dead]) successfully deleted
W20250903 20:05:12.832948 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet ecd339ec12e84e88acec427630cc8668 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecd339ec12e84e88acec427630cc8668 already in progress: creating tablet
W20250903 20:05:12.833127 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 44db427ee53a4d0a907dfd63282d6d4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44db427ee53a4d0a907dfd63282d6d4f already in progress: creating tablet
W20250903 20:05:12.833310 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 7c398634d6304a12b35c0579fba4c813 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c398634d6304a12b35c0579fba4c813 already in progress: creating tablet
W20250903 20:05:12.833526 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 9509364ba3ec4e9bad09db09406a978a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9509364ba3ec4e9bad09db09406a978a already in progress: creating tablet
I20250903 20:05:12.833714 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet a45604ad611e418e84dab793e04101a0 (table default.Table_7d8b7496fbc04f35bab6d6da573d4868 [id=5383eb1038e6480893e6eab6c7f7dd39]) successfully deleted
I20250903 20:05:12.837636 27687 ts_tablet_manager.cc:1905] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:12.842234 27573 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
I20250903 20:05:12.844368 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet 369c750ff5bf449e989d95ffd89497d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:12.845289 27879 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [term 2 FOLLOWER]: Advancing to term 3
I20250903 20:05:12.852550 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet 885fff47fea14b84ac6b34a57b827b4b with delete_type TABLET_DATA_DELETED (replaced by tablet 5f27f525fdab407280aa23df1ae71b43) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:12.856114 27051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.125:44059, user_credentials={real_user=slave}} blocked reactor thread for 131166us
I20250903 20:05:12.861924 27693 tablet_bootstrap.cc:492] T eaa11d17303142ff935705c16f0b9690 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:12.862102 27693 ts_tablet_manager.cc:1397] T eaa11d17303142ff935705c16f0b9690 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.348s user 0.000s sys 0.004s
I20250903 20:05:12.863004 27693 raft_consensus.cc:357] T eaa11d17303142ff935705c16f0b9690 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.863176 27693 raft_consensus.cc:383] T eaa11d17303142ff935705c16f0b9690 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:12.863263 27693 raft_consensus.cc:738] T eaa11d17303142ff935705c16f0b9690 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:12.863476 27693 consensus_queue.cc:260] T eaa11d17303142ff935705c16f0b9690 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.863978 27693 ts_tablet_manager.cc:1428] T eaa11d17303142ff935705c16f0b9690 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:12.864462 27573 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
I20250903 20:05:12.865224 27432 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
I20250903 20:05:12.866078 27693 tablet_bootstrap.cc:492] T 7e0810c96eb24b8ab4278038801be245 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:12.868266 27693 tablet_bootstrap.cc:654] T 7e0810c96eb24b8ab4278038801be245 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:12.876754 27295 heartbeater.cc:344] Connected to a master server at 127.26.65.126:39015
I20250903 20:05:12.877074 27295 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
I20250903 20:05:12.878247 27685 ts_tablet_manager.cc:1918] T 76742d0df44c43729197f7e09f96cad8 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:12.878461 27685 log.cc:1199] T 76742d0df44c43729197f7e09f96cad8 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/76742d0df44c43729197f7e09f96cad8
I20250903 20:05:12.879918 27951 consensus_queue.cc:1035] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:12.880695 27693 tablet_bootstrap.cc:492] T 7e0810c96eb24b8ab4278038801be245 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:12.880905 27693 ts_tablet_manager.cc:1397] T 7e0810c96eb24b8ab4278038801be245 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.001s
I20250903 20:05:12.881744 27248 raft_consensus.cc:2466] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:12.882450 27462 leader_election.cc:304] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:12.883406 27692 tablet_bootstrap.cc:492] T 925c0ab4873041da82702cd9377ec7bc P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:12.883323 27693 raft_consensus.cc:357] T 7e0810c96eb24b8ab4278038801be245 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.883545 27693 raft_consensus.cc:383] T 7e0810c96eb24b8ab4278038801be245 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:12.883626 27693 raft_consensus.cc:738] T 7e0810c96eb24b8ab4278038801be245 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:12.883752 27692 ts_tablet_manager.cc:1397] T 925c0ab4873041da82702cd9377ec7bc P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.088s user 0.006s sys 0.006s
I20250903 20:05:12.883925 27693 consensus_queue.cc:260] T 7e0810c96eb24b8ab4278038801be245 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.884403 27432 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
I20250903 20:05:12.884722 27692 raft_consensus.cc:357] T 925c0ab4873041da82702cd9377ec7bc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:12.885218 27692 raft_consensus.cc:383] T 925c0ab4873041da82702cd9377ec7bc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:12.885430 27692 raft_consensus.cc:738] T 925c0ab4873041da82702cd9377ec7bc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:12.885752 27692 consensus_queue.cc:260] T 925c0ab4873041da82702cd9377ec7bc P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:12.885118 27951 raft_consensus.cc:2802] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:12.886623 27951 raft_consensus.cc:695] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:12.886960 27951 consensus_queue.cc:237] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.887089 27693 ts_tablet_manager.cc:1428] T 7e0810c96eb24b8ab4278038801be245 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20250903 20:05:12.887683 27692 ts_tablet_manager.cc:1428] T 925c0ab4873041da82702cd9377ec7bc P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250903 20:05:12.888962 27692 tablet_bootstrap.cc:492] T 76742d0df44c43729197f7e09f96cad8 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:12.889429 27693 tablet_bootstrap.cc:492] T 99a21a3e49934a79a66700c0b0d86307 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:12.890339 27685 ts_tablet_manager.cc:1939] T 76742d0df44c43729197f7e09f96cad8 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:12.890585 27573 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
I20250903 20:05:12.891327 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 76742d0df44c43729197f7e09f96cad8 (table default.Table_3055928e679e4dd8b8559b363b580e13 [id=9545d6a8d12e4937bb2b4cdbfc6933de]) successfully deleted
I20250903 20:05:12.892714 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet 885fff47fea14b84ac6b34a57b827b4b with delete_type TABLET_DATA_DELETED (replaced by tablet 5f27f525fdab407280aa23df1ae71b43) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:12.898392 27693 tablet_bootstrap.cc:654] T 99a21a3e49934a79a66700c0b0d86307 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:12.899628 27685 tablet_replica.cc:331] T 46b5d491888147bbb101963641fd05f8 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:12.901829 27692 tablet_bootstrap.cc:654] T 76742d0df44c43729197f7e09f96cad8 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:12.902071 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet 369c750ff5bf449e989d95ffd89497d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:12.903744 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet 29581a9dba46479884a0e6e01b7923de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:12.904071 27684 tablet_replica.cc:331] T 29581a9dba46479884a0e6e01b7923de P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:12.904335 27684 raft_consensus.cc:2241] T 29581a9dba46479884a0e6e01b7923de P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:12.904559 27684 raft_consensus.cc:2270] T 29581a9dba46479884a0e6e01b7923de P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:12.905380 27684 ts_tablet_manager.cc:1905] T 29581a9dba46479884a0e6e01b7923de P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:12.909911 27432 heartbeater.cc:344] Connected to a master server at 127.26.65.126:39015
I20250903 20:05:12.910130 27432 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
I20250903 20:05:12.911044 27693 tablet_bootstrap.cc:492] T 99a21a3e49934a79a66700c0b0d86307 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:12.911232 27693 ts_tablet_manager.cc:1397] T 99a21a3e49934a79a66700c0b0d86307 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.003s
I20250903 20:05:12.912343 27693 raft_consensus.cc:357] T 99a21a3e49934a79a66700c0b0d86307 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.912566 27693 raft_consensus.cc:383] T 99a21a3e49934a79a66700c0b0d86307 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:12.912669 27693 raft_consensus.cc:738] T 99a21a3e49934a79a66700c0b0d86307 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:12.912915 27684 ts_tablet_manager.cc:1918] T 29581a9dba46479884a0e6e01b7923de P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:12.912995 27693 consensus_queue.cc:260] T 99a21a3e49934a79a66700c0b0d86307 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.913093 27684 log.cc:1199] T 29581a9dba46479884a0e6e01b7923de P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/29581a9dba46479884a0e6e01b7923de
I20250903 20:05:12.913585 27684 ts_tablet_manager.cc:1939] T 29581a9dba46479884a0e6e01b7923de P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:12.913710 27693 ts_tablet_manager.cc:1428] T 99a21a3e49934a79a66700c0b0d86307 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:12.917343 27693 tablet_bootstrap.cc:492] T be278233c7c34061988b267a44ee674f P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:12.917984 27685 raft_consensus.cc:2241] T 46b5d491888147bbb101963641fd05f8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:12.919596 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet 4692dd93b76c42c4883c7e22c6032250 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:12.921118 27693 tablet_bootstrap.cc:654] T be278233c7c34061988b267a44ee674f P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:12.924809 27685 raft_consensus.cc:2270] T 46b5d491888147bbb101963641fd05f8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:12.925921 27685 ts_tablet_manager.cc:1905] T 46b5d491888147bbb101963641fd05f8 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:12.931113 27684 tablet_replica.cc:331] T 4692dd93b76c42c4883c7e22c6032250 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:12.931497 27684 raft_consensus.cc:2241] T 4692dd93b76c42c4883c7e22c6032250 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:12.931733 27684 raft_consensus.cc:2270] T 4692dd93b76c42c4883c7e22c6032250 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:12.932596 27684 ts_tablet_manager.cc:1905] T 4692dd93b76c42c4883c7e22c6032250 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:12.937106 27692 tablet_bootstrap.cc:492] T 76742d0df44c43729197f7e09f96cad8 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:12.938970 27432 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
I20250903 20:05:12.940964 27692 ts_tablet_manager.cc:1397] T 76742d0df44c43729197f7e09f96cad8 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.001s
I20250903 20:05:12.944530 27692 raft_consensus.cc:357] T 76742d0df44c43729197f7e09f96cad8 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.944793 27692 raft_consensus.cc:383] T 76742d0df44c43729197f7e09f96cad8 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:12.944877 27692 raft_consensus.cc:738] T 76742d0df44c43729197f7e09f96cad8 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:12.947268 27693 tablet_bootstrap.cc:492] T be278233c7c34061988b267a44ee674f P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:12.947537 27693 ts_tablet_manager.cc:1397] T be278233c7c34061988b267a44ee674f P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.003s
I20250903 20:05:12.948550 27692 consensus_queue.cc:260] T 76742d0df44c43729197f7e09f96cad8 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.948678 27693 raft_consensus.cc:357] T be278233c7c34061988b267a44ee674f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.949007 27693 raft_consensus.cc:383] T be278233c7c34061988b267a44ee674f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:12.949126 27693 raft_consensus.cc:738] T be278233c7c34061988b267a44ee674f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:12.949167 27692 ts_tablet_manager.cc:1428] T 76742d0df44c43729197f7e09f96cad8 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250903 20:05:12.949577 27692 tablet_bootstrap.cc:492] T d8ed7b3e292241cb9525745a797ff13a P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:12.951448 27692 tablet_bootstrap.cc:654] T d8ed7b3e292241cb9525745a797ff13a P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:12.955250 27693 consensus_queue.cc:260] T be278233c7c34061988b267a44ee674f P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.956053 27693 ts_tablet_manager.cc:1428] T be278233c7c34061988b267a44ee674f P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250903 20:05:12.956634 27693 tablet_bootstrap.cc:492] T e1a49d0599654552bf1c72857031699b P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:12.957201 27684 ts_tablet_manager.cc:1918] T 4692dd93b76c42c4883c7e22c6032250 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:12.957360 27684 log.cc:1199] T 4692dd93b76c42c4883c7e22c6032250 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/4692dd93b76c42c4883c7e22c6032250
I20250903 20:05:12.957736 27684 ts_tablet_manager.cc:1939] T 4692dd93b76c42c4883c7e22c6032250 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:12.961395 27693 tablet_bootstrap.cc:654] T e1a49d0599654552bf1c72857031699b P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:12.964715 27573 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
I20250903 20:05:12.968688 27432 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
W20250903 20:05:12.979538 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 885fff47fea14b84ac6b34a57b827b4b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 885fff47fea14b84ac6b34a57b827b4b already in progress: creating tablet
W20250903 20:05:12.979892 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 369c750ff5bf449e989d95ffd89497d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 369c750ff5bf449e989d95ffd89497d4 already in progress: creating tablet
I20250903 20:05:12.980101 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 29581a9dba46479884a0e6e01b7923de (table default.Table_7e75298df3b54e3794ab39c8b0fbffc7 [id=cf5354e1f11f4a228ea25ec2baa569a4]) successfully deleted
I20250903 20:05:12.980297 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 4692dd93b76c42c4883c7e22c6032250 (table default.Table_7dfd9faa02434381b40a56369804413b [id=f11bdb54ed5c4b1ca1e1306d39aaf25f]) successfully deleted
I20250903 20:05:12.980398 27693 tablet_bootstrap.cc:492] T e1a49d0599654552bf1c72857031699b P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:12.980574 27693 ts_tablet_manager.cc:1397] T e1a49d0599654552bf1c72857031699b P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20250903 20:05:12.981480 27693 raft_consensus.cc:357] T e1a49d0599654552bf1c72857031699b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:12.981679 27693 raft_consensus.cc:383] T e1a49d0599654552bf1c72857031699b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:12.981757 27693 raft_consensus.cc:738] T e1a49d0599654552bf1c72857031699b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:12.982854 27693 consensus_queue.cc:260] T e1a49d0599654552bf1c72857031699b P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:12.983640 27183 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 150703us
I20250903 20:05:12.984180 27183 leader_election.cc:304] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:12.987217 27687 ts_tablet_manager.cc:1918] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:12.987474 27687 log.cc:1199] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/fb76ba1d7cd54d6aa69d06a9a5618cd3
I20250903 20:05:12.987861 27693 ts_tablet_manager.cc:1428] T e1a49d0599654552bf1c72857031699b P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250903 20:05:12.987999 27687 ts_tablet_manager.cc:1939] T fb76ba1d7cd54d6aa69d06a9a5618cd3 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:12.994530 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet fb76ba1d7cd54d6aa69d06a9a5618cd3 (table default.Table_a249558ab60e466ca7734ffeb24ac5fd [id=d1ed6b104f9541fbaf34c6b03b297f0f]) successfully deleted
W20250903 20:05:12.994910 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 885fff47fea14b84ac6b34a57b827b4b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 885fff47fea14b84ac6b34a57b827b4b already in progress: creating tablet
I20250903 20:05:12.995136 27693 tablet_bootstrap.cc:492] T f3a33e90dc1f4540aee457de96f5b95b P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
W20250903 20:05:12.995152 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 369c750ff5bf449e989d95ffd89497d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 369c750ff5bf449e989d95ffd89497d4 already in progress: creating tablet
I20250903 20:05:12.996996 27693 tablet_bootstrap.cc:654] T f3a33e90dc1f4540aee457de96f5b95b P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:13.001017 27063 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet bf05d8d8cda949959e8bef8820de0032
I20250903 20:05:13.003773 27695 tablet_bootstrap.cc:492] T 710e3bc3c4194efbbeef6bc76ee64f24 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:13.003974 27695 ts_tablet_manager.cc:1397] T 710e3bc3c4194efbbeef6bc76ee64f24 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.237s user 0.001s sys 0.029s
I20250903 20:05:13.004288 27432 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
I20250903 20:05:13.004624 27063 catalog_manager.cc:5582] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f reported cstate change: term changed from 0 to 1, leader changed from <none> to e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67). New cstate: current_term: 1 leader_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:13.005093 27063 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 9367b3a22f624c4087848099de32c080 (table default.Table_08918e8b515e4487bf4aee0a33234bd5 [id=891b259c1c6b44e2a82dc982c24e6cb6]). Expected version 1 got 0
I20250903 20:05:13.005497 27063 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 99a21a3e49934a79a66700c0b0d86307 (table default.Table_7325977ff92e45bab0b59b79579ecae4 [id=428657b402d748c0a9e3942d13342b4e]). Expected version 1 got 0
I20250903 20:05:13.006496 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet a0293a69f1c246d08d414e95729b4ab4
I20250903 20:05:13.006667 27953 raft_consensus.cc:2802] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:13.006764 27953 raft_consensus.cc:491] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:13.006870 27953 raft_consensus.cc:3058] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:13.007948 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet a0293a69f1c246d08d414e95729b4ab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:13.008503 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_43133b63f38043d6bb2757731065b56f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:13.009183 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43133b63f38043d6bb2757731065b56f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:13.009765 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet a0293a69f1c246d08d414e95729b4ab4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0293a69f1c246d08d414e95729b4ab4 already in progress: creating tablet
I20250903 20:05:13.009889 27695 raft_consensus.cc:357] T 710e3bc3c4194efbbeef6bc76ee64f24 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.010066 27695 raft_consensus.cc:383] T 710e3bc3c4194efbbeef6bc76ee64f24 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:13.010532 27693 tablet_bootstrap.cc:492] T f3a33e90dc1f4540aee457de96f5b95b P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:13.010617 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet cfa8a1d9011e460681443a6b77b95958
I20250903 20:05:13.010746 27693 ts_tablet_manager.cc:1397] T f3a33e90dc1f4540aee457de96f5b95b P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.002s
I20250903 20:05:13.010730 27695 raft_consensus.cc:738] T 710e3bc3c4194efbbeef6bc76ee64f24 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:13.011088 27062 catalog_manager.cc:6652] ReplaceTablet: tablet cfa8a1d9011e460681443a6b77b95958 deleted and replaced by tablet 4af8c03c9d834405b4e25d35441db961
I20250903 20:05:13.010978 27695 consensus_queue.cc:260] T 710e3bc3c4194efbbeef6bc76ee64f24 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.011525 27695 ts_tablet_manager.cc:1428] T 710e3bc3c4194efbbeef6bc76ee64f24 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250903 20:05:13.011427 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_58997b732a0f46e69409a76f193c699b" } new_table_name: "default.Table_3185c25ec3784288bd52ccf75830589d" modify_external_catalogs: true
I20250903 20:05:13.011638 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet cfa8a1d9011e460681443a6b77b95958 with delete_type TABLET_DATA_DELETED (replaced by tablet 4af8c03c9d834405b4e25d35441db961) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:13.011849 27695 tablet_bootstrap.cc:492] T e5136d5388bc4c2ba71e09be40e8f412 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
W20250903 20:05:13.013082 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet cfa8a1d9011e460681443a6b77b95958 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cfa8a1d9011e460681443a6b77b95958 already in progress: creating tablet
I20250903 20:05:13.013756 27693 raft_consensus.cc:357] T f3a33e90dc1f4540aee457de96f5b95b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.013970 27693 raft_consensus.cc:383] T f3a33e90dc1f4540aee457de96f5b95b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:13.014082 27693 raft_consensus.cc:738] T f3a33e90dc1f4540aee457de96f5b95b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:13.014358 27693 consensus_queue.cc:260] T f3a33e90dc1f4540aee457de96f5b95b P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.014838 27693 ts_tablet_manager.cc:1428] T f3a33e90dc1f4540aee457de96f5b95b P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250903 20:05:13.015165 27693 tablet_bootstrap.cc:492] T 4bd2650e4d9e40f5ba5c09e367918224 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:13.017035 27693 tablet_bootstrap.cc:654] T 4bd2650e4d9e40f5ba5c09e367918224 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:13.027056 27386 raft_consensus.cc:3058] T 99a21a3e49934a79a66700c0b0d86307 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:13.033598 27695 tablet_bootstrap.cc:654] T e5136d5388bc4c2ba71e09be40e8f412 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:13.035364 27432 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
I20250903 20:05:13.037999 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet cfa8a1d9011e460681443a6b77b95958 with delete_type TABLET_DATA_DELETED (replaced by tablet 4af8c03c9d834405b4e25d35441db961) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:13.038420 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 9509364ba3ec4e9bad09db09406a978a with delete_type TABLET_DATA_DELETED (replaced by tablet 1fbe54dc40dd47daa109ceeb07546b0a) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:13.038461 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet a0293a69f1c246d08d414e95729b4ab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:13.038758 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet cfa8a1d9011e460681443a6b77b95958 with delete_type TABLET_DATA_DELETED (replaced by tablet 4af8c03c9d834405b4e25d35441db961) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:13.038923 27213 tablet_service.cc:1515] Processing DeleteTablet for tablet a0293a69f1c246d08d414e95729b4ab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:13.039310 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet a0293a69f1c246d08d414e95729b4ab4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0293a69f1c246d08d414e95729b4ab4 already in progress: creating tablet
I20250903 20:05:13.042074 27953 raft_consensus.cc:513] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.049480 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1396d4f8a93b412ca8397368fb98834e" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
W20250903 20:05:13.050150 27184 leader_election.cc:343] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:13.050783 27953 leader_election.cc:290] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:13.051312 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1396d4f8a93b412ca8397368fb98834e" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:13.051569 27526 raft_consensus.cc:3058] T 1396d4f8a93b412ca8397368fb98834e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:13.053064 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet cfa8a1d9011e460681443a6b77b95958 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cfa8a1d9011e460681443a6b77b95958 already in progress: creating tablet
I20250903 20:05:13.054611 27526 raft_consensus.cc:2466] T 1396d4f8a93b412ca8397368fb98834e P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:13.055059 27183 leader_election.cc:304] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:13.061694 27754 raft_consensus.cc:491] T f3a33e90dc1f4540aee457de96f5b95b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:13.061836 27958 raft_consensus.cc:2802] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:13.061898 27754 raft_consensus.cc:513] T f3a33e90dc1f4540aee457de96f5b95b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.062745 27754 leader_election.cc:290] T f3a33e90dc1f4540aee457de96f5b95b P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:13.063109 27982 consensus_queue.cc:1035] T 99a21a3e49934a79a66700c0b0d86307 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250903 20:05:13.063802 27958 raft_consensus.cc:695] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
W20250903 20:05:13.063802 27321 leader_election.cc:343] T f3a33e90dc1f4540aee457de96f5b95b P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: f3a33e90dc1f4540aee457de96f5b95b
I20250903 20:05:13.064134 27958 consensus_queue.cc:237] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:13.064325 27320 leader_election.cc:343] T f3a33e90dc1f4540aee457de96f5b95b P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: f3a33e90dc1f4540aee457de96f5b95b
I20250903 20:05:13.064456 27320 leader_election.cc:304] T f3a33e90dc1f4540aee457de96f5b95b P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:13.065035 27754 raft_consensus.cc:2747] T f3a33e90dc1f4540aee457de96f5b95b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:13.067878 27692 tablet_bootstrap.cc:492] T d8ed7b3e292241cb9525745a797ff13a P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:13.068056 27692 ts_tablet_manager.cc:1397] T d8ed7b3e292241cb9525745a797ff13a P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.119s user 0.004s sys 0.000s
I20250903 20:05:13.068851 27692 raft_consensus.cc:357] T d8ed7b3e292241cb9525745a797ff13a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.069069 27692 raft_consensus.cc:383] T d8ed7b3e292241cb9525745a797ff13a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:13.069160 27692 raft_consensus.cc:738] T d8ed7b3e292241cb9525745a797ff13a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:13.069424 27692 consensus_queue.cc:260] T d8ed7b3e292241cb9525745a797ff13a P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.069962 27692 ts_tablet_manager.cc:1428] T d8ed7b3e292241cb9525745a797ff13a P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20250903 20:05:13.071018 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b0d79c78e0024b7ba3bb3356be7c8ceb consensus state has no leader
I20250903 20:05:13.071478 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_c005844d740343c0a67ba9977747f574"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:13.072062 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c005844d740343c0a67ba9977747f574 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:13.072255 27692 tablet_bootstrap.cc:492] T 8945085bdc174e6db22a5a9477810297 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:13.073575 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_957f51b8fb684329a6e6fac205da5164" } new_table_name: "default.Table_1c679659094d4666aaaf68f76f667890" modify_external_catalogs: true
I20250903 20:05:13.074995 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet eced730bdbc14a8c9f6b7b7e53435fd0
I20250903 20:05:13.075583 27692 tablet_bootstrap.cc:654] T 8945085bdc174e6db22a5a9477810297 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:13.075810 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_2fa1d4a13c1840cd88c36e5fa2c6475f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:13.076342 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2fa1d4a13c1840cd88c36e5fa2c6475f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:13.078157 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 6788b1683f984949816edfc5b68de8db
I20250903 20:05:13.078346 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet eced730bdbc14a8c9f6b7b7e53435fd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:13.078780 27685 ts_tablet_manager.cc:1918] T 46b5d491888147bbb101963641fd05f8 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:13.078943 27685 log.cc:1199] T 46b5d491888147bbb101963641fd05f8 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/46b5d491888147bbb101963641fd05f8
I20250903 20:05:13.078943 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_07ba95129d584160ae8b768f77aa044d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:13.079463 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07ba95129d584160ae8b768f77aa044d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:13.080359 27685 ts_tablet_manager.cc:1939] T 46b5d491888147bbb101963641fd05f8 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:13.081413 27685 tablet_replica.cc:331] T 9509364ba3ec4e9bad09db09406a978a P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:13.081604 27685 raft_consensus.cc:2241] T 9509364ba3ec4e9bad09db09406a978a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:13.081777 27685 raft_consensus.cc:2270] T 9509364ba3ec4e9bad09db09406a978a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:13.083808 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet eced730bdbc14a8c9f6b7b7e53435fd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:13.084095 27501 tablet_service.cc:1515] Processing DeleteTablet for tablet 6788b1683f984949816edfc5b68de8db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:13.084842 27573 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
I20250903 20:05:13.085978 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_19e44161daa5469da23656554f451dd0" } new_table_name: "default.Table_37d794e122f2457294c1eb6115be9472" modify_external_catalogs: true
W20250903 20:05:13.087888 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet eced730bdbc14a8c9f6b7b7e53435fd0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eced730bdbc14a8c9f6b7b7e53435fd0 already in progress: creating tablet
W20250903 20:05:13.088230 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 6788b1683f984949816edfc5b68de8db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6788b1683f984949816edfc5b68de8db already in progress: creating tablet
I20250903 20:05:13.090848 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 46b5d491888147bbb101963641fd05f8 (table default.Table_b88b7b0fe749402e90c3826d01e8cc8e [id=3b241b63c32f42c0966e9af3dc06d701]) successfully deleted
I20250903 20:05:13.090760 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_6656b728e12a482fa9e7893ef4842ac7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:13.091076 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 369c750ff5bf449e989d95ffd89497d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 369c750ff5bf449e989d95ffd89497d4 already in progress: creating tablet
W20250903 20:05:13.091444 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 885fff47fea14b84ac6b34a57b827b4b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 885fff47fea14b84ac6b34a57b827b4b already in progress: creating tablet
W20250903 20:05:13.091650 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet cfa8a1d9011e460681443a6b77b95958 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cfa8a1d9011e460681443a6b77b95958 already in progress: creating tablet
I20250903 20:05:13.092415 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 6788b1683f984949816edfc5b68de8db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:13.092612 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet eced730bdbc14a8c9f6b7b7e53435fd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:13.092865 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6656b728e12a482fa9e7893ef4842ac7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:13.092959 27213 tablet_service.cc:1515] Processing DeleteTablet for tablet 6788b1683f984949816edfc5b68de8db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:13.093998 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet eced730bdbc14a8c9f6b7b7e53435fd0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eced730bdbc14a8c9f6b7b7e53435fd0 already in progress: creating tablet
I20250903 20:05:13.094156 27432 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
W20250903 20:05:13.096678 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7e2f98f760cf4df4bc2a10530e2157c6 has no consensus state
I20250903 20:05:13.097061 27062 master_service.cc:948] ReplaceTablet: received request to replace tablet 04ccb3e4a8bd4789b9f19edd4a5b6145 from {username='slave'} at 127.0.0.1:45936
W20250903 20:05:13.097201 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 6788b1683f984949816edfc5b68de8db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6788b1683f984949816edfc5b68de8db already in progress: creating tablet
I20250903 20:05:13.097575 27062 master_service.cc:948] ReplaceTablet: received request to replace tablet 31342583c026438cbbae14106e75bbe5 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:13.099045 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2f2c8693b96741448fbd96ff5eb9ad8b" } new_table_name: "default.Table_7ea9ceb5ab084d6082d52019feeaf414" modify_external_catalogs: true
I20250903 20:05:13.111116 27066 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 99a21a3e49934a79a66700c0b0d86307 (table default.Table_7325977ff92e45bab0b59b79579ecae4 [id=428657b402d748c0a9e3942d13342b4e]). Expected version 1 got 0
I20250903 20:05:13.113670 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b4634182d45b490eb24ae0950859ab2c" } modify_external_catalogs: true
I20250903 20:05:13.113930 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b4634182d45b490eb24ae0950859ab2c" } modify_external_catalogs: true
W20250903 20:05:13.114830 27183 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 57650.4us
I20250903 20:05:13.115257 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet be278233c7c34061988b267a44ee674f with delete_type TABLET_DATA_DELETED (replaced by tablet 5bc3b9771a0042fd87e706204dcbb712) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:13.115295 27366 tablet_service.cc:1515] Processing DeleteTablet for tablet f3a33e90dc1f4540aee457de96f5b95b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:13.115445 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e0810c96eb24b8ab4278038801be245 with delete_type TABLET_DATA_DELETED (replaced by tablet e5136d5388bc4c2ba71e09be40e8f412) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:13.115607 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet e1a49d0599654552bf1c72857031699b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:13.115705 27687 tablet_replica.cc:331] T be278233c7c34061988b267a44ee674f P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:13.115954 27687 raft_consensus.cc:2241] T be278233c7c34061988b267a44ee674f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:13.116092 27695 tablet_bootstrap.cc:492] T e5136d5388bc4c2ba71e09be40e8f412 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:13.116168 27687 raft_consensus.cc:2270] T be278233c7c34061988b267a44ee674f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:13.116261 27695 ts_tablet_manager.cc:1397] T e5136d5388bc4c2ba71e09be40e8f412 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.104s user 0.000s sys 0.010s
I20250903 20:05:13.116917 27687 ts_tablet_manager.cc:1905] T be278233c7c34061988b267a44ee674f P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:13.117201 27695 raft_consensus.cc:357] T e5136d5388bc4c2ba71e09be40e8f412 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:13.117442 27695 raft_consensus.cc:383] T e5136d5388bc4c2ba71e09be40e8f412 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:13.119336 27695 raft_consensus.cc:738] T e5136d5388bc4c2ba71e09be40e8f412 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:13.119931 27695 consensus_queue.cc:260] T e5136d5388bc4c2ba71e09be40e8f412 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:13.120589 27695 ts_tablet_manager.cc:1428] T e5136d5388bc4c2ba71e09be40e8f412 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20250903 20:05:13.120918 27695 tablet_bootstrap.cc:492] T c25aaa6ff310432bb27a0695394ec420 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:13.121896 27685 ts_tablet_manager.cc:1905] T 9509364ba3ec4e9bad09db09406a978a P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:13.126827 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 283a3afed7414a958b2dc4176f00bb01 consensus state has no leader
W20250903 20:05:13.131259 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5f27f525fdab407280aa23df1ae71b43 has no consensus state
I20250903 20:05:13.132308 27432 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
I20250903 20:05:13.135198 27501 tablet_service.cc:1515] Processing DeleteTablet for tablet 925c0ab4873041da82702cd9377ec7bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:13.135546 27684 tablet_replica.cc:331] T 925c0ab4873041da82702cd9377ec7bc P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:13.135826 27684 raft_consensus.cc:2241] T 925c0ab4873041da82702cd9377ec7bc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:13.136071 27684 raft_consensus.cc:2270] T 925c0ab4873041da82702cd9377ec7bc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:13.138787 27684 ts_tablet_manager.cc:1905] T 925c0ab4873041da82702cd9377ec7bc P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:13.142586 27687 ts_tablet_manager.cc:1918] T be278233c7c34061988b267a44ee674f P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:13.142751 27687 log.cc:1199] T be278233c7c34061988b267a44ee674f P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/be278233c7c34061988b267a44ee674f
I20250903 20:05:13.143178 27687 ts_tablet_manager.cc:1939] T be278233c7c34061988b267a44ee674f P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:13.143842 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet be278233c7c34061988b267a44ee674f (table default.Table_459903503b5148d09d3cb8f2f5519fab [id=3c69612aaf164dc899dcc252d85636f6]) successfully deleted
I20250903 20:05:13.144565 27695 tablet_bootstrap.cc:654] T c25aaa6ff310432bb27a0695394ec420 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:13.145995 27213 tablet_service.cc:1515] Processing DeleteTablet for tablet 710e3bc3c4194efbbeef6bc76ee64f24 with delete_type TABLET_DATA_DELETED (replaced by tablet 4be9bdc390de4a74a396003c9c3c2b71) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:13.147581 27065 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 99a21a3e49934a79a66700c0b0d86307 (table default.Table_7325977ff92e45bab0b59b79579ecae4 [id=428657b402d748c0a9e3942d13342b4e]). Expected version 1 got 0
I20250903 20:05:13.149212 27501 tablet_service.cc:1515] Processing DeleteTablet for tablet 46b5d491888147bbb101963641fd05f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:13.152187 27059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0ad64c699b294679b6fec37b5ad23528 consensus state has no leader
I20250903 20:05:13.152725 27685 ts_tablet_manager.cc:1918] T 9509364ba3ec4e9bad09db09406a978a P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:13.153848 27687 tablet_replica.cc:331] T f3a33e90dc1f4540aee457de96f5b95b P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:13.154084 27685 log.cc:1199] T 9509364ba3ec4e9bad09db09406a978a P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/9509364ba3ec4e9bad09db09406a978a
I20250903 20:05:13.154083 27687 raft_consensus.cc:2241] T f3a33e90dc1f4540aee457de96f5b95b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:13.154299 27687 raft_consensus.cc:2270] T f3a33e90dc1f4540aee457de96f5b95b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:13.156121 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_83ea6dfcee144a4fbc86931567b676bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:13.156913 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83ea6dfcee144a4fbc86931567b676bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:13.157992 27687 ts_tablet_manager.cc:1905] T f3a33e90dc1f4540aee457de96f5b95b P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:13.160235 27685 ts_tablet_manager.cc:1939] T 9509364ba3ec4e9bad09db09406a978a P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:13.161038 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 9509364ba3ec4e9bad09db09406a978a (table default.Table_03ce0e151fcc4259ba52504c9c5ddfcd [id=f261e84232234244824d0232efbb4399]) successfully deleted
W20250903 20:05:13.162001 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet a0293a69f1c246d08d414e95729b4ab4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0293a69f1c246d08d414e95729b4ab4 already in progress: creating tablet
I20250903 20:05:13.163843 27066 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 8ce630060b924dff98083eaa0dde30c6
I20250903 20:05:13.164222 27573 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
W20250903 20:05:13.168617 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet eced730bdbc14a8c9f6b7b7e53435fd0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eced730bdbc14a8c9f6b7b7e53435fd0 already in progress: creating tablet
I20250903 20:05:13.169345 27695 tablet_bootstrap.cc:492] T c25aaa6ff310432bb27a0695394ec420 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
W20250903 20:05:13.169476 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 6788b1683f984949816edfc5b68de8db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6788b1683f984949816edfc5b68de8db already in progress: creating tablet
I20250903 20:05:13.169536 27695 ts_tablet_manager.cc:1397] T c25aaa6ff310432bb27a0695394ec420 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.001s
I20250903 20:05:13.170473 27695 raft_consensus.cc:357] T c25aaa6ff310432bb27a0695394ec420 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.170672 27695 raft_consensus.cc:383] T c25aaa6ff310432bb27a0695394ec420 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:13.171394 27059 catalog_manager.cc:5582] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a reported cstate change: term changed from 0 to 1, leader changed from <none> to 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65). New cstate: current_term: 1 leader_uuid: "89f8f0ee68b940db9ce250c0604f934a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: HEALTHY } } }
I20250903 20:05:13.171969 27059 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet e5136d5388bc4c2ba71e09be40e8f412 (table default.Table_9b80dba44fe8402999adc810b230d87a [id=790f57164c7d4183a3df17ba7f230d24]). Expected version 2 got 0
I20250903 20:05:13.172019 27359 tablet_service.cc:1468] Processing CreateTablet for tablet f0efbb7eefb84ab6bee5b9625b91e298 (DEFAULT_TABLE table=default.Table_781d39d6b06f49c28fa93c3692c7b5f7 [id=8d664216ea8e47fdaba51b2d6ca9d0a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.172489 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0efbb7eefb84ab6bee5b9625b91e298. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.172953 27501 tablet_service.cc:1468] Processing CreateTablet for tablet f0efbb7eefb84ab6bee5b9625b91e298 (DEFAULT_TABLE table=default.Table_781d39d6b06f49c28fa93c3692c7b5f7 [id=8d664216ea8e47fdaba51b2d6ca9d0a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.173329 27695 raft_consensus.cc:738] T c25aaa6ff310432bb27a0695394ec420 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:13.173403 27501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0efbb7eefb84ab6bee5b9625b91e298. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.173666 27695 consensus_queue.cc:260] T c25aaa6ff310432bb27a0695394ec420 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.174301 27213 tablet_service.cc:1468] Processing CreateTablet for tablet f0efbb7eefb84ab6bee5b9625b91e298 (DEFAULT_TABLE table=default.Table_781d39d6b06f49c28fa93c3692c7b5f7 [id=8d664216ea8e47fdaba51b2d6ca9d0a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.175405 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0efbb7eefb84ab6bee5b9625b91e298. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.175886 27502 tablet_service.cc:1468] Processing CreateTablet for tablet 6f8cfe53b91b41d7ba7f9074e1005af3 (DEFAULT_TABLE table=default.Table_e5def563dd414101b9ee8fbaf7156e29 [id=7e67c0d1d2f34172a987a8ea785c5a6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.176317 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f8cfe53b91b41d7ba7f9074e1005af3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.176595 27220 tablet_service.cc:1468] Processing CreateTablet for tablet 6f8cfe53b91b41d7ba7f9074e1005af3 (DEFAULT_TABLE table=default.Table_e5def563dd414101b9ee8fbaf7156e29 [id=7e67c0d1d2f34172a987a8ea785c5a6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.177007 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f8cfe53b91b41d7ba7f9074e1005af3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.177495 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 2d9c1642fec841d497ff3496dd4564db (DEFAULT_TABLE table=default.Table_923868d510f34889be990c9cd4180adc [id=f5cb786b03db4395a0f40b2956f5d331]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.177883 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d9c1642fec841d497ff3496dd4564db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.183385 27695 ts_tablet_manager.cc:1428] T c25aaa6ff310432bb27a0695394ec420 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250903 20:05:13.183761 27695 tablet_bootstrap.cc:492] T 8945085bdc174e6db22a5a9477810297 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:13.185125 27366 tablet_service.cc:1468] Processing CreateTablet for tablet 2d9c1642fec841d497ff3496dd4564db (DEFAULT_TABLE table=default.Table_923868d510f34889be990c9cd4180adc [id=f5cb786b03db4395a0f40b2956f5d331]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.185578 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d9c1642fec841d497ff3496dd4564db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.187824 27951 raft_consensus.cc:491] T c0343d2226e345669d42375de2545986 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:13.187986 27951 raft_consensus.cc:513] T c0343d2226e345669d42375de2545986 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.189494 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0343d2226e345669d42375de2545986" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:13.189772 27695 tablet_bootstrap.cc:654] T 8945085bdc174e6db22a5a9477810297 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:13.185091 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 6f8cfe53b91b41d7ba7f9074e1005af3 (DEFAULT_TABLE table=default.Table_e5def563dd414101b9ee8fbaf7156e29 [id=7e67c0d1d2f34172a987a8ea785c5a6a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:13.190399 27462 leader_election.cc:343] T c0343d2226e345669d42375de2545986 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:13.190781 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f8cfe53b91b41d7ba7f9074e1005af3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.190743 27982 raft_consensus.cc:491] T c0343d2226e345669d42375de2545986 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:13.190913 27982 raft_consensus.cc:513] T c0343d2226e345669d42375de2545986 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.191488 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0343d2226e345669d42375de2545986" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:13.191684 27249 raft_consensus.cc:2466] T c0343d2226e345669d42375de2545986 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:13.191720 27982 leader_election.cc:290] T c0343d2226e345669d42375de2545986 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:13.192092 27462 leader_election.cc:304] T c0343d2226e345669d42375de2545986 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:13.192428 27952 raft_consensus.cc:2802] T c0343d2226e345669d42375de2545986 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:13.192507 27952 raft_consensus.cc:491] T c0343d2226e345669d42375de2545986 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:13.192587 27952 raft_consensus.cc:3058] T c0343d2226e345669d42375de2545986 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:13.194432 27800 raft_consensus.cc:491] T d8ed7b3e292241cb9525745a797ff13a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:13.194645 27800 raft_consensus.cc:513] T d8ed7b3e292241cb9525745a797ff13a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.195274 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0343d2226e345669d42375de2545986" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:13.195441 27800 leader_election.cc:290] T d8ed7b3e292241cb9525745a797ff13a P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:13.197052 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ed7b3e292241cb9525745a797ff13a" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:13.156925 27063 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 99a21a3e49934a79a66700c0b0d86307 (table default.Table_7325977ff92e45bab0b59b79579ecae4 [id=428657b402d748c0a9e3942d13342b4e]). Expected version 1 got 0
I20250903 20:05:13.197567 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0343d2226e345669d42375de2545986" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:13.198138 27184 leader_election.cc:343] T c0343d2226e345669d42375de2545986 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:13.198263 27462 leader_election.cc:343] T d8ed7b3e292241cb9525745a797ff13a P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:13.198478 27462 leader_election.cc:343] T d8ed7b3e292241cb9525745a797ff13a P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: d8ed7b3e292241cb9525745a797ff13a
I20250903 20:05:13.198598 27462 leader_election.cc:304] T d8ed7b3e292241cb9525745a797ff13a P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:13.199209 27800 raft_consensus.cc:2747] T d8ed7b3e292241cb9525745a797ff13a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:13.201104 27952 raft_consensus.cc:513] T c0343d2226e345669d42375de2545986 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.201566 27525 raft_consensus.cc:2391] T c0343d2226e345669d42375de2545986 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89f8f0ee68b940db9ce250c0604f934a in current term 1: Already voted for candidate e72ef1dbcc5143cf9f04a808f101c80f in this term.
I20250903 20:05:13.202710 27693 tablet_bootstrap.cc:492] T 4bd2650e4d9e40f5ba5c09e367918224 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:13.202877 27693 ts_tablet_manager.cc:1397] T 4bd2650e4d9e40f5ba5c09e367918224 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.188s user 0.010s sys 0.022s
I20250903 20:05:13.203300 27183 leader_election.cc:304] T c0343d2226e345669d42375de2545986 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:13.203528 27952 leader_election.cc:290] T c0343d2226e345669d42375de2545986 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:13.203729 27693 raft_consensus.cc:357] T 4bd2650e4d9e40f5ba5c09e367918224 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.203905 27693 raft_consensus.cc:383] T 4bd2650e4d9e40f5ba5c09e367918224 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:13.203995 27693 raft_consensus.cc:738] T 4bd2650e4d9e40f5ba5c09e367918224 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:13.204020 27982 raft_consensus.cc:3058] T c0343d2226e345669d42375de2545986 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:13.204236 27693 consensus_queue.cc:260] T 4bd2650e4d9e40f5ba5c09e367918224 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.204749 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0343d2226e345669d42375de2545986" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
W20250903 20:05:13.205227 27462 leader_election.cc:343] T c0343d2226e345669d42375de2545986 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:13.205713 27693 ts_tablet_manager.cc:1428] T 4bd2650e4d9e40f5ba5c09e367918224 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:13.206001 27693 tablet_bootstrap.cc:492] T 9367b3a22f624c4087848099de32c080 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:13.207751 27501 tablet_service.cc:1515] Processing DeleteTablet for tablet 76742d0df44c43729197f7e09f96cad8 with delete_type TABLET_DATA_DELETED (replaced by tablet 0940159eaea740f99be2b67d5abacd2d) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:13.209484 27501 tablet_service.cc:1515] Processing DeleteTablet for tablet d8ed7b3e292241cb9525745a797ff13a with delete_type TABLET_DATA_DELETED (replaced by tablet 639ecb9caa4a423899ab8eb35bb716d7) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:13.210175 27065 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3185c25ec3784288bd52ccf75830589d" } modify_external_catalogs: true
I20250903 20:05:13.210436 27065 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3185c25ec3784288bd52ccf75830589d" } modify_external_catalogs: true
I20250903 20:05:13.211719 27982 raft_consensus.cc:2747] T c0343d2226e345669d42375de2545986 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:13.213408 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_43133b63f38043d6bb2757731065b56f" } modify_external_catalogs: true
I20250903 20:05:13.214054 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_43133b63f38043d6bb2757731065b56f" } modify_external_catalogs: true
I20250903 20:05:13.214154 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0343d2226e345669d42375de2545986" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:13.220742 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet 4bd2650e4d9e40f5ba5c09e367918224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:13.228515 27994 tablet.cc:1722] T 99a21a3e49934a79a66700c0b0d86307 P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:13.228787 27295 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
W20250903 20:05:13.232465 27987 log.cc:927] Time spent T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2: Append to log took a long time: real 0.143s user 0.001s sys 0.000s
I20250903 20:05:13.232620 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 31342583c026438cbbae14106e75bbe5
I20250903 20:05:13.233318 27062 catalog_manager.cc:6652] ReplaceTablet: tablet 31342583c026438cbbae14106e75bbe5 deleted and replaced by tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8
I20250903 20:05:13.234153 27059 master_service.cc:948] ReplaceTablet: received request to replace tablet c9e90abef9814338b70a3550ab5686e0 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:13.234866 27951 leader_election.cc:290] T c0343d2226e345669d42375de2545986 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:13.235251 27213 tablet_service.cc:1515] Processing DeleteTablet for tablet 31342583c026438cbbae14106e75bbe5 with delete_type TABLET_DATA_DELETED (replaced by tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:13.235714 27501 tablet_service.cc:1515] Processing DeleteTablet for tablet 31342583c026438cbbae14106e75bbe5 with delete_type TABLET_DATA_DELETED (replaced by tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:13.236483 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet 31342583c026438cbbae14106e75bbe5 with delete_type TABLET_DATA_DELETED (replaced by tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:13.237278 27062 master_service.cc:948] ReplaceTablet: received request to replace tablet 4268316c28e8471a9f28807d20dbb395 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:13.194437 27684 ts_tablet_manager.cc:1918] T 925c0ab4873041da82702cd9377ec7bc P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:13.239679 27684 log.cc:1199] T 925c0ab4873041da82702cd9377ec7bc P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/925c0ab4873041da82702cd9377ec7bc
W20250903 20:05:13.241483 27462 consensus_peers.cc:489] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:13.242704 27249 raft_consensus.cc:1273] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Refusing update from remote peer e72ef1dbcc5143cf9f04a808f101c80f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 20:05:13.244503 27213 tablet_service.cc:1468] Processing CreateTablet for tablet 2d9c1642fec841d497ff3496dd4564db (DEFAULT_TABLE table=default.Table_923868d510f34889be990c9cd4180adc [id=f5cb786b03db4395a0f40b2956f5d331]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.244565 27951 consensus_queue.cc:1035] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:13.245229 27684 ts_tablet_manager.cc:1939] T 925c0ab4873041da82702cd9377ec7bc P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:13.245481 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d9c1642fec841d497ff3496dd4564db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.247125 27357 tablet_service.cc:1468] Processing CreateTablet for tablet 1bad3c509460422396a17a3c4ae3d1e8 (DEFAULT_TABLE table=default.Table_60d0c49d795c4cad954d2b7327485eb9 [id=41f9673d6e604f45a8def10948ce21ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.247639 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bad3c509460422396a17a3c4ae3d1e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.249408 27693 tablet_bootstrap.cc:654] T 9367b3a22f624c4087848099de32c080 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:13.252171 27501 tablet_service.cc:1468] Processing CreateTablet for tablet 1bad3c509460422396a17a3c4ae3d1e8 (DEFAULT_TABLE table=default.Table_60d0c49d795c4cad954d2b7327485eb9 [id=41f9673d6e604f45a8def10948ce21ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.252671 27501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bad3c509460422396a17a3c4ae3d1e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.252918 27692 tablet_bootstrap.cc:492] T 8945085bdc174e6db22a5a9477810297 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:13.253098 27692 ts_tablet_manager.cc:1397] T 8945085bdc174e6db22a5a9477810297 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.183s user 0.007s sys 0.003s
I20250903 20:05:13.254009 27692 raft_consensus.cc:357] T 8945085bdc174e6db22a5a9477810297 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.254261 27692 raft_consensus.cc:383] T 8945085bdc174e6db22a5a9477810297 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:13.254351 27692 raft_consensus.cc:738] T 8945085bdc174e6db22a5a9477810297 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:13.255345 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 925c0ab4873041da82702cd9377ec7bc (table default.Table_1a5c8c3256c24c7f8ef8597efa53dde5 [id=7caef592f80a4d6cb6df4e2c3971de49]) successfully deleted
I20250903 20:05:13.257627 27684 tablet_replica.cc:331] T 46b5d491888147bbb101963641fd05f8 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:13.258328 27951 raft_consensus.cc:491] T 46b5d491888147bbb101963641fd05f8 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:13.258977 27951 raft_consensus.cc:513] T 46b5d491888147bbb101963641fd05f8 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.257042 27364 tablet_service.cc:1468] Processing CreateTablet for tablet 59a1dd48d76241d7ba817070fbffd853 (DEFAULT_TABLE table=default.Table_d342b3c06e1c4b98bc7f07e88844cdc9 [id=34db41bcdc604ee08da11d0e559b4d35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.261009 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59a1dd48d76241d7ba817070fbffd853. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.261227 27951 leader_election.cc:290] T 46b5d491888147bbb101963641fd05f8 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:13.262995 27432 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
I20250903 20:05:13.263391 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 7e2f98f760cf4df4bc2a10530e2157c6
I20250903 20:05:13.260111 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46b5d491888147bbb101963641fd05f8" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:13.266700 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e2f98f760cf4df4bc2a10530e2157c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:13.267045 27063 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 76320903ae124ae29120cfc8e8730296
W20250903 20:05:13.267004 27462 leader_election.cc:343] T 46b5d491888147bbb101963641fd05f8 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:13.268599 27684 raft_consensus.cc:2241] T 46b5d491888147bbb101963641fd05f8 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:13.268755 27502 tablet_service.cc:1468] Processing CreateTablet for tablet 59a1dd48d76241d7ba817070fbffd853 (DEFAULT_TABLE table=default.Table_d342b3c06e1c4b98bc7f07e88844cdc9 [id=34db41bcdc604ee08da11d0e559b4d35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.269523 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59a1dd48d76241d7ba817070fbffd853. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.269863 27684 raft_consensus.cc:2270] T 46b5d491888147bbb101963641fd05f8 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:13.270958 27364 tablet_service.cc:1468] Processing CreateTablet for tablet 53881ad30bc843839008ace4012a3f70 (DEFAULT_TABLE table=default.Table_1480d4baa8e149bb80b62f09d4cf3f90 [id=cedbbfab7d0e41369cafd9ce516f2dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.271400 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53881ad30bc843839008ace4012a3f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.276469 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet c9e90abef9814338b70a3550ab5686e0
I20250903 20:05:13.277012 27059 catalog_manager.cc:6652] ReplaceTablet: tablet c9e90abef9814338b70a3550ab5686e0 deleted and replaced by tablet 607c9a9cbc5540e5bd0fa436fcba564c
I20250903 20:05:13.277012 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 4268316c28e8471a9f28807d20dbb395
I20250903 20:05:13.277772 27062 catalog_manager.cc:6652] ReplaceTablet: tablet 4268316c28e8471a9f28807d20dbb395 deleted and replaced by tablet a9f43781299e47a8aecc37a0b73d7a64
I20250903 20:05:13.278173 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet 4268316c28e8471a9f28807d20dbb395 with delete_type TABLET_DATA_DELETED (replaced by tablet a9f43781299e47a8aecc37a0b73d7a64) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:13.280598 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet c9e90abef9814338b70a3550ab5686e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 607c9a9cbc5540e5bd0fa436fcba564c) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:13.285115 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_187e391a6d6d4d0d96d4bb8a0581face"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:13.285375 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_27706ce3e18b47078bb7d3071c12a1a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:13.285833 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27706ce3e18b47078bb7d3071c12a1a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:13.286398 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_44d61e5bea3c4bbab999158701853479" } new_table_name: "default.Table_43ea308240ac4628a72434608a22b975" modify_external_catalogs: true
W20250903 20:05:13.285833 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_187e391a6d6d4d0d96d4bb8a0581face in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:13.290181 27065 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0064d6a69e9a474bb17ad72deb3fbbc8" } modify_external_catalogs: true
I20250903 20:05:13.290390 27065 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0064d6a69e9a474bb17ad72deb3fbbc8" } modify_external_catalogs: true
I20250903 20:05:13.286845 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 1bad3c509460422396a17a3c4ae3d1e8 (DEFAULT_TABLE table=default.Table_60d0c49d795c4cad954d2b7327485eb9 [id=41f9673d6e604f45a8def10948ce21ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.280449 27685 tablet_replica.cc:331] T 710e3bc3c4194efbbeef6bc76ee64f24 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:13.278306 27501 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e2f98f760cf4df4bc2a10530e2157c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:13.278898 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet c9e90abef9814338b70a3550ab5686e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 607c9a9cbc5540e5bd0fa436fcba564c) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:13.279683 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 4268316c28e8471a9f28807d20dbb395 with delete_type TABLET_DATA_DELETED (replaced by tablet a9f43781299e47a8aecc37a0b73d7a64) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:13.293009 27995 tablet.cc:1722] T 99a21a3e49934a79a66700c0b0d86307 P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:13.295631 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_35d35c0c3a6048bc9d7def161d9e16db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:13.296273 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35d35c0c3a6048bc9d7def161d9e16db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:13.299017 27064 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1840897b7e7f4f1e8052d16ca5e77a4f" } modify_external_catalogs: true
I20250903 20:05:13.299291 27064 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1840897b7e7f4f1e8052d16ca5e77a4f" } modify_external_catalogs: true
I20250903 20:05:13.299160 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_df2f2840b3c7420bac713679e7eb971d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:13.300525 27685 raft_consensus.cc:2241] T 710e3bc3c4194efbbeef6bc76ee64f24 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:13.300762 27685 raft_consensus.cc:2270] T 710e3bc3c4194efbbeef6bc76ee64f24 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:13.300902 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df2f2840b3c7420bac713679e7eb971d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:13.301914 27502 tablet_service.cc:1468] Processing CreateTablet for tablet 53881ad30bc843839008ace4012a3f70 (DEFAULT_TABLE table=default.Table_1480d4baa8e149bb80b62f09d4cf3f90 [id=cedbbfab7d0e41369cafd9ce516f2dba]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:13.303058 27462 leader_election.cc:343] T 46b5d491888147bbb101963641fd05f8 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 46b5d491888147bbb101963641fd05f8
I20250903 20:05:13.303248 27462 leader_election.cc:304] T 46b5d491888147bbb101963641fd05f8 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:13.303493 27684 ts_tablet_manager.cc:1905] T 46b5d491888147bbb101963641fd05f8 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:13.303536 27462 raft_consensus.cc:2684] T 46b5d491888147bbb101963641fd05f8 P e72ef1dbcc5143cf9f04a808f101c80f: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250903 20:05:13.303107 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53881ad30bc843839008ace4012a3f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.293756 27692 consensus_queue.cc:260] T 8945085bdc174e6db22a5a9477810297 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.304512 27224 tablet_service.cc:1515] Processing DeleteTablet for tablet c9e90abef9814338b70a3550ab5686e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 607c9a9cbc5540e5bd0fa436fcba564c) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:13.304826 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e2f98f760cf4df4bc2a10530e2157c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:13.305634 27573 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
I20250903 20:05:13.306784 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bad3c509460422396a17a3c4ae3d1e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.307839 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 59a1dd48d76241d7ba817070fbffd853 (DEFAULT_TABLE table=default.Table_d342b3c06e1c4b98bc7f07e88844cdc9 [id=34db41bcdc604ee08da11d0e559b4d35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.308224 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59a1dd48d76241d7ba817070fbffd853. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.308941 27248 raft_consensus.cc:2466] T c0343d2226e345669d42375de2545986 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:13.310468 27685 ts_tablet_manager.cc:1905] T 710e3bc3c4194efbbeef6bc76ee64f24 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:13.310858 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet 4268316c28e8471a9f28807d20dbb395 with delete_type TABLET_DATA_DELETED (replaced by tablet a9f43781299e47a8aecc37a0b73d7a64) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:13.311470 27224 tablet_service.cc:1468] Processing CreateTablet for tablet 53881ad30bc843839008ace4012a3f70 (DEFAULT_TABLE table=default.Table_1480d4baa8e149bb80b62f09d4cf3f90 [id=cedbbfab7d0e41369cafd9ce516f2dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.315450 27462 leader_election.cc:304] T c0343d2226e345669d42375de2545986 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:13.316238 27800 raft_consensus.cc:2802] T c0343d2226e345669d42375de2545986 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:13.316426 27800 raft_consensus.cc:695] T c0343d2226e345669d42375de2545986 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:13.316808 27800 consensus_queue.cc:237] T c0343d2226e345669d42375de2545986 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.317710 27994 tablet.cc:1722] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:13.320860 27973 mvcc.cc:204] Tried to move back new op lower bound from 7196384924628967424 to 7196384923190206464. Current Snapshot: MvccSnapshot[applied={T|T < 7196384924628967424}]
I20250903 20:05:13.327361 27693 tablet_bootstrap.cc:492] T 9367b3a22f624c4087848099de32c080 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:13.327831 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53881ad30bc843839008ace4012a3f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.330340 27693 ts_tablet_manager.cc:1397] T 9367b3a22f624c4087848099de32c080 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.124s user 0.004s sys 0.000s
I20250903 20:05:13.332134 27693 raft_consensus.cc:357] T 9367b3a22f624c4087848099de32c080 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.338339 27840 raft_consensus.cc:491] T e5136d5388bc4c2ba71e09be40e8f412 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:13.338351 27995 tablet.cc:1722] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:13.338601 27840 raft_consensus.cc:513] T e5136d5388bc4c2ba71e09be40e8f412 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:13.341426 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5136d5388bc4c2ba71e09be40e8f412" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:13.340286 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5136d5388bc4c2ba71e09be40e8f412" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:13.344236 27693 raft_consensus.cc:383] T 9367b3a22f624c4087848099de32c080 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250903 20:05:13.347194 27183 leader_election.cc:343] T e5136d5388bc4c2ba71e09be40e8f412 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:13.352531 27295 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
I20250903 20:05:13.353178 27840 leader_election.cc:290] T e5136d5388bc4c2ba71e09be40e8f412 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
W20250903 20:05:13.353837 27184 leader_election.cc:343] T e5136d5388bc4c2ba71e09be40e8f412 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:13.354071 27184 leader_election.cc:304] T e5136d5388bc4c2ba71e09be40e8f412 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:13.354065 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b86a1d18f92c4844997ebd323da582b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:13.354717 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b86a1d18f92c4844997ebd323da582b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:13.355325 27982 raft_consensus.cc:2747] T e5136d5388bc4c2ba71e09be40e8f412 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:13.356613 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e3d549d1397a4167a21010a1041c6ab2" } new_table_name: "default.Table_a71ff73faf3a43dbb95677820a0561b6" modify_external_catalogs: true
I20250903 20:05:13.356719 27220 tablet_service.cc:1468] Processing CreateTablet for tablet a4f3230f50ec4c95b528fead8c191224 (DEFAULT_TABLE table=default.Table_be001b7aec594b93a1b45e9313b772ca [id=7a045d80e0eb42cb95fdca51d339da0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.362057 27693 raft_consensus.cc:738] T 9367b3a22f624c4087848099de32c080 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:13.362578 27693 consensus_queue.cc:260] T 9367b3a22f624c4087848099de32c080 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.362939 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4f3230f50ec4c95b528fead8c191224. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.364071 27432 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
I20250903 20:05:13.364538 27693 ts_tablet_manager.cc:1428] T 9367b3a22f624c4087848099de32c080 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.033s user 0.002s sys 0.000s
I20250903 20:05:13.364889 27693 tablet_bootstrap.cc:492] T 4692dd93b76c42c4883c7e22c6032250 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:13.365857 27226 tablet_service.cc:1468] Processing CreateTablet for tablet 1fbe54dc40dd47daa109ceeb07546b0a (DEFAULT_TABLE table=default.Table_03ce0e151fcc4259ba52504c9c5ddfcd [id=f261e84232234244824d0232efbb4399]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.366333 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 17e0864a32d5452d84ac470b55fbaaf4 (DEFAULT_TABLE table=default.Table_c21ce98311d847f2aaae2ea2c0549856 [id=263d1af690fb4868a4bd92ae090dfd0a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:13.366624 27059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 673c45f3f2a14da9bd306011d396eaca has no consensus state
I20250903 20:05:13.366806 27693 tablet_bootstrap.cc:654] T 4692dd93b76c42c4883c7e22c6032250 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:13.367059 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f83028b9d34949b982a5bf7ffa812c27" } new_table_name: "default.Table_c5256f043b61433b8debe9e13625bd0d" modify_external_catalogs: true
I20250903 20:05:13.367465 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fbe54dc40dd47daa109ceeb07546b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.368829 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_0bbccf098ebf46ae8d825952b89a2357"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:13.368994 27949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35bdb0090d8747059b5ee103e786e979. Latest consensus state: current_term: 3 leader_uuid: "35bdb0090d8747059b5ee103e786e979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:13.369238 27949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [sys.catalog]: This master's current role is: FOLLOWER
W20250903 20:05:13.369506 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0bbccf098ebf46ae8d825952b89a2357 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:13.370275 27993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "35bdb0090d8747059b5ee103e786e979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:13.370575 27993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 20:05:13.371517 27065 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 7f24a917bbbc45de84399cd9fea5d8a8
I20250903 20:05:13.371940 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_c967f8b0d1314f589eae529cc492493b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:13.372673 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c967f8b0d1314f589eae529cc492493b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:13.374259 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17e0864a32d5452d84ac470b55fbaaf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.350248 28003 tablet.cc:1722] T 99a21a3e49934a79a66700c0b0d86307 P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:13.355018 27357 tablet_service.cc:1468] Processing CreateTablet for tablet a4f3230f50ec4c95b528fead8c191224 (DEFAULT_TABLE table=default.Table_be001b7aec594b93a1b45e9313b772ca [id=7a045d80e0eb42cb95fdca51d339da0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.376513 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4f3230f50ec4c95b528fead8c191224. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.357362 27366 tablet_service.cc:1468] Processing CreateTablet for tablet 17e0864a32d5452d84ac470b55fbaaf4 (DEFAULT_TABLE table=default.Table_c21ce98311d847f2aaae2ea2c0549856 [id=263d1af690fb4868a4bd92ae090dfd0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.361374 27687 ts_tablet_manager.cc:1918] T f3a33e90dc1f4540aee457de96f5b95b P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:13.378389 27687 log.cc:1199] T f3a33e90dc1f4540aee457de96f5b95b P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/f3a33e90dc1f4540aee457de96f5b95b
I20250903 20:05:13.378535 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17e0864a32d5452d84ac470b55fbaaf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.378801 27687 ts_tablet_manager.cc:1939] T f3a33e90dc1f4540aee457de96f5b95b P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:13.379859 27687 tablet_replica.cc:331] T 7e0810c96eb24b8ab4278038801be245 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:13.380067 27687 raft_consensus.cc:2241] T 7e0810c96eb24b8ab4278038801be245 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:13.380271 27687 raft_consensus.cc:2270] T 7e0810c96eb24b8ab4278038801be245 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:13.381052 27687 ts_tablet_manager.cc:1905] T 7e0810c96eb24b8ab4278038801be245 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:13.381760 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_362ac8cb413e426ebf154eae4b6b5c7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:13.382433 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_362ac8cb413e426ebf154eae4b6b5c7e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:13.385193 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet f3a33e90dc1f4540aee457de96f5b95b (table default.Table_f6203681100e4faeb016f0e6afc283f7 [id=91268ab201a4423faedd970b9a296152]) successfully deleted
I20250903 20:05:13.387385 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_983904dd0bb4486493150ce49e43a229" } new_table_name: "default.Table_143fea9b5e3d4acbb92adc441aeede47" modify_external_catalogs: true
I20250903 20:05:13.387986 27687 ts_tablet_manager.cc:1918] T 7e0810c96eb24b8ab4278038801be245 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:13.388123 27687 log.cc:1199] T 7e0810c96eb24b8ab4278038801be245 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/7e0810c96eb24b8ab4278038801be245
I20250903 20:05:13.388512 27687 ts_tablet_manager.cc:1939] T 7e0810c96eb24b8ab4278038801be245 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:13.389444 27687 tablet_replica.cc:331] T e1a49d0599654552bf1c72857031699b P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:13.389660 27687 raft_consensus.cc:2241] T e1a49d0599654552bf1c72857031699b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:13.389829 27687 raft_consensus.cc:2270] T e1a49d0599654552bf1c72857031699b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:13.390348 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 7e0810c96eb24b8ab4278038801be245 (table default.Table_9b80dba44fe8402999adc810b230d87a [id=790f57164c7d4183a3df17ba7f230d24]) successfully deleted
I20250903 20:05:13.391345 27695 tablet_bootstrap.cc:492] T 8945085bdc174e6db22a5a9477810297 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:13.391527 27993 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 35bdb0090d8747059b5ee103e786e979)
I20250903 20:05:13.391811 27993 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } }
I20250903 20:05:13.392182 27695 ts_tablet_manager.cc:1397] T 8945085bdc174e6db22a5a9477810297 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.208s user 0.054s sys 0.027s
I20250903 20:05:13.361740 27364 tablet_service.cc:1468] Processing CreateTablet for tablet 1fbe54dc40dd47daa109ceeb07546b0a (DEFAULT_TABLE table=default.Table_03ce0e151fcc4259ba52504c9c5ddfcd [id=f261e84232234244824d0232efbb4399]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.350147 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 7e2f98f760cf4df4bc2a10530e2157c6 (DEFAULT_TABLE table=default.Table_3185c25ec3784288bd52ccf75830589d [id=4d1a27b6dfb74dd69ba27d0f1e767dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.393361 27695 raft_consensus.cc:357] T 8945085bdc174e6db22a5a9477810297 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.393704 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e2f98f760cf4df4bc2a10530e2157c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.394584 27432 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
I20250903 20:05:13.400138 27695 raft_consensus.cc:383] T 8945085bdc174e6db22a5a9477810297 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:13.400283 27695 raft_consensus.cc:738] T 8945085bdc174e6db22a5a9477810297 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:13.401089 27993 leader_election.cc:290] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 35dd69f3fa7a4fd8843b97f8a5d68296 (127.26.65.125:44059), 35bdb0090d8747059b5ee103e786e979 (127.26.65.124:40911)
I20250903 20:05:13.403841 27695 consensus_queue.cc:260] T 8945085bdc174e6db22a5a9477810297 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.406513 27695 ts_tablet_manager.cc:1428] T 8945085bdc174e6db22a5a9477810297 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250903 20:05:13.406929 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 17e0864a32d5452d84ac470b55fbaaf4 (DEFAULT_TABLE table=default.Table_c21ce98311d847f2aaae2ea2c0549856 [id=263d1af690fb4868a4bd92ae090dfd0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.407490 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17e0864a32d5452d84ac470b55fbaaf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.408118 27695 tablet_bootstrap.cc:492] T 925c0ab4873041da82702cd9377ec7bc P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:13.409325 27687 ts_tablet_manager.cc:1905] T e1a49d0599654552bf1c72857031699b P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:13.412930 27695 tablet_bootstrap.cc:654] T 925c0ab4873041da82702cd9377ec7bc P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:13.414592 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 8945085bdc174e6db22a5a9477810297 with delete_type TABLET_DATA_DELETED (replaced by tablet f71d3dd9af2543299296f21381d1d563) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:13.416587 27925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "35bdb0090d8747059b5ee103e786e979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:13.416812 27925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [sys.catalog]: This master's current role is: LEADER
I20250903 20:05:13.418088 27687 ts_tablet_manager.cc:1918] T e1a49d0599654552bf1c72857031699b P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:13.418263 27687 log.cc:1199] T e1a49d0599654552bf1c72857031699b P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/e1a49d0599654552bf1c72857031699b
I20250903 20:05:13.418583 27687 ts_tablet_manager.cc:1939] T e1a49d0599654552bf1c72857031699b P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:13.419202 27685 ts_tablet_manager.cc:1918] T 710e3bc3c4194efbbeef6bc76ee64f24 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:13.424086 27498 tablet_service.cc:1468] Processing CreateTablet for tablet 1fbe54dc40dd47daa109ceeb07546b0a (DEFAULT_TABLE table=default.Table_03ce0e151fcc4259ba52504c9c5ddfcd [id=f261e84232234244824d0232efbb4399]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.424314 27692 ts_tablet_manager.cc:1428] T 8945085bdc174e6db22a5a9477810297 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.097s user 0.000s sys 0.003s
I20250903 20:05:13.424610 27498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fbe54dc40dd47daa109ceeb07546b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.427459 27501 tablet_service.cc:1468] Processing CreateTablet for tablet a4f3230f50ec4c95b528fead8c191224 (DEFAULT_TABLE table=default.Table_be001b7aec594b93a1b45e9313b772ca [id=7a045d80e0eb42cb95fdca51d339da0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.432713 27754 raft_consensus.cc:491] T eaa11d17303142ff935705c16f0b9690 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:13.432926 27754 raft_consensus.cc:513] T eaa11d17303142ff935705c16f0b9690 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.433485 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 7e2f98f760cf4df4bc2a10530e2157c6 (DEFAULT_TABLE table=default.Table_3185c25ec3784288bd52ccf75830589d [id=4d1a27b6dfb74dd69ba27d0f1e767dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.433660 27754 leader_election.cc:290] T eaa11d17303142ff935705c16f0b9690 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:13.433971 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e2f98f760cf4df4bc2a10530e2157c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.434609 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eaa11d17303142ff935705c16f0b9690" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:13.444891 27685 log.cc:1199] T 710e3bc3c4194efbbeef6bc76ee64f24 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/710e3bc3c4194efbbeef6bc76ee64f24
I20250903 20:05:13.445842 27502 tablet_service.cc:1468] Processing CreateTablet for tablet 7e2f98f760cf4df4bc2a10530e2157c6 (DEFAULT_TABLE table=default.Table_3185c25ec3784288bd52ccf75830589d [id=4d1a27b6dfb74dd69ba27d0f1e767dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.446408 27501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4f3230f50ec4c95b528fead8c191224. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.446784 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e2f98f760cf4df4bc2a10530e2157c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.456004 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eaa11d17303142ff935705c16f0b9690" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:13.456948 27320 leader_election.cc:304] T eaa11d17303142ff935705c16f0b9690 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:13.457507 27754 raft_consensus.cc:2747] T eaa11d17303142ff935705c16f0b9690 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:13.458058 27692 tablet_bootstrap.cc:492] T c25aaa6ff310432bb27a0695394ec420 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:13.459991 27692 tablet_bootstrap.cc:654] T c25aaa6ff310432bb27a0695394ec420 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:13.469827 27685 ts_tablet_manager.cc:1939] T 710e3bc3c4194efbbeef6bc76ee64f24 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:13.473335 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 710e3bc3c4194efbbeef6bc76ee64f24 (table default.Table_452fdb47678d4520828b9a3db6ddb237 [id=d945a830c86244bca66cb2652e64966d]) successfully deleted
W20250903 20:05:13.473765 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 31342583c026438cbbae14106e75bbe5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31342583c026438cbbae14106e75bbe5 already in progress: creating tablet
W20250903 20:05:13.474099 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet c9e90abef9814338b70a3550ab5686e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9e90abef9814338b70a3550ab5686e0 already in progress: creating tablet
W20250903 20:05:13.474510 27052 catalog_manager.cc:4895] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 7e2f98f760cf4df4bc2a10530e2157c6 because the tablet was not found. No further retry: Not found: Tablet not found: 7e2f98f760cf4df4bc2a10530e2157c6
W20250903 20:05:13.474789 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 4268316c28e8471a9f28807d20dbb395 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4268316c28e8471a9f28807d20dbb395 already in progress: creating tablet
I20250903 20:05:13.478634 28005 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250903 20:05:13.492925 27840 raft_consensus.cc:491] T 46e3a65118b744298b5536f88c22f1db P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:13.493302 27840 raft_consensus.cc:513] T 46e3a65118b744298b5536f88c22f1db P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.494292 27840 leader_election.cc:290] T 46e3a65118b744298b5536f88c22f1db P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:13.495129 27573 heartbeater.cc:499] Master 127.26.65.126:39015 was elected leader, sending a full tablet report...
I20250903 20:05:13.498304 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46e3a65118b744298b5536f88c22f1db" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:13.499109 27183 leader_election.cc:343] T 46e3a65118b744298b5536f88c22f1db P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:13.499214 27687 tablet_replica.cc:331] T 4bd2650e4d9e40f5ba5c09e367918224 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:13.499393 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet e1a49d0599654552bf1c72857031699b (table default.Table_f9e41274ca534e46b7906480d7fc32fe [id=febabbd421d241948dd40f5480c052bb]) successfully deleted
I20250903 20:05:13.499432 27687 raft_consensus.cc:2241] T 4bd2650e4d9e40f5ba5c09e367918224 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:13.499650 27687 raft_consensus.cc:2270] T 4bd2650e4d9e40f5ba5c09e367918224 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:13.501502 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46e3a65118b744298b5536f88c22f1db" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:13.507499 27687 ts_tablet_manager.cc:1905] T 4bd2650e4d9e40f5ba5c09e367918224 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:13.510366 27184 leader_election.cc:343] T 46e3a65118b744298b5536f88c22f1db P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:13.510637 27184 leader_election.cc:304] T 46e3a65118b744298b5536f88c22f1db P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:13.510987 27840 raft_consensus.cc:2747] T 46e3a65118b744298b5536f88c22f1db P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:13.514590 27687 ts_tablet_manager.cc:1918] T 4bd2650e4d9e40f5ba5c09e367918224 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:13.514747 27687 log.cc:1199] T 4bd2650e4d9e40f5ba5c09e367918224 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/4bd2650e4d9e40f5ba5c09e367918224
I20250903 20:05:13.515182 27687 ts_tablet_manager.cc:1939] T 4bd2650e4d9e40f5ba5c09e367918224 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:13.515182 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fbe54dc40dd47daa109ceeb07546b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.517012 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 4bd2650e4d9e40f5ba5c09e367918224 (table default.Table_2ace21da933742e69b8ec85acbd919c2 [id=dcc845689a884dc69c7217043de59d32]) successfully deleted
W20250903 20:05:13.517432 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 31342583c026438cbbae14106e75bbe5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31342583c026438cbbae14106e75bbe5 already in progress: creating tablet
W20250903 20:05:13.517732 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 7e2f98f760cf4df4bc2a10530e2157c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e2f98f760cf4df4bc2a10530e2157c6 already in progress: creating tablet
W20250903 20:05:13.518005 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 4268316c28e8471a9f28807d20dbb395 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4268316c28e8471a9f28807d20dbb395 already in progress: creating tablet
W20250903 20:05:13.518325 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet c9e90abef9814338b70a3550ab5686e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9e90abef9814338b70a3550ab5686e0 already in progress: creating tablet
I20250903 20:05:13.519959 27695 tablet_bootstrap.cc:492] T 925c0ab4873041da82702cd9377ec7bc P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:13.520133 27695 ts_tablet_manager.cc:1397] T 925c0ab4873041da82702cd9377ec7bc P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.113s user 0.000s sys 0.011s
I20250903 20:05:13.520962 27695 raft_consensus.cc:357] T 925c0ab4873041da82702cd9377ec7bc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:13.521173 27695 raft_consensus.cc:383] T 925c0ab4873041da82702cd9377ec7bc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:13.521288 27695 raft_consensus.cc:738] T 925c0ab4873041da82702cd9377ec7bc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:13.521524 27695 consensus_queue.cc:260] T 925c0ab4873041da82702cd9377ec7bc P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:13.522169 27695 ts_tablet_manager.cc:1428] T 925c0ab4873041da82702cd9377ec7bc P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250903 20:05:13.522258 27851 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" instance_seqno: 1756929907080274) as {username='slave'} at 127.26.65.66:45443; Asking this server to re-register.
I20250903 20:05:13.522507 27695 tablet_bootstrap.cc:492] T 4268316c28e8471a9f28807d20dbb395 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:13.524045 27695 tablet_bootstrap.cc:654] T 4268316c28e8471a9f28807d20dbb395 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:13.526386 27060 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 9367b3a22f624c4087848099de32c080 (table default.Table_08918e8b515e4487bf4aee0a33234bd5 [id=891b259c1c6b44e2a82dc982c24e6cb6]). Expected version 1 got 0
I20250903 20:05:13.527379 27695 tablet_bootstrap.cc:492] T 4268316c28e8471a9f28807d20dbb395 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:13.527534 27695 ts_tablet_manager.cc:1397] T 4268316c28e8471a9f28807d20dbb395 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250903 20:05:13.528594 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet fe902f1b58924be1ac3a5585413f7789
I20250903 20:05:13.529302 27432 heartbeater.cc:461] Registering TS with master...
I20250903 20:05:13.529251 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_a6608fc0162446a480d8e8ecb07db28f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:13.529832 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6608fc0162446a480d8e8ecb07db28f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:13.530596 27695 raft_consensus.cc:357] T 4268316c28e8471a9f28807d20dbb395 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.530791 27695 raft_consensus.cc:383] T 4268316c28e8471a9f28807d20dbb395 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:13.530866 27695 raft_consensus.cc:738] T 4268316c28e8471a9f28807d20dbb395 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
W20250903 20:05:13.531297 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d859c0222c3647dab3dddd105e4cf0fc consensus state has no leader
I20250903 20:05:13.531328 27695 consensus_queue.cc:260] T 4268316c28e8471a9f28807d20dbb395 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.531634 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_51d81329a9c84f5bb4a0973d5410d115" } new_table_name: "default.Table_7f07933eeaf8443384edb1e59396e36e" modify_external_catalogs: true
I20250903 20:05:13.531912 27695 ts_tablet_manager.cc:1428] T 4268316c28e8471a9f28807d20dbb395 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20250903 20:05:13.531941 27851 ts_manager.cc:194] Registered new tserver with Master: 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:13.532232 27695 tablet_bootstrap.cc:492] T f7ab722c37264973b204cd4f1f95052d P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
W20250903 20:05:13.532830 27059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e8bc99de4ffe43d58351d170b7c61d8a consensus state has no leader
I20250903 20:05:13.533169 27059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_9c5c22a18c1d4913abdfab7d6a589058" } modify_external_catalogs: true
I20250903 20:05:13.533313 27059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_9c5c22a18c1d4913abdfab7d6a589058" } modify_external_catalogs: true
I20250903 20:05:13.533924 27695 tablet_bootstrap.cc:654] T f7ab722c37264973b204cd4f1f95052d P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:13.533998 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_3cf8165949c5457cb1fa8cd412628763"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:13.534536 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3cf8165949c5457cb1fa8cd412628763 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:13.539187 27061 master_service.cc:948] ReplaceTablet: received request to replace tablet 925c0ab4873041da82702cd9377ec7bc from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:13.539355 27695 tablet_bootstrap.cc:492] T f7ab722c37264973b204cd4f1f95052d P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:13.539525 27695 ts_tablet_manager.cc:1397] T f7ab722c37264973b204cd4f1f95052d P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20250903 20:05:13.539767 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_509afdab983d4eed97a5d124ff19ab3b" } modify_external_catalogs: true
I20250903 20:05:13.539947 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_509afdab983d4eed97a5d124ff19ab3b" } modify_external_catalogs: true
I20250903 20:05:13.540867 27695 raft_consensus.cc:357] T f7ab722c37264973b204cd4f1f95052d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.541059 27695 raft_consensus.cc:383] T f7ab722c37264973b204cd4f1f95052d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:13.541150 27695 raft_consensus.cc:738] T f7ab722c37264973b204cd4f1f95052d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:13.542515 27065 master_service.cc:948] ReplaceTablet: received request to replace tablet c25aaa6ff310432bb27a0695394ec420 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:13.543061 27065 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_8b0346420a89417b9ae2005be49e9f3a" } modify_external_catalogs: true
I20250903 20:05:13.543215 27065 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_8b0346420a89417b9ae2005be49e9f3a" } modify_external_catalogs: true
I20250903 20:05:13.541373 27695 consensus_queue.cc:260] T f7ab722c37264973b204cd4f1f95052d P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.546727 27695 ts_tablet_manager.cc:1428] T f7ab722c37264973b204cd4f1f95052d P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20250903 20:05:13.546705 27684 ts_tablet_manager.cc:1918] T 46b5d491888147bbb101963641fd05f8 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:13.546857 27684 log.cc:1199] T 46b5d491888147bbb101963641fd05f8 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/46b5d491888147bbb101963641fd05f8
I20250903 20:05:13.547060 27695 tablet_bootstrap.cc:492] T e20fcd28c3ae4355a033d2b8429f84e2 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:13.547235 27684 ts_tablet_manager.cc:1939] T 46b5d491888147bbb101963641fd05f8 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:13.548247 27684 tablet_replica.cc:331] T 76742d0df44c43729197f7e09f96cad8 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:13.548425 27684 raft_consensus.cc:2241] T 76742d0df44c43729197f7e09f96cad8 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:13.548784 27695 tablet_bootstrap.cc:654] T e20fcd28c3ae4355a033d2b8429f84e2 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:13.554302 27066 master_service.cc:948] ReplaceTablet: received request to replace tablet c9c870fb14e3469fb2979d8462b49121 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:13.555474 27005 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" candidate_term: 4 candidate_status { last_received { term: 3 index: 324 } } ignore_live_leader: false dest_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" is_pre_election: true
I20250903 20:05:13.555567 27684 raft_consensus.cc:2270] T 76742d0df44c43729197f7e09f96cad8 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:13.556289 27684 ts_tablet_manager.cc:1905] T 76742d0df44c43729197f7e09f96cad8 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:13.578677 27693 tablet_bootstrap.cc:492] T 4692dd93b76c42c4883c7e22c6032250 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:13.579015 27693 ts_tablet_manager.cc:1397] T 4692dd93b76c42c4883c7e22c6032250 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.214s user 0.006s sys 0.003s
I20250903 20:05:13.579799 27695 tablet_bootstrap.cc:492] T e20fcd28c3ae4355a033d2b8429f84e2 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:13.579986 27695 ts_tablet_manager.cc:1397] T e20fcd28c3ae4355a033d2b8429f84e2 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.017s
I20250903 20:05:13.580096 27693 raft_consensus.cc:357] T 4692dd93b76c42c4883c7e22c6032250 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.580320 27684 ts_tablet_manager.cc:1918] T 76742d0df44c43729197f7e09f96cad8 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:13.580365 27693 raft_consensus.cc:383] T 4692dd93b76c42c4883c7e22c6032250 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:13.580451 27693 raft_consensus.cc:738] T 4692dd93b76c42c4883c7e22c6032250 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:13.580575 27684 log.cc:1199] T 76742d0df44c43729197f7e09f96cad8 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/76742d0df44c43729197f7e09f96cad8
I20250903 20:05:13.580783 27693 consensus_queue.cc:260] T 4692dd93b76c42c4883c7e22c6032250 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.580837 27695 raft_consensus.cc:357] T e20fcd28c3ae4355a033d2b8429f84e2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.581069 27695 raft_consensus.cc:383] T e20fcd28c3ae4355a033d2b8429f84e2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:13.581112 27684 ts_tablet_manager.cc:1939] T 76742d0df44c43729197f7e09f96cad8 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:13.581153 27695 raft_consensus.cc:738] T e20fcd28c3ae4355a033d2b8429f84e2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:13.581421 27695 consensus_queue.cc:260] T e20fcd28c3ae4355a033d2b8429f84e2 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.581972 27695 ts_tablet_manager.cc:1428] T e20fcd28c3ae4355a033d2b8429f84e2 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20250903 20:05:13.582270 27050 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet d859c0222c3647dab3dddd105e4cf0fc (table default.Table_a71ff73faf3a43dbb95677820a0561b6 [id=7add4b636f1b4bffae663fdf432de05f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d859c0222c3647dab3dddd105e4cf0fc consensus state has no leader
I20250903 20:05:13.582710 27684 tablet_replica.cc:331] T d8ed7b3e292241cb9525745a797ff13a P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:13.583277 27693 ts_tablet_manager.cc:1428] T 4692dd93b76c42c4883c7e22c6032250 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:13.583599 27693 tablet_bootstrap.cc:492] T 1396d4f8a93b412ca8397368fb98834e P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:13.584141 27684 raft_consensus.cc:2241] T d8ed7b3e292241cb9525745a797ff13a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:13.584355 27684 raft_consensus.cc:2270] T d8ed7b3e292241cb9525745a797ff13a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:13.584951 27684 ts_tablet_manager.cc:1905] T d8ed7b3e292241cb9525745a797ff13a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:13.587483 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bf2545037c214bb1b5060575e3cdf81a consensus state has no leader
I20250903 20:05:13.589336 27693 tablet_bootstrap.cc:654] T 1396d4f8a93b412ca8397368fb98834e P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:13.590008 27695 tablet_bootstrap.cc:492] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:13.591773 27695 tablet_bootstrap.cc:654] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:13.592243 27684 ts_tablet_manager.cc:1918] T d8ed7b3e292241cb9525745a797ff13a P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:13.595748 27501 tablet_service.cc:1515] Processing DeleteTablet for tablet fe902f1b58924be1ac3a5585413f7789 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:13.597936 27684 log.cc:1199] T d8ed7b3e292241cb9525745a797ff13a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/d8ed7b3e292241cb9525745a797ff13a
I20250903 20:05:13.598476 27684 ts_tablet_manager.cc:1939] T d8ed7b3e292241cb9525745a797ff13a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:13.599404 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 46b5d491888147bbb101963641fd05f8 (table default.Table_b88b7b0fe749402e90c3826d01e8cc8e [id=3b241b63c32f42c0966e9af3dc06d701]) successfully deleted
I20250903 20:05:13.599781 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 76742d0df44c43729197f7e09f96cad8 (table default.Table_3055928e679e4dd8b8559b363b580e13 [id=9545d6a8d12e4937bb2b4cdbfc6933de]) successfully deleted
I20250903 20:05:13.600504 27063 master_service.cc:948] ReplaceTablet: received request to replace tablet 1fef9400227449ed892daff588999caa from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:13.601406 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet fe902f1b58924be1ac3a5585413f7789 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:13.601799 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet fe902f1b58924be1ac3a5585413f7789 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:13.603521 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet fe902f1b58924be1ac3a5585413f7789 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe902f1b58924be1ac3a5585413f7789 already in progress: creating tablet
W20250903 20:05:13.603828 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet fe902f1b58924be1ac3a5585413f7789 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe902f1b58924be1ac3a5585413f7789 already in progress: creating tablet
W20250903 20:05:13.604671 27053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.126:39015, user_credentials={real_user=slave}} blocked reactor thread for 85470.2us
I20250903 20:05:13.606318 27692 tablet_bootstrap.cc:492] T c25aaa6ff310432bb27a0695394ec420 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:13.606495 27692 ts_tablet_manager.cc:1397] T c25aaa6ff310432bb27a0695394ec420 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.182s user 0.004s sys 0.000s
I20250903 20:05:13.607425 27692 raft_consensus.cc:357] T c25aaa6ff310432bb27a0695394ec420 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.609782 27692 raft_consensus.cc:383] T c25aaa6ff310432bb27a0695394ec420 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:13.609858 27692 raft_consensus.cc:738] T c25aaa6ff310432bb27a0695394ec420 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:13.610074 27692 consensus_queue.cc:260] T c25aaa6ff310432bb27a0695394ec420 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.610638 27692 ts_tablet_manager.cc:1428] T c25aaa6ff310432bb27a0695394ec420 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:13.610949 27692 tablet_bootstrap.cc:492] T e5136d5388bc4c2ba71e09be40e8f412 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:13.613067 27692 tablet_bootstrap.cc:654] T e5136d5388bc4c2ba71e09be40e8f412 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:13.613535 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e78e6d212daf4589a6b4b65711e356cb consensus state has no leader
I20250903 20:05:13.607107 27684 tablet_replica.cc:331] T 8945085bdc174e6db22a5a9477810297 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:13.616523 27684 raft_consensus.cc:2241] T 8945085bdc174e6db22a5a9477810297 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:13.618232 27684 raft_consensus.cc:2270] T 8945085bdc174e6db22a5a9477810297 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:13.618417 27386 raft_consensus.cc:3058] T eaa11d17303142ff935705c16f0b9690 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:13.623335 27684 ts_tablet_manager.cc:1905] T 8945085bdc174e6db22a5a9477810297 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:13.638170 27060 catalog_manager.cc:5582] T c0343d2226e345669d42375de2545986 P e72ef1dbcc5143cf9f04a808f101c80f reported cstate change: term changed from 0 to 1, leader changed from <none> to e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67). New cstate: current_term: 1 leader_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:13.639904 27057 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_7325977ff92e45bab0b59b79579ecae4 [id=428657b402d748c0a9e3942d13342b4e] alter complete (version 1)
I20250903 20:05:13.653653 27684 ts_tablet_manager.cc:1918] T 8945085bdc174e6db22a5a9477810297 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:13.657799 27684 log.cc:1199] T 8945085bdc174e6db22a5a9477810297 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/8945085bdc174e6db22a5a9477810297
I20250903 20:05:13.657832 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet c25aaa6ff310432bb27a0695394ec420 with delete_type TABLET_DATA_DELETED (replaced by tablet 3f78a951bb024cf290116b0fe438ddb4) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:13.658135 27685 tablet_replica.cc:331] T c25aaa6ff310432bb27a0695394ec420 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:13.658371 27685 raft_consensus.cc:2241] T c25aaa6ff310432bb27a0695394ec420 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:13.658593 27685 raft_consensus.cc:2270] T c25aaa6ff310432bb27a0695394ec420 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:13.658850 27364 tablet_service.cc:1468] Processing CreateTablet for tablet e0347eab6c8642beabdc2d7e7bbfdc9b (DEFAULT_TABLE table=default.Table_187e391a6d6d4d0d96d4bb8a0581face [id=54ebade293b7466aa39f00ec876f27eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.659371 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0347eab6c8642beabdc2d7e7bbfdc9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.659729 27685 ts_tablet_manager.cc:1905] T c25aaa6ff310432bb27a0695394ec420 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:13.660971 27217 tablet_service.cc:1468] Processing CreateTablet for tablet e0347eab6c8642beabdc2d7e7bbfdc9b (DEFAULT_TABLE table=default.Table_187e391a6d6d4d0d96d4bb8a0581face [id=54ebade293b7466aa39f00ec876f27eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.661417 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0347eab6c8642beabdc2d7e7bbfdc9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.669739 27685 ts_tablet_manager.cc:1918] T c25aaa6ff310432bb27a0695394ec420 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:13.669958 27685 log.cc:1199] T c25aaa6ff310432bb27a0695394ec420 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/c25aaa6ff310432bb27a0695394ec420
I20250903 20:05:13.670425 27685 ts_tablet_manager.cc:1939] T c25aaa6ff310432bb27a0695394ec420 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:13.671264 27754 raft_consensus.cc:491] T 4692dd93b76c42c4883c7e22c6032250 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:13.671482 27754 raft_consensus.cc:513] T 4692dd93b76c42c4883c7e22c6032250 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.672212 27754 leader_election.cc:290] T 4692dd93b76c42c4883c7e22c6032250 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:13.672422 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 466e501e424e46bc98e8bbc64f97005e (DEFAULT_TABLE table=default.Table_83ea6dfcee144a4fbc86931567b676bb [id=f321a32c1ac043238c2fc92bfd5ec1ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.672526 27684 ts_tablet_manager.cc:1939] T 8945085bdc174e6db22a5a9477810297 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:13.672859 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 466e501e424e46bc98e8bbc64f97005e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:13.673184 27321 leader_election.cc:343] T 4692dd93b76c42c4883c7e22c6032250 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 4692dd93b76c42c4883c7e22c6032250
I20250903 20:05:13.674083 27364 tablet_service.cc:1468] Processing CreateTablet for tablet 466e501e424e46bc98e8bbc64f97005e (DEFAULT_TABLE table=default.Table_83ea6dfcee144a4fbc86931567b676bb [id=f321a32c1ac043238c2fc92bfd5ec1ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.674510 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 466e501e424e46bc98e8bbc64f97005e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.676743 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 45d9b6ff520348628d19671127b9426c
I20250903 20:05:13.678457 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 1fef9400227449ed892daff588999caa
W20250903 20:05:13.679312 27320 leader_election.cc:343] T 4692dd93b76c42c4883c7e22c6032250 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 4692dd93b76c42c4883c7e22c6032250
I20250903 20:05:13.679483 27320 leader_election.cc:304] T 4692dd93b76c42c4883c7e22c6032250 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:13.679827 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 8a223f75045345ae833f52bcc8f239f9 (DEFAULT_TABLE table=default.Table_c005844d740343c0a67ba9977747f574 [id=02a2c253dfb64f2cac5f70bde132aceb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.680030 27754 raft_consensus.cc:2747] T 4692dd93b76c42c4883c7e22c6032250 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:13.680253 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a223f75045345ae833f52bcc8f239f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.690150 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 8a223f75045345ae833f52bcc8f239f9 (DEFAULT_TABLE table=default.Table_c005844d740343c0a67ba9977747f574 [id=02a2c253dfb64f2cac5f70bde132aceb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.690814 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a223f75045345ae833f52bcc8f239f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.707545 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 45d9b6ff520348628d19671127b9426c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:13.711947 27851 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" instance_seqno: 1756929907545471) as {username='slave'} at 127.26.65.67:39445; Asking this server to re-register.
I20250903 20:05:13.713737 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet 45d9b6ff520348628d19671127b9426c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:13.717545 27853 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" instance_seqno: 1756929906558199) as {username='slave'} at 127.26.65.65:37603; Asking this server to re-register.
I20250903 20:05:13.725390 27295 heartbeater.cc:461] Registering TS with master...
I20250903 20:05:13.736119 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 1fef9400227449ed892daff588999caa with delete_type TABLET_DATA_DELETED (replaced by tablet 67ca1dcf6fc7435db555392235f9900f) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:13.738629 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet 1fef9400227449ed892daff588999caa with delete_type TABLET_DATA_DELETED (replaced by tablet 67ca1dcf6fc7435db555392235f9900f) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:13.740940 27692 tablet_bootstrap.cc:492] T e5136d5388bc4c2ba71e09be40e8f412 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:13.741250 27692 ts_tablet_manager.cc:1397] T e5136d5388bc4c2ba71e09be40e8f412 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.130s user 0.017s sys 0.013s
I20250903 20:05:13.742466 27692 raft_consensus.cc:357] T e5136d5388bc4c2ba71e09be40e8f412 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:13.742831 27692 raft_consensus.cc:383] T e5136d5388bc4c2ba71e09be40e8f412 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:13.742975 27692 raft_consensus.cc:738] T e5136d5388bc4c2ba71e09be40e8f412 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:13.743302 27692 consensus_queue.cc:260] T e5136d5388bc4c2ba71e09be40e8f412 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:13.745122 27066 master_service.cc:948] ReplaceTablet: received request to replace tablet f2cc35f4244a42eeb4a65de749824c11 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:13.745841 27692 ts_tablet_manager.cc:1428] T e5136d5388bc4c2ba71e09be40e8f412 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:13.746183 27692 tablet_bootstrap.cc:492] T 9509364ba3ec4e9bad09db09406a978a P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:13.749138 27692 tablet_bootstrap.cc:654] T 9509364ba3ec4e9bad09db09406a978a P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:13.750293 27063 catalog_manager.cc:6652] ReplaceTablet: tablet 1fef9400227449ed892daff588999caa deleted and replaced by tablet 67ca1dcf6fc7435db555392235f9900f
I20250903 20:05:13.755621 27501 tablet_service.cc:1468] Processing CreateTablet for tablet e0347eab6c8642beabdc2d7e7bbfdc9b (DEFAULT_TABLE table=default.Table_187e391a6d6d4d0d96d4bb8a0581face [id=54ebade293b7466aa39f00ec876f27eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.756150 27501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0347eab6c8642beabdc2d7e7bbfdc9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.769385 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet c25aaa6ff310432bb27a0695394ec420 (table default.Table_835ced1f8b194192809c3b6fbf4d1d96 [id=23f448924ec04ef08a541a14bd2ce5df]) successfully deleted
W20250903 20:05:13.770051 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 45d9b6ff520348628d19671127b9426c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 45d9b6ff520348628d19671127b9426c already in progress: creating tablet
W20250903 20:05:13.770499 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 1fef9400227449ed892daff588999caa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fef9400227449ed892daff588999caa already in progress: creating tablet
W20250903 20:05:13.771162 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 45d9b6ff520348628d19671127b9426c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 45d9b6ff520348628d19671127b9426c already in progress: creating tablet
W20250903 20:05:13.771502 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 1fef9400227449ed892daff588999caa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fef9400227449ed892daff588999caa already in progress: creating tablet
I20250903 20:05:13.772871 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 0a9b7770432942da9373d5100c71bd6a (DEFAULT_TABLE table=default.Table_6656b728e12a482fa9e7893ef4842ac7 [id=6788b524a8654e34a4d9ee8cc8fc84d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.773538 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a9b7770432942da9373d5100c71bd6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.780570 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_5fb98b65f51b4d628de6bed5d36b53d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:13.781419 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5fb98b65f51b4d628de6bed5d36b53d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:13.783572 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_1742550430f54b36b962c92f5c81f08a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:13.784358 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1742550430f54b36b962c92f5c81f08a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:13.786314 27064 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_9b80dba44fe8402999adc810b230d87a" } modify_external_catalogs: true
I20250903 20:05:13.786532 27064 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_9b80dba44fe8402999adc810b230d87a" } modify_external_catalogs: true
I20250903 20:05:13.788391 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7ea9ceb5ab084d6082d52019feeaf414" } new_table_name: "default.Table_7592bead271d4c929d27693318d7c1c3" modify_external_catalogs: true
I20250903 20:05:13.792191 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 8605bc329627438998812ed028327f53
I20250903 20:05:13.805294 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_0ac35f89d4d94e8fa94f9c3f853bf275"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:13.806442 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ac35f89d4d94e8fa94f9c3f853bf275 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:13.809648 27502 tablet_service.cc:1468] Processing CreateTablet for tablet 8a223f75045345ae833f52bcc8f239f9 (DEFAULT_TABLE table=default.Table_c005844d740343c0a67ba9977747f574 [id=02a2c253dfb64f2cac5f70bde132aceb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.810302 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a223f75045345ae833f52bcc8f239f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.829260 27853 ts_manager.cc:194] Registered new tserver with Master: 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:13.829835 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 45d9b6ff520348628d19671127b9426c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:13.831457 27501 tablet_service.cc:1468] Processing CreateTablet for tablet 466e501e424e46bc98e8bbc64f97005e (DEFAULT_TABLE table=default.Table_83ea6dfcee144a4fbc86931567b676bb [id=f321a32c1ac043238c2fc92bfd5ec1ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.831877 27501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 466e501e424e46bc98e8bbc64f97005e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.852371 27060 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 9367b3a22f624c4087848099de32c080 (table default.Table_08918e8b515e4487bf4aee0a33234bd5 [id=891b259c1c6b44e2a82dc982c24e6cb6]). Expected version 1 got 0
I20250903 20:05:13.853029 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet d8ed7b3e292241cb9525745a797ff13a (table default.Table_3da0f03ae8114e348baf44301196b2e5 [id=ad8b3a9bedef4c188b89640ab5e2473a]) successfully deleted
W20250903 20:05:13.853382 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 31342583c026438cbbae14106e75bbe5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31342583c026438cbbae14106e75bbe5 already in progress: creating tablet
W20250903 20:05:13.853626 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 7e2f98f760cf4df4bc2a10530e2157c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e2f98f760cf4df4bc2a10530e2157c6 already in progress: creating tablet
W20250903 20:05:13.853876 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 4268316c28e8471a9f28807d20dbb395 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4268316c28e8471a9f28807d20dbb395 already in progress: creating tablet
W20250903 20:05:13.854106 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet c9e90abef9814338b70a3550ab5686e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9e90abef9814338b70a3550ab5686e0 already in progress: creating tablet
I20250903 20:05:13.854399 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 8945085bdc174e6db22a5a9477810297 (table default.Table_a0582ccb94a64603a5fbdd0e531a9edf [id=6f7fece7a8d54d4786f1bab10da9129c]) successfully deleted
W20250903 20:05:13.854640 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet fe902f1b58924be1ac3a5585413f7789 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe902f1b58924be1ac3a5585413f7789 already in progress: creating tablet
I20250903 20:05:13.859254 27573 heartbeater.cc:461] Registering TS with master...
I20250903 20:05:13.862444 27853 ts_manager.cc:194] Registered new tserver with Master: e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:13.874236 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet 1fef9400227449ed892daff588999caa with delete_type TABLET_DATA_DELETED (replaced by tablet 67ca1dcf6fc7435db555392235f9900f) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:13.875691 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 0a9b7770432942da9373d5100c71bd6a (DEFAULT_TABLE table=default.Table_6656b728e12a482fa9e7893ef4842ac7 [id=6788b524a8654e34a4d9ee8cc8fc84d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.876305 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a9b7770432942da9373d5100c71bd6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.887961 28008 raft_consensus.cc:491] T c25aaa6ff310432bb27a0695394ec420 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:13.888173 28008 raft_consensus.cc:513] T c25aaa6ff310432bb27a0695394ec420 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.889273 28008 leader_election.cc:290] T c25aaa6ff310432bb27a0695394ec420 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:13.890573 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c25aaa6ff310432bb27a0695394ec420" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:13.890995 27462 leader_election.cc:343] T c25aaa6ff310432bb27a0695394ec420 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: c25aaa6ff310432bb27a0695394ec420
W20250903 20:05:13.891237 27462 leader_election.cc:343] T c25aaa6ff310432bb27a0695394ec420 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:13.891341 27462 leader_election.cc:304] T c25aaa6ff310432bb27a0695394ec420 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:13.892060 27364 tablet_service.cc:1468] Processing CreateTablet for tablet a72b85dfb34047528b2b28ff0d40f828 (DEFAULT_TABLE table=default.Table_2fa1d4a13c1840cd88c36e5fa2c6475f [id=d7b6b14b360e45a59034d2eed00e2e62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.892565 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a72b85dfb34047528b2b28ff0d40f828. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.893234 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet 8605bc329627438998812ed028327f53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:13.894362 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet 8605bc329627438998812ed028327f53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:13.894832 28008 raft_consensus.cc:2747] T c25aaa6ff310432bb27a0695394ec420 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:13.903635 27225 tablet_service.cc:1468] Processing CreateTablet for tablet a72b85dfb34047528b2b28ff0d40f828 (DEFAULT_TABLE table=default.Table_2fa1d4a13c1840cd88c36e5fa2c6475f [id=d7b6b14b360e45a59034d2eed00e2e62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.904289 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a72b85dfb34047528b2b28ff0d40f828. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.907577 27364 tablet_service.cc:1468] Processing CreateTablet for tablet 673c45f3f2a14da9bd306011d396eaca (DEFAULT_TABLE table=default.Table_43ea308240ac4628a72434608a22b975 [id=8de1ad04c15648d9bcccfc56efc9e7fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.908097 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 673c45f3f2a14da9bd306011d396eaca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.908385 27693 tablet_bootstrap.cc:492] T 1396d4f8a93b412ca8397368fb98834e P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:13.908550 27693 ts_tablet_manager.cc:1397] T 1396d4f8a93b412ca8397368fb98834e P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.325s user 0.057s sys 0.175s
I20250903 20:05:13.909510 27693 raft_consensus.cc:357] T 1396d4f8a93b412ca8397368fb98834e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.909729 27693 raft_consensus.cc:383] T 1396d4f8a93b412ca8397368fb98834e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:13.909799 27693 raft_consensus.cc:738] T 1396d4f8a93b412ca8397368fb98834e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:13.910094 27693 consensus_queue.cc:260] T 1396d4f8a93b412ca8397368fb98834e P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.910661 27693 ts_tablet_manager.cc:1428] T 1396d4f8a93b412ca8397368fb98834e P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:13.911015 27693 tablet_bootstrap.cc:492] T 29581a9dba46479884a0e6e01b7923de P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:13.913650 27693 tablet_bootstrap.cc:654] T 29581a9dba46479884a0e6e01b7923de P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:13.926916 27224 tablet_service.cc:1468] Processing CreateTablet for tablet 673c45f3f2a14da9bd306011d396eaca (DEFAULT_TABLE table=default.Table_43ea308240ac4628a72434608a22b975 [id=8de1ad04c15648d9bcccfc56efc9e7fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.927378 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 673c45f3f2a14da9bd306011d396eaca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.927362 27357 tablet_service.cc:1468] Processing CreateTablet for tablet 20c25c84c6fb4609a1964a4ff3c526da (DEFAULT_TABLE table=default.Table_6b435f3c61214c74a632c94e955abcdf [id=c2cf9ce651fc41a9b2b223b998e45d9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.927635 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 90baf83357494a0aae325908c118b471 (DEFAULT_TABLE table=default.Table_36a3bbdf545442dcacd498d928fa996b [id=7e9a08baa86e406c85192816b4a5d26d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.927829 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20c25c84c6fb4609a1964a4ff3c526da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.928098 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90baf83357494a0aae325908c118b471. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.927362 27364 tablet_service.cc:1468] Processing CreateTablet for tablet 90baf83357494a0aae325908c118b471 (DEFAULT_TABLE table=default.Table_36a3bbdf545442dcacd498d928fa996b [id=7e9a08baa86e406c85192816b4a5d26d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.934388 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90baf83357494a0aae325908c118b471. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.938374 27840 raft_consensus.cc:491] T e20fcd28c3ae4355a033d2b8429f84e2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:13.938632 27840 raft_consensus.cc:513] T e20fcd28c3ae4355a033d2b8429f84e2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.939002 27224 tablet_service.cc:1468] Processing CreateTablet for tablet 20c25c84c6fb4609a1964a4ff3c526da (DEFAULT_TABLE table=default.Table_6b435f3c61214c74a632c94e955abcdf [id=c2cf9ce651fc41a9b2b223b998e45d9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.939316 27840 leader_election.cc:290] T e20fcd28c3ae4355a033d2b8429f84e2 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:13.939271 27357 tablet_service.cc:1468] Processing CreateTablet for tablet f9709dfd18654c5bb071e39da7c8889f (DEFAULT_TABLE table=default.Table_07ba95129d584160ae8b768f77aa044d [id=39cbb7f0d8e145c6b5f283133017b80f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.939414 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20c25c84c6fb4609a1964a4ff3c526da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.939628 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9709dfd18654c5bb071e39da7c8889f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.940037 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20fcd28c3ae4355a033d2b8429f84e2" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:13.940768 27183 leader_election.cc:343] T e20fcd28c3ae4355a033d2b8429f84e2 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:13.941145 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20fcd28c3ae4355a033d2b8429f84e2" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:13.941754 27184 leader_election.cc:343] T e20fcd28c3ae4355a033d2b8429f84e2 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:13.941874 27184 leader_election.cc:304] T e20fcd28c3ae4355a033d2b8429f84e2 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:13.942628 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 607c9a9cbc5540e5bd0fa436fcba564c (DEFAULT_TABLE table=default.Table_5b44669e91a8427c8e7d96b866fdb3b5 [id=96167b0d9f9b47168a9fc59635b7136d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.942878 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 607c9a9cbc5540e5bd0fa436fcba564c (DEFAULT_TABLE table=default.Table_5b44669e91a8427c8e7d96b866fdb3b5 [id=96167b0d9f9b47168a9fc59635b7136d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.942970 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 607c9a9cbc5540e5bd0fa436fcba564c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.943259 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 607c9a9cbc5540e5bd0fa436fcba564c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.946492 27695 tablet_bootstrap.cc:492] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:13.946641 27695 ts_tablet_manager.cc:1397] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.357s user 0.050s sys 0.121s
I20250903 20:05:13.946861 27366 tablet_service.cc:1468] Processing CreateTablet for tablet a9f43781299e47a8aecc37a0b73d7a64 (DEFAULT_TABLE table=default.Table_cdb258f01fd94ff48ddd75c368e68d6c [id=f70d7ea93e154e6199f6766316353862]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.947098 27840 raft_consensus.cc:2747] T e20fcd28c3ae4355a033d2b8429f84e2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:13.947232 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9f43781299e47a8aecc37a0b73d7a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.947284 27840 consensus_queue.cc:1035] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:13.947515 27695 raft_consensus.cc:357] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.947677 27695 raft_consensus.cc:383] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:13.947768 27695 raft_consensus.cc:738] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:13.948026 27695 consensus_queue.cc:260] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.948598 27695 ts_tablet_manager.cc:1428] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250903 20:05:13.950196 27695 tablet_bootstrap.cc:492] T cfa8a1d9011e460681443a6b77b95958 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:13.952703 27695 tablet_bootstrap.cc:654] T cfa8a1d9011e460681443a6b77b95958 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:13.955168 27693 tablet_bootstrap.cc:492] T 29581a9dba46479884a0e6e01b7923de P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:13.955356 27693 ts_tablet_manager.cc:1397] T 29581a9dba46479884a0e6e01b7923de P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.004s
I20250903 20:05:13.956390 27693 raft_consensus.cc:357] T 29581a9dba46479884a0e6e01b7923de P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:13.956522 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 45d9b6ff520348628d19671127b9426c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 45d9b6ff520348628d19671127b9426c already in progress: creating tablet
I20250903 20:05:13.956640 27693 raft_consensus.cc:383] T 29581a9dba46479884a0e6e01b7923de P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:13.956741 27693 raft_consensus.cc:738] T 29581a9dba46479884a0e6e01b7923de P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
W20250903 20:05:13.956986 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 1fef9400227449ed892daff588999caa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fef9400227449ed892daff588999caa already in progress: creating tablet
I20250903 20:05:13.957012 27693 consensus_queue.cc:260] T 29581a9dba46479884a0e6e01b7923de P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.957517 27386 raft_consensus.cc:3058] T 9367b3a22f624c4087848099de32c080 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:13.957648 27693 ts_tablet_manager.cc:1428] T 29581a9dba46479884a0e6e01b7923de P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:13.957978 27693 tablet_bootstrap.cc:492] T 46b5d491888147bbb101963641fd05f8 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:13.958197 27692 tablet_bootstrap.cc:492] T 9509364ba3ec4e9bad09db09406a978a P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:13.959030 27692 ts_tablet_manager.cc:1397] T 9509364ba3ec4e9bad09db09406a978a P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.213s user 0.005s sys 0.039s
I20250903 20:05:13.960036 27693 tablet_bootstrap.cc:654] T 46b5d491888147bbb101963641fd05f8 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:13.960984 27224 tablet_service.cc:1468] Processing CreateTablet for tablet a9f43781299e47a8aecc37a0b73d7a64 (DEFAULT_TABLE table=default.Table_cdb258f01fd94ff48ddd75c368e68d6c [id=f70d7ea93e154e6199f6766316353862]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.961316 27217 tablet_service.cc:1468] Processing CreateTablet for tablet f9709dfd18654c5bb071e39da7c8889f (DEFAULT_TABLE table=default.Table_07ba95129d584160ae8b768f77aa044d [id=39cbb7f0d8e145c6b5f283133017b80f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.961493 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9f43781299e47a8aecc37a0b73d7a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.959825 27692 raft_consensus.cc:357] T 9509364ba3ec4e9bad09db09406a978a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.961941 27692 raft_consensus.cc:383] T 9509364ba3ec4e9bad09db09406a978a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:13.962021 27692 raft_consensus.cc:738] T 9509364ba3ec4e9bad09db09406a978a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:13.962354 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9709dfd18654c5bb071e39da7c8889f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.962584 27495 tablet_service.cc:1468] Processing CreateTablet for tablet f2629f46704e4a0fa7922533928ca2ed (DEFAULT_TABLE table=default.Table_27706ce3e18b47078bb7d3071c12a1a2 [id=2051ecb140f24088911aad60fdb9082b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.962972 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2629f46704e4a0fa7922533928ca2ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.963132 27502 tablet_service.cc:1468] Processing CreateTablet for tablet 0a9b7770432942da9373d5100c71bd6a (DEFAULT_TABLE table=default.Table_6656b728e12a482fa9e7893ef4842ac7 [id=6788b524a8654e34a4d9ee8cc8fc84d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.963466 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a9b7770432942da9373d5100c71bd6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.966367 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 8605bc329627438998812ed028327f53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:13.966670 27692 consensus_queue.cc:260] T 9509364ba3ec4e9bad09db09406a978a P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.966845 27498 tablet_service.cc:1468] Processing CreateTablet for tablet a72b85dfb34047528b2b28ff0d40f828 (DEFAULT_TABLE table=default.Table_2fa1d4a13c1840cd88c36e5fa2c6475f [id=d7b6b14b360e45a59034d2eed00e2e62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.967304 27692 ts_tablet_manager.cc:1428] T 9509364ba3ec4e9bad09db09406a978a P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250903 20:05:13.967288 27076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" candidate_term: 4 candidate_status { last_received { term: 3 index: 324 } } ignore_live_leader: false dest_uuid: "35bdb0090d8747059b5ee103e786e979" is_pre_election: true
I20250903 20:05:13.967358 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 673c45f3f2a14da9bd306011d396eaca (DEFAULT_TABLE table=default.Table_43ea308240ac4628a72434608a22b975 [id=8de1ad04c15648d9bcccfc56efc9e7fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.967837 27503 tablet_service.cc:1468] Processing CreateTablet for tablet 90baf83357494a0aae325908c118b471 (DEFAULT_TABLE table=default.Table_36a3bbdf545442dcacd498d928fa996b [id=7e9a08baa86e406c85192816b4a5d26d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.968254 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 20c25c84c6fb4609a1964a4ff3c526da (DEFAULT_TABLE table=default.Table_6b435f3c61214c74a632c94e955abcdf [id=c2cf9ce651fc41a9b2b223b998e45d9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.968940 27500 tablet_service.cc:1468] Processing CreateTablet for tablet f9709dfd18654c5bb071e39da7c8889f (DEFAULT_TABLE table=default.Table_07ba95129d584160ae8b768f77aa044d [id=39cbb7f0d8e145c6b5f283133017b80f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.969292 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 607c9a9cbc5540e5bd0fa436fcba564c (DEFAULT_TABLE table=default.Table_5b44669e91a8427c8e7d96b866fdb3b5 [id=96167b0d9f9b47168a9fc59635b7136d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.969652 27499 tablet_service.cc:1468] Processing CreateTablet for tablet a9f43781299e47a8aecc37a0b73d7a64 (DEFAULT_TABLE table=default.Table_cdb258f01fd94ff48ddd75c368e68d6c [id=f70d7ea93e154e6199f6766316353862]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.970038 27496 tablet_service.cc:1468] Processing CreateTablet for tablet 9e98cd1d9bd246e182e2969d147e68bf (DEFAULT_TABLE table=default.Table_35d35c0c3a6048bc9d7def161d9e16db [id=b27f08d56c3c43e6a09a465fdda4d3db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.970539 27497 tablet_service.cc:1468] Processing CreateTablet for tablet 5f27f525fdab407280aa23df1ae71b43 (DEFAULT_TABLE table=default.Table_1c679659094d4666aaaf68f76f667890 [id=61a25739b03042f7bd5bb693b68656d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:13.972843 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90baf83357494a0aae325908c118b471. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.974503 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet fdc77810f27a40758e063b8b1150ab71
I20250903 20:05:13.974963 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 673c45f3f2a14da9bd306011d396eaca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.975512 27498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a72b85dfb34047528b2b28ff0d40f828. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:13.978497 27801 leader_election.cc:304] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8857b5d4ea1e4f45aafb8e029837cca2; no voters: 35bdb0090d8747059b5ee103e786e979, 35dd69f3fa7a4fd8843b97f8a5d68296
I20250903 20:05:13.979194 27692 tablet_bootstrap.cc:492] T 710e3bc3c4194efbbeef6bc76ee64f24 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:13.980545 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_cc37561faad74a2093d52d313b9f1010" } new_table_name: "default.Table_57cd5123dc7042c1887af6e833af4d7a" modify_external_catalogs: true
I20250903 20:05:13.982053 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_644507bb34724729854f122d8191cf82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:13.982298 27695 tablet_bootstrap.cc:492] T cfa8a1d9011e460681443a6b77b95958 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:13.982478 27695 ts_tablet_manager.cc:1397] T cfa8a1d9011e460681443a6b77b95958 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.005s
W20250903 20:05:13.982777 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_644507bb34724729854f122d8191cf82 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:13.983485 27695 raft_consensus.cc:357] T cfa8a1d9011e460681443a6b77b95958 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.983747 27695 raft_consensus.cc:383] T cfa8a1d9011e460681443a6b77b95958 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:13.983834 27695 raft_consensus.cc:738] T cfa8a1d9011e460681443a6b77b95958 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:13.984126 27695 consensus_queue.cc:260] T cfa8a1d9011e460681443a6b77b95958 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:13.986513 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet fdc77810f27a40758e063b8b1150ab71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:13.986639 27692 tablet_bootstrap.cc:654] T 710e3bc3c4194efbbeef6bc76ee64f24 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:13.991364 27952 raft_consensus.cc:491] T e5136d5388bc4c2ba71e09be40e8f412 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:13.991578 27952 raft_consensus.cc:513] T e5136d5388bc4c2ba71e09be40e8f412 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:13.992244 27952 leader_election.cc:290] T e5136d5388bc4c2ba71e09be40e8f412 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:13.992940 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5136d5388bc4c2ba71e09be40e8f412" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:13.993178 27248 raft_consensus.cc:2466] T e5136d5388bc4c2ba71e09be40e8f412 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:13.993818 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5136d5388bc4c2ba71e09be40e8f412" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:13.994235 27695 ts_tablet_manager.cc:1428] T cfa8a1d9011e460681443a6b77b95958 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250903 20:05:13.994460 27462 leader_election.cc:304] T e5136d5388bc4c2ba71e09be40e8f412 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:13.994664 27695 tablet_bootstrap.cc:492] T 8023fa7537234ce783c2bf90b575b00c P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
W20250903 20:05:13.994949 27462 leader_election.cc:343] T e5136d5388bc4c2ba71e09be40e8f412 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:13.995357 27952 raft_consensus.cc:2802] T e5136d5388bc4c2ba71e09be40e8f412 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:13.995469 27952 raft_consensus.cc:491] T e5136d5388bc4c2ba71e09be40e8f412 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:13.995561 27952 raft_consensus.cc:3058] T e5136d5388bc4c2ba71e09be40e8f412 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:13.997042 27695 tablet_bootstrap.cc:654] T 8023fa7537234ce783c2bf90b575b00c P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:13.999059 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9f43781299e47a8aecc37a0b73d7a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.002538 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9709dfd18654c5bb071e39da7c8889f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.006481 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 607c9a9cbc5540e5bd0fa436fcba564c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.010141 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20c25c84c6fb4609a1964a4ff3c526da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.012020 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_66babebee1c9462ea057c8489d43846d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:14.012722 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66babebee1c9462ea057c8489d43846d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:14.014132 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f27f525fdab407280aa23df1ae71b43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.018139 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e98cd1d9bd246e182e2969d147e68bf. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:14.022902 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 8605bc329627438998812ed028327f53 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8605bc329627438998812ed028327f53 already in progress: creating tablet
W20250903 20:05:14.023209 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet fdc77810f27a40758e063b8b1150ab71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fdc77810f27a40758e063b8b1150ab71 already in progress: creating tablet
I20250903 20:05:14.026240 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet e5136d5388bc4c2ba71e09be40e8f412
I20250903 20:05:14.029630 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet e5136d5388bc4c2ba71e09be40e8f412 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:14.030048 27684 tablet_replica.cc:331] T e5136d5388bc4c2ba71e09be40e8f412 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:14.030258 27495 tablet_service.cc:1468] Processing CreateTablet for tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8 (DEFAULT_TABLE table=default.Table_943bea896c744410a2a4d80ffb9ca439 [id=a950fbaecd7341989961ff3ea9a713e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.031154 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.034984 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_2f6f89e1dd48493f9e989d4526caffed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:14.035667 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f6f89e1dd48493f9e989d4526caffed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:14.043104 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0ad64c699b294679b6fec37b5ad23528 consensus state has no leader
I20250903 20:05:14.044111 27952 raft_consensus.cc:513] T e5136d5388bc4c2ba71e09be40e8f412 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:14.044449 27684 raft_consensus.cc:2241] T e5136d5388bc4c2ba71e09be40e8f412 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:14.044804 27684 raft_consensus.cc:2270] T e5136d5388bc4c2ba71e09be40e8f412 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:14.046285 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5136d5388bc4c2ba71e09be40e8f412" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:14.046516 27248 raft_consensus.cc:3058] T e5136d5388bc4c2ba71e09be40e8f412 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:14.049314 27248 raft_consensus.cc:2466] T e5136d5388bc4c2ba71e09be40e8f412 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:14.049922 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_56d2ca07338b4ce4ad732c5b2401812b" } new_table_name: "default.Table_9b20469406c94986ab496ea3e678b586" modify_external_catalogs: true
I20250903 20:05:14.050032 27462 leader_election.cc:304] T e5136d5388bc4c2ba71e09be40e8f412 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
W20250903 20:05:14.050344 27462 raft_consensus.cc:2684] T e5136d5388bc4c2ba71e09be40e8f412 P e72ef1dbcc5143cf9f04a808f101c80f: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250903 20:05:14.052948 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5136d5388bc4c2ba71e09be40e8f412" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
W20250903 20:05:14.053561 27462 leader_election.cc:343] T e5136d5388bc4c2ba71e09be40e8f412 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:14.053810 27952 leader_election.cc:290] T e5136d5388bc4c2ba71e09be40e8f412 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:14.054960 27684 ts_tablet_manager.cc:1905] T e5136d5388bc4c2ba71e09be40e8f412 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:14.061970 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet f2cc35f4244a42eeb4a65de749824c11
I20250903 20:05:14.062572 27066 catalog_manager.cc:6652] ReplaceTablet: tablet f2cc35f4244a42eeb4a65de749824c11 deleted and replaced by tablet fc634116a78f45dd833034ddda90e9bf
I20250903 20:05:14.083501 27684 ts_tablet_manager.cc:1918] T e5136d5388bc4c2ba71e09be40e8f412 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:14.083786 27684 log.cc:1199] T e5136d5388bc4c2ba71e09be40e8f412 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/e5136d5388bc4c2ba71e09be40e8f412
I20250903 20:05:14.084307 27684 ts_tablet_manager.cc:1939] T e5136d5388bc4c2ba71e09be40e8f412 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:14.085402 27385 raft_consensus.cc:3058] T 1396d4f8a93b412ca8397368fb98834e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:14.089376 27840 consensus_queue.cc:1035] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:14.096935 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b20633c3b14e463aac0b7bd917c3ab9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:14.097674 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b20633c3b14e463aac0b7bd917c3ab9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:14.099812 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet f2cc35f4244a42eeb4a65de749824c11 with delete_type TABLET_DATA_DELETED (replaced by tablet fc634116a78f45dd833034ddda90e9bf) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:14.110193 27840 raft_consensus.cc:491] T 4268316c28e8471a9f28807d20dbb395 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:14.110476 27840 raft_consensus.cc:513] T 4268316c28e8471a9f28807d20dbb395 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.111155 27840 leader_election.cc:290] T 4268316c28e8471a9f28807d20dbb395 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:14.112629 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4268316c28e8471a9f28807d20dbb395" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:14.112962 27057 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_08918e8b515e4487bf4aee0a33234bd5 [id=891b259c1c6b44e2a82dc982c24e6cb6] alter complete (version 1)
W20250903 20:05:14.113392 27183 leader_election.cc:343] T 4268316c28e8471a9f28807d20dbb395 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:14.119556 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4268316c28e8471a9f28807d20dbb395" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:14.120213 27184 leader_election.cc:343] T 4268316c28e8471a9f28807d20dbb395 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:14.120385 27184 leader_election.cc:304] T 4268316c28e8471a9f28807d20dbb395 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:14.120667 27840 raft_consensus.cc:2747] T 4268316c28e8471a9f28807d20dbb395 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:14.121243 27693 tablet_bootstrap.cc:492] T 46b5d491888147bbb101963641fd05f8 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:14.122571 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d342b3c06e1c4b98bc7f07e88844cdc9" } modify_external_catalogs: true
I20250903 20:05:14.122763 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d342b3c06e1c4b98bc7f07e88844cdc9" } modify_external_catalogs: true
I20250903 20:05:14.123129 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_01215977307c453e9236e5bd72bc8427"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:14.123498 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7325977ff92e45bab0b59b79579ecae4" } new_table_name: "default.Table_a9a6eefd24394081aad7a7b56c2613f5" modify_external_catalogs: true
W20250903 20:05:14.123803 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01215977307c453e9236e5bd72bc8427 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:14.124915 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet e5136d5388bc4c2ba71e09be40e8f412 (table default.Table_9b80dba44fe8402999adc810b230d87a [id=790f57164c7d4183a3df17ba7f230d24]) successfully deleted
I20250903 20:05:14.125811 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet c25aaa6ff310432bb27a0695394ec420 with delete_type TABLET_DATA_DELETED (replaced by tablet 3f78a951bb024cf290116b0fe438ddb4) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:14.126559 27684 tablet_replica.cc:331] T c25aaa6ff310432bb27a0695394ec420 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:14.126942 27684 raft_consensus.cc:2241] T c25aaa6ff310432bb27a0695394ec420 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:14.127209 27684 raft_consensus.cc:2270] T c25aaa6ff310432bb27a0695394ec420 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:14.127787 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet f2cc35f4244a42eeb4a65de749824c11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2cc35f4244a42eeb4a65de749824c11 already in progress: creating tablet
I20250903 20:05:14.128875 27695 tablet_bootstrap.cc:492] T 8023fa7537234ce783c2bf90b575b00c P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:14.129034 27695 ts_tablet_manager.cc:1397] T 8023fa7537234ce783c2bf90b575b00c P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.135s user 0.008s sys 0.012s
I20250903 20:05:14.129207 27684 ts_tablet_manager.cc:1905] T c25aaa6ff310432bb27a0695394ec420 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:14.129856 27695 raft_consensus.cc:357] T 8023fa7537234ce783c2bf90b575b00c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.130048 27695 raft_consensus.cc:383] T 8023fa7537234ce783c2bf90b575b00c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:14.130141 27695 raft_consensus.cc:738] T 8023fa7537234ce783c2bf90b575b00c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:14.130455 27695 consensus_queue.cc:260] T 8023fa7537234ce783c2bf90b575b00c P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.133996 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet 9509364ba3ec4e9bad09db09406a978a with delete_type TABLET_DATA_DELETED (replaced by tablet 1fbe54dc40dd47daa109ceeb07546b0a) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:14.137072 27501 tablet_service.cc:1468] Processing CreateTablet for tablet 4af8c03c9d834405b4e25d35441db961 (DEFAULT_TABLE table=default.Table_6d201f66980a4a7ab430058673f8d513 [id=058a64daaf104dfdace7d05d52388b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.137584 27501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4af8c03c9d834405b4e25d35441db961. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:14.140162 27059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4157633525034a019fd6a80185892cec consensus state has no leader
I20250903 20:05:14.140523 27059 master_service.cc:948] ReplaceTablet: received request to replace tablet 7e2f98f760cf4df4bc2a10530e2157c6 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:14.141000 27059 master_service.cc:948] ReplaceTablet: received request to replace tablet c6d67cd0c0304c77ba571b414878dad2 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:14.141098 27695 ts_tablet_manager.cc:1428] T 8023fa7537234ce783c2bf90b575b00c P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20250903 20:05:14.141532 27065 master_service.cc:948] ReplaceTablet: received request to replace tablet 4be9bdc390de4a74a396003c9c3c2b71 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:14.142071 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_df2f2840b3c7420bac713679e7eb971d" } new_table_name: "default.Table_d9276aa6946f41d7ac0fabc39338c23d" modify_external_catalogs: true
I20250903 20:05:14.146075 27695 tablet_bootstrap.cc:492] T 44db427ee53a4d0a907dfd63282d6d4f P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:14.148002 27695 tablet_bootstrap.cc:654] T 44db427ee53a4d0a907dfd63282d6d4f P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:14.151929 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 9e98cd1d9bd246e182e2969d147e68bf (DEFAULT_TABLE table=default.Table_35d35c0c3a6048bc9d7def161d9e16db [id=b27f08d56c3c43e6a09a465fdda4d3db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.152467 27215 tablet_service.cc:1468] Processing CreateTablet for tablet f2629f46704e4a0fa7922533928ca2ed (DEFAULT_TABLE table=default.Table_27706ce3e18b47078bb7d3071c12a1a2 [id=2051ecb140f24088911aad60fdb9082b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.152805 27224 tablet_service.cc:1468] Processing CreateTablet for tablet 5f27f525fdab407280aa23df1ae71b43 (DEFAULT_TABLE table=default.Table_1c679659094d4666aaaf68f76f667890 [id=61a25739b03042f7bd5bb693b68656d5]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:14.153631 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 8605bc329627438998812ed028327f53 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8605bc329627438998812ed028327f53 already in progress: creating tablet
I20250903 20:05:14.152498 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e98cd1d9bd246e182e2969d147e68bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.154642 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2629f46704e4a0fa7922533928ca2ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.154942 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f27f525fdab407280aa23df1ae71b43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.164378 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ff22ae81dcc340ca897c8e8667aa83d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:14.165308 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff22ae81dcc340ca897c8e8667aa83d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:14.173223 27952 raft_consensus.cc:491] T 9509364ba3ec4e9bad09db09406a978a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:14.173453 27952 raft_consensus.cc:513] T 9509364ba3ec4e9bad09db09406a978a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:14.175495 27462 leader_election.cc:343] T 9509364ba3ec4e9bad09db09406a978a P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 9509364ba3ec4e9bad09db09406a978a
I20250903 20:05:14.175778 27952 leader_election.cc:290] T 9509364ba3ec4e9bad09db09406a978a P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:14.181494 27960 mvcc.cc:204] Tried to move back new op lower bound from 7196384922079830016 to 7196384904500998144. Current Snapshot: MvccSnapshot[applied={T|T < 7196384922079830016 or (T in {7196384922079830016})}]
I20250903 20:05:14.184193 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet fdc77810f27a40758e063b8b1150ab71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:14.184502 27801 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.124:40911, user_credentials={real_user=slave}} blocked reactor thread for 206052us
I20250903 20:05:14.182595 28034 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250903 20:05:14.185418 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 4af8c03c9d834405b4e25d35441db961 (DEFAULT_TABLE table=default.Table_6d201f66980a4a7ab430058673f8d513 [id=058a64daaf104dfdace7d05d52388b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.185928 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4af8c03c9d834405b4e25d35441db961. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.189823 27695 tablet_bootstrap.cc:492] T 44db427ee53a4d0a907dfd63282d6d4f P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:14.190038 27695 ts_tablet_manager.cc:1397] T 44db427ee53a4d0a907dfd63282d6d4f P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.014s
I20250903 20:05:14.190958 27695 raft_consensus.cc:357] T 44db427ee53a4d0a907dfd63282d6d4f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.191185 27695 raft_consensus.cc:383] T 44db427ee53a4d0a907dfd63282d6d4f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:14.191269 27695 raft_consensus.cc:738] T 44db427ee53a4d0a907dfd63282d6d4f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:14.191709 28030 raft_consensus.cc:491] T 925c0ab4873041da82702cd9377ec7bc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:14.191872 28030 raft_consensus.cc:513] T 925c0ab4873041da82702cd9377ec7bc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:14.192620 28030 leader_election.cc:290] T 925c0ab4873041da82702cd9377ec7bc P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
W20250903 20:05:14.193568 27183 leader_election.cc:343] T 925c0ab4873041da82702cd9377ec7bc P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 925c0ab4873041da82702cd9377ec7bc
I20250903 20:05:14.194067 27224 tablet_service.cc:1515] Processing DeleteTablet for tablet e5136d5388bc4c2ba71e09be40e8f412 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:14.194396 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet 925c0ab4873041da82702cd9377ec7bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:14.194473 27685 tablet_replica.cc:331] T e5136d5388bc4c2ba71e09be40e8f412 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:14.194720 27685 raft_consensus.cc:2241] T e5136d5388bc4c2ba71e09be40e8f412 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:14.194746 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet 8945085bdc174e6db22a5a9477810297 with delete_type TABLET_DATA_DELETED (replaced by tablet f71d3dd9af2543299296f21381d1d563) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:14.194864 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet 4268316c28e8471a9f28807d20dbb395 with delete_type TABLET_DATA_DELETED (replaced by tablet a9f43781299e47a8aecc37a0b73d7a64) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:14.194914 27685 raft_consensus.cc:2270] T e5136d5388bc4c2ba71e09be40e8f412 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:14.195106 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet f2cc35f4244a42eeb4a65de749824c11 with delete_type TABLET_DATA_DELETED (replaced by tablet fc634116a78f45dd833034ddda90e9bf) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:14.191594 27695 consensus_queue.cc:260] T 44db427ee53a4d0a907dfd63282d6d4f P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.195513 27685 ts_tablet_manager.cc:1905] T e5136d5388bc4c2ba71e09be40e8f412 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:14.197252 27684 ts_tablet_manager.cc:1918] T c25aaa6ff310432bb27a0695394ec420 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:14.197396 27684 log.cc:1199] T c25aaa6ff310432bb27a0695394ec420 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/c25aaa6ff310432bb27a0695394ec420
I20250903 20:05:14.198050 27695 ts_tablet_manager.cc:1428] T 44db427ee53a4d0a907dfd63282d6d4f P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20250903 20:05:14.198346 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet e20fcd28c3ae4355a033d2b8429f84e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:14.198437 27695 tablet_bootstrap.cc:492] T c9c870fb14e3469fb2979d8462b49121 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:14.199725 27224 tablet_service.cc:1515] Processing DeleteTablet for tablet f7ab722c37264973b204cd4f1f95052d with delete_type TABLET_DATA_DELETED (replaced by tablet 57a1167457b84ea8a493be232b1d088a) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:14.201431 27695 tablet_bootstrap.cc:654] T c9c870fb14e3469fb2979d8462b49121 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:14.205267 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8 (DEFAULT_TABLE table=default.Table_943bea896c744410a2a4d80ffb9ca439 [id=a950fbaecd7341989961ff3ea9a713e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.205296 27685 ts_tablet_manager.cc:1918] T e5136d5388bc4c2ba71e09be40e8f412 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:14.205878 27685 log.cc:1199] T e5136d5388bc4c2ba71e09be40e8f412 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/e5136d5388bc4c2ba71e09be40e8f412
I20250903 20:05:14.206383 27685 ts_tablet_manager.cc:1939] T e5136d5388bc4c2ba71e09be40e8f412 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
W20250903 20:05:14.208698 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3b3bfc63aa0b4e56bec189e834dc6a8d consensus state has no leader
I20250903 20:05:14.209765 27685 tablet_replica.cc:331] T 925c0ab4873041da82702cd9377ec7bc P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:14.210011 27685 raft_consensus.cc:2241] T 925c0ab4873041da82702cd9377ec7bc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:14.210249 27685 raft_consensus.cc:2270] T 925c0ab4873041da82702cd9377ec7bc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:14.210899 27685 ts_tablet_manager.cc:1905] T 925c0ab4873041da82702cd9377ec7bc P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:14.213369 27060 master_service.cc:948] ReplaceTablet: received request to replace tablet 4268316c28e8471a9f28807d20dbb395 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:14.214841 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_f2d0d1d8bddc492dbccd2c5c0a4e1a16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:14.215936 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_43ea308240ac4628a72434608a22b975" } modify_external_catalogs: true
I20250903 20:05:14.216122 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_43ea308240ac4628a72434608a22b975" } modify_external_catalogs: true
I20250903 20:05:14.217121 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_bffb6801f48c4124bfe24c49586d6e48" } modify_external_catalogs: true
I20250903 20:05:14.217322 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_bffb6801f48c4124bfe24c49586d6e48" } modify_external_catalogs: true
I20250903 20:05:14.218837 27693 ts_tablet_manager.cc:1397] T 46b5d491888147bbb101963641fd05f8 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.164s user 0.026s sys 0.034s
W20250903 20:05:14.219936 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 8605bc329627438998812ed028327f53 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8605bc329627438998812ed028327f53 already in progress: creating tablet
I20250903 20:05:14.221182 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 4692dd93b76c42c4883c7e22c6032250 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:14.205730 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.222556 27364 tablet_service.cc:1468] Processing CreateTablet for tablet f2629f46704e4a0fa7922533928ca2ed (DEFAULT_TABLE table=default.Table_27706ce3e18b47078bb7d3071c12a1a2 [id=2051ecb140f24088911aad60fdb9082b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.222555 27687 tablet_replica.cc:331] T 4692dd93b76c42c4883c7e22c6032250 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:14.222594 27384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "925c0ab4873041da82702cd9377ec7bc" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:14.225178 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2d0d1d8bddc492dbccd2c5c0a4e1a16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:14.225559 27363 tablet_service.cc:1515] Processing DeleteTablet for tablet e5136d5388bc4c2ba71e09be40e8f412 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:14.226276 27184 leader_election.cc:343] T 925c0ab4873041da82702cd9377ec7bc P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:14.226446 27184 leader_election.cc:304] T 925c0ab4873041da82702cd9377ec7bc P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:14.226660 27184 raft_consensus.cc:2684] T 925c0ab4873041da82702cd9377ec7bc P 89f8f0ee68b940db9ce250c0604f934a: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250903 20:05:14.227010 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet f2cc35f4244a42eeb4a65de749824c11 with delete_type TABLET_DATA_DELETED (replaced by tablet fc634116a78f45dd833034ddda90e9bf) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:14.227298 27356 tablet_service.cc:1468] Processing CreateTablet for tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8 (DEFAULT_TABLE table=default.Table_943bea896c744410a2a4d80ffb9ca439 [id=a950fbaecd7341989961ff3ea9a713e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.227636 27359 tablet_service.cc:1468] Processing CreateTablet for tablet 4af8c03c9d834405b4e25d35441db961 (DEFAULT_TABLE table=default.Table_6d201f66980a4a7ab430058673f8d513 [id=058a64daaf104dfdace7d05d52388b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.228669 27693 raft_consensus.cc:357] T 46b5d491888147bbb101963641fd05f8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.228865 27693 raft_consensus.cc:383] T 46b5d491888147bbb101963641fd05f8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:14.228932 27693 raft_consensus.cc:738] T 46b5d491888147bbb101963641fd05f8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:14.229149 27693 consensus_queue.cc:260] T 46b5d491888147bbb101963641fd05f8 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.229631 27693 ts_tablet_manager.cc:1428] T 46b5d491888147bbb101963641fd05f8 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250903 20:05:14.229964 27693 tablet_bootstrap.cc:492] T b5bdc27275cd4156afe767a1fb33a8bd P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:14.230481 27064 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5b44669e91a8427c8e7d96b866fdb3b5" } modify_external_catalogs: true
I20250903 20:05:14.230727 27064 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5b44669e91a8427c8e7d96b866fdb3b5" } modify_external_catalogs: true
I20250903 20:05:14.231431 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 5f27f525fdab407280aa23df1ae71b43 (DEFAULT_TABLE table=default.Table_1c679659094d4666aaaf68f76f667890 [id=61a25739b03042f7bd5bb693b68656d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.232163 27366 tablet_service.cc:1515] Processing DeleteTablet for tablet fdc77810f27a40758e063b8b1150ab71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:14.232775 27687 raft_consensus.cc:2241] T 4692dd93b76c42c4883c7e22c6032250 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:14.232825 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9509364ba3ec4e9bad09db09406a978a" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:14.233054 27687 raft_consensus.cc:2270] T 4692dd93b76c42c4883c7e22c6032250 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:14.233625 27687 ts_tablet_manager.cc:1905] T 4692dd93b76c42c4883c7e22c6032250 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:14.233587 27462 leader_election.cc:343] T 9509364ba3ec4e9bad09db09406a978a P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:14.233752 27462 leader_election.cc:304] T 9509364ba3ec4e9bad09db09406a978a P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:14.234169 27952 raft_consensus.cc:2747] T 9509364ba3ec4e9bad09db09406a978a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:14.234560 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2629f46704e4a0fa7922533928ca2ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.238085 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4af8c03c9d834405b4e25d35441db961. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.242475 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.242707 27684 ts_tablet_manager.cc:1939] T c25aaa6ff310432bb27a0695394ec420 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:14.243713 27687 ts_tablet_manager.cc:1918] T 4692dd93b76c42c4883c7e22c6032250 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:14.243867 27687 log.cc:1199] T 4692dd93b76c42c4883c7e22c6032250 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/4692dd93b76c42c4883c7e22c6032250
I20250903 20:05:14.244257 27687 ts_tablet_manager.cc:1939] T 4692dd93b76c42c4883c7e22c6032250 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:14.248766 27692 tablet_bootstrap.cc:492] T 710e3bc3c4194efbbeef6bc76ee64f24 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:14.248960 27692 ts_tablet_manager.cc:1397] T 710e3bc3c4194efbbeef6bc76ee64f24 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.270s user 0.011s sys 0.042s
I20250903 20:05:14.249776 27685 ts_tablet_manager.cc:1918] T 925c0ab4873041da82702cd9377ec7bc P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:14.249807 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet c25aaa6ff310432bb27a0695394ec420 (table default.Table_835ced1f8b194192809c3b6fbf4d1d96 [id=23f448924ec04ef08a541a14bd2ce5df]) successfully deleted
I20250903 20:05:14.249914 27685 log.cc:1199] T 925c0ab4873041da82702cd9377ec7bc P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/925c0ab4873041da82702cd9377ec7bc
I20250903 20:05:14.249922 27692 raft_consensus.cc:357] T 710e3bc3c4194efbbeef6bc76ee64f24 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.250166 27692 raft_consensus.cc:383] T 710e3bc3c4194efbbeef6bc76ee64f24 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:14.250277 27692 raft_consensus.cc:738] T 710e3bc3c4194efbbeef6bc76ee64f24 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:14.250326 27685 ts_tablet_manager.cc:1939] T 925c0ab4873041da82702cd9377ec7bc P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:14.250591 27692 consensus_queue.cc:260] T 710e3bc3c4194efbbeef6bc76ee64f24 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.250758 27684 tablet_replica.cc:331] T 9509364ba3ec4e9bad09db09406a978a P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:14.251081 27692 ts_tablet_manager.cc:1428] T 710e3bc3c4194efbbeef6bc76ee64f24 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:14.251269 27685 tablet_replica.cc:331] T 8945085bdc174e6db22a5a9477810297 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:14.251415 27692 tablet_bootstrap.cc:492] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:14.251469 27685 raft_consensus.cc:2241] T 8945085bdc174e6db22a5a9477810297 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:14.251636 27685 raft_consensus.cc:2270] T 8945085bdc174e6db22a5a9477810297 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:14.252179 27685 ts_tablet_manager.cc:1905] T 8945085bdc174e6db22a5a9477810297 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:14.252599 27693 tablet_bootstrap.cc:654] T b5bdc27275cd4156afe767a1fb33a8bd P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:14.253351 27692 tablet_bootstrap.cc:654] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:14.255510 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_83be2fe3517249a39f2aa821588442b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:14.256131 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83be2fe3517249a39f2aa821588442b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:14.256199 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f27f525fdab407280aa23df1ae71b43. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:14.261524 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet fdc77810f27a40758e063b8b1150ab71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fdc77810f27a40758e063b8b1150ab71 already in progress: creating tablet
I20250903 20:05:14.261840 27840 raft_consensus.cc:491] T cfa8a1d9011e460681443a6b77b95958 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:14.261907 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet e5136d5388bc4c2ba71e09be40e8f412 (table default.Table_9b80dba44fe8402999adc810b230d87a [id=790f57164c7d4183a3df17ba7f230d24]) successfully deleted
I20250903 20:05:14.262008 27840 raft_consensus.cc:513] T cfa8a1d9011e460681443a6b77b95958 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.262665 27840 leader_election.cc:290] T cfa8a1d9011e460681443a6b77b95958 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:14.263327 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cfa8a1d9011e460681443a6b77b95958" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:14.263844 27183 leader_election.cc:343] T cfa8a1d9011e460681443a6b77b95958 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:14.264127 27384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cfa8a1d9011e460681443a6b77b95958" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:14.265161 27184 leader_election.cc:343] T cfa8a1d9011e460681443a6b77b95958 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:14.265305 27184 leader_election.cc:304] T cfa8a1d9011e460681443a6b77b95958 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:14.265667 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 925c0ab4873041da82702cd9377ec7bc (table default.Table_1a5c8c3256c24c7f8ef8597efa53dde5 [id=7caef592f80a4d6cb6df4e2c3971de49]) successfully deleted
I20250903 20:05:14.265715 27840 raft_consensus.cc:2747] T cfa8a1d9011e460681443a6b77b95958 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:14.267971 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 512908bc4d6941ec81b145ee5795405f has no consensus state
I20250903 20:05:14.269800 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_3d20da97fa86497e9376f8aa69f1c3fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:14.270670 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1c679659094d4666aaaf68f76f667890" } modify_external_catalogs: true
I20250903 20:05:14.270835 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1c679659094d4666aaaf68f76f667890" } modify_external_catalogs: true
W20250903 20:05:14.271700 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d20da97fa86497e9376f8aa69f1c3fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:14.275965 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 4692dd93b76c42c4883c7e22c6032250 (table default.Table_7dfd9faa02434381b40a56369804413b [id=f11bdb54ed5c4b1ca1e1306d39aaf25f]) successfully deleted
I20250903 20:05:14.276929 27501 tablet_service.cc:1515] Processing DeleteTablet for tablet 710e3bc3c4194efbbeef6bc76ee64f24 with delete_type TABLET_DATA_DELETED (replaced by tablet 4be9bdc390de4a74a396003c9c3c2b71) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:14.277516 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet e5136d5388bc4c2ba71e09be40e8f412 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5136d5388bc4c2ba71e09be40e8f412 already in progress: creating tablet
W20250903 20:05:14.277906 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet f2cc35f4244a42eeb4a65de749824c11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2cc35f4244a42eeb4a65de749824c11 already in progress: creating tablet
I20250903 20:05:14.278398 28050 tablet.cc:1722] T 99a21a3e49934a79a66700c0b0d86307 P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250903 20:05:14.278772 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet fdc77810f27a40758e063b8b1150ab71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fdc77810f27a40758e063b8b1150ab71 already in progress: creating tablet
I20250903 20:05:14.278954 28049 tablet.cc:1722] T 99a21a3e49934a79a66700c0b0d86307 P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:14.285851 27684 raft_consensus.cc:2241] T 9509364ba3ec4e9bad09db09406a978a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:14.286242 27684 raft_consensus.cc:2270] T 9509364ba3ec4e9bad09db09406a978a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:14.286682 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_13dbf08231794aa085d17ca500361b14" } new_table_name: "default.Table_fa8b4f79747046748c7082603e9d2bfa" modify_external_catalogs: true
I20250903 20:05:14.232589 27357 tablet_service.cc:1468] Processing CreateTablet for tablet 9e98cd1d9bd246e182e2969d147e68bf (DEFAULT_TABLE table=default.Table_35d35c0c3a6048bc9d7def161d9e16db [id=b27f08d56c3c43e6a09a465fdda4d3db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.290990 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e98cd1d9bd246e182e2969d147e68bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.295531 27695 tablet_bootstrap.cc:492] T c9c870fb14e3469fb2979d8462b49121 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:14.295742 27695 ts_tablet_manager.cc:1397] T c9c870fb14e3469fb2979d8462b49121 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.097s user 0.007s sys 0.004s
I20250903 20:05:14.296604 27695 raft_consensus.cc:357] T c9c870fb14e3469fb2979d8462b49121 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.296814 27695 raft_consensus.cc:383] T c9c870fb14e3469fb2979d8462b49121 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:14.296911 27695 raft_consensus.cc:738] T c9c870fb14e3469fb2979d8462b49121 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:14.297099 27058 master_service.cc:948] ReplaceTablet: received request to replace tablet 466e501e424e46bc98e8bbc64f97005e from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:14.304421 28051 tablet.cc:1722] T 99a21a3e49934a79a66700c0b0d86307 P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:14.304576 27695 consensus_queue.cc:260] T c9c870fb14e3469fb2979d8462b49121 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.307955 27695 ts_tablet_manager.cc:1428] T c9c870fb14e3469fb2979d8462b49121 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.012s user 0.001s sys 0.002s
I20250903 20:05:14.310684 27695 tablet_bootstrap.cc:492] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:14.314049 27063 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 68cff0e416a54a5eb37071f5a12f07ab (table default.Table_1e3f794ae5ec4ad6a9b4351b4bda7b15 [id=71d850a0cea14b0ba337a822ad99d513]). Expected version 3 got 0
I20250903 20:05:14.316074 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 120ca61b3da742dc9a2efa483ead7e97 (DEFAULT_TABLE table=default.Table_1742550430f54b36b962c92f5c81f08a [id=a10d0a5b11484d078240afb2ad212bff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.316543 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 120ca61b3da742dc9a2efa483ead7e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.316715 27359 tablet_service.cc:1468] Processing CreateTablet for tablet 726f4d920f3e47fea41b4e12c021c988 (DEFAULT_TABLE table=default.Table_a6608fc0162446a480d8e8ecb07db28f [id=66b42d6c99de4096998cba421c5b4663]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.317473 27224 tablet_service.cc:1468] Processing CreateTablet for tablet 726f4d920f3e47fea41b4e12c021c988 (DEFAULT_TABLE table=default.Table_a6608fc0162446a480d8e8ecb07db28f [id=66b42d6c99de4096998cba421c5b4663]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.317816 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 726f4d920f3e47fea41b4e12c021c988. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.318151 27364 tablet_service.cc:1468] Processing CreateTablet for tablet 2fed2821d2644f75aff6e0ceb5944a37 (DEFAULT_TABLE table=default.Table_3cf8165949c5457cb1fa8cd412628763 [id=bb5aff0bf44b4ce0b834aa50461d1717]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.318869 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet cfa8a1d9011e460681443a6b77b95958 with delete_type TABLET_DATA_DELETED (replaced by tablet 4af8c03c9d834405b4e25d35441db961) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:14.319312 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 120ca61b3da742dc9a2efa483ead7e97 (DEFAULT_TABLE table=default.Table_1742550430f54b36b962c92f5c81f08a [id=a10d0a5b11484d078240afb2ad212bff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.319597 27226 tablet_service.cc:1468] Processing CreateTablet for tablet 2fed2821d2644f75aff6e0ceb5944a37 (DEFAULT_TABLE table=default.Table_3cf8165949c5457cb1fa8cd412628763 [id=bb5aff0bf44b4ce0b834aa50461d1717]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.320010 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fed2821d2644f75aff6e0ceb5944a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.320319 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 726f4d920f3e47fea41b4e12c021c988. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.324149 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fed2821d2644f75aff6e0ceb5944a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.327333 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 120ca61b3da742dc9a2efa483ead7e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.328229 28030 raft_consensus.cc:491] T 44db427ee53a4d0a907dfd63282d6d4f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:14.328418 28030 raft_consensus.cc:513] T 44db427ee53a4d0a907dfd63282d6d4f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.329108 28030 leader_election.cc:290] T 44db427ee53a4d0a907dfd63282d6d4f P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:14.330282 27356 tablet_service.cc:1468] Processing CreateTablet for tablet b7a6d5d8a97a4769a81ec7224d591f8c (DEFAULT_TABLE table=default.Table_b86a1d18f92c4844997ebd323da582b6 [id=21b4fd8022f6461691f87200e040e264]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.330704 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7a6d5d8a97a4769a81ec7224d591f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.331143 27063 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 8023fa7537234ce783c2bf90b575b00c (table default.Table_9ce3e6d31c314cd0bdbd59ea1260d46a [id=7032fe3b789041d380f5db114f241e29]). Expected version 1 got 0
I20250903 20:05:14.334009 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44db427ee53a4d0a907dfd63282d6d4f" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:14.334647 27184 leader_election.cc:343] T 44db427ee53a4d0a907dfd63282d6d4f P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:14.335675 27684 ts_tablet_manager.cc:1905] T 9509364ba3ec4e9bad09db09406a978a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:14.335796 27685 ts_tablet_manager.cc:1918] T 8945085bdc174e6db22a5a9477810297 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:14.335748 27495 tablet_service.cc:1468] Processing CreateTablet for tablet 726f4d920f3e47fea41b4e12c021c988 (DEFAULT_TABLE table=default.Table_a6608fc0162446a480d8e8ecb07db28f [id=66b42d6c99de4096998cba421c5b4663]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.340474 27685 log.cc:1199] T 8945085bdc174e6db22a5a9477810297 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/8945085bdc174e6db22a5a9477810297
I20250903 20:05:14.341023 27685 ts_tablet_manager.cc:1939] T 8945085bdc174e6db22a5a9477810297 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:14.342979 27685 tablet_replica.cc:331] T 4268316c28e8471a9f28807d20dbb395 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:14.343596 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 466e501e424e46bc98e8bbc64f97005e
I20250903 20:05:14.344115 27058 catalog_manager.cc:6652] ReplaceTablet: tablet 466e501e424e46bc98e8bbc64f97005e deleted and replaced by tablet 06a79706f1184fa49d10182ba669b0ab
I20250903 20:05:14.345139 27693 tablet_bootstrap.cc:492] T b5bdc27275cd4156afe767a1fb33a8bd P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:14.345293 27693 ts_tablet_manager.cc:1397] T b5bdc27275cd4156afe767a1fb33a8bd P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.115s user 0.002s sys 0.007s
W20250903 20:05:14.343600 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cd92e3dafca6419f8eba7a4ef51a911e consensus state has no leader
I20250903 20:05:14.346151 27693 raft_consensus.cc:357] T b5bdc27275cd4156afe767a1fb33a8bd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.346375 27693 raft_consensus.cc:383] T b5bdc27275cd4156afe767a1fb33a8bd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:14.346553 27693 raft_consensus.cc:738] T b5bdc27275cd4156afe767a1fb33a8bd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:14.346850 27693 consensus_queue.cc:260] T b5bdc27275cd4156afe767a1fb33a8bd P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.347502 27693 ts_tablet_manager.cc:1428] T b5bdc27275cd4156afe767a1fb33a8bd P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:14.353509 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 673c45f3f2a14da9bd306011d396eaca
I20250903 20:05:14.354233 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 7cc6cf579e2642e99fefed723b0ee900
I20250903 20:05:14.354327 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 59a1dd48d76241d7ba817070fbffd853
I20250903 20:05:14.359532 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 726f4d920f3e47fea41b4e12c021c988. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.359563 27366 tablet_service.cc:1468] Processing CreateTablet for tablet 512908bc4d6941ec81b145ee5795405f (DEFAULT_TABLE table=default.Table_d9276aa6946f41d7ac0fabc39338c23d [id=60e7bfe4938540e4a3c34b248356969f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.360131 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 512908bc4d6941ec81b145ee5795405f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.362533 27217 tablet_service.cc:1468] Processing CreateTablet for tablet b7a6d5d8a97a4769a81ec7224d591f8c (DEFAULT_TABLE table=default.Table_b86a1d18f92c4844997ebd323da582b6 [id=21b4fd8022f6461691f87200e040e264]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.363104 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7a6d5d8a97a4769a81ec7224d591f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.363073 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 2fed2821d2644f75aff6e0ceb5944a37 (DEFAULT_TABLE table=default.Table_3cf8165949c5457cb1fa8cd412628763 [id=bb5aff0bf44b4ce0b834aa50461d1717]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.363529 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fed2821d2644f75aff6e0ceb5944a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.365924 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44db427ee53a4d0a907dfd63282d6d4f" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:14.370131 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 1bf69315f07a470bb2b897ad7c590729 (DEFAULT_TABLE table=default.Table_5fb98b65f51b4d628de6bed5d36b53d1 [id=b82c8f8f0f6b415580c2b590f05c0f7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.370621 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bf69315f07a470bb2b897ad7c590729. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.371618 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 466e501e424e46bc98e8bbc64f97005e with delete_type TABLET_DATA_DELETED (replaced by tablet 06a79706f1184fa49d10182ba669b0ab) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:14.337246 27695 tablet_bootstrap.cc:654] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:14.366752 27183 leader_election.cc:343] T 44db427ee53a4d0a907dfd63282d6d4f P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:14.371891 27183 leader_election.cc:304] T 44db427ee53a4d0a907dfd63282d6d4f P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:14.372367 28030 raft_consensus.cc:2747] T 44db427ee53a4d0a907dfd63282d6d4f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:14.367174 27224 tablet_service.cc:1468] Processing CreateTablet for tablet 512908bc4d6941ec81b145ee5795405f (DEFAULT_TABLE table=default.Table_d9276aa6946f41d7ac0fabc39338c23d [id=60e7bfe4938540e4a3c34b248356969f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.369889 27226 tablet_service.cc:1468] Processing CreateTablet for tablet 22bc87d1a4804b6c80f77f9692500d63 (DEFAULT_TABLE table=default.Table_c967f8b0d1314f589eae529cc492493b [id=93b379ddcb834087a42ce197c318f764]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.372563 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet 466e501e424e46bc98e8bbc64f97005e with delete_type TABLET_DATA_DELETED (replaced by tablet 06a79706f1184fa49d10182ba669b0ab) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:14.372833 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22bc87d1a4804b6c80f77f9692500d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.373890 27220 tablet_service.cc:1468] Processing CreateTablet for tablet 2059ccbd6d884085886d7d564afe03aa (DEFAULT_TABLE table=default.Table_0bbccf098ebf46ae8d825952b89a2357 [id=91abc1158ea047849f398926e3695927]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.374416 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2059ccbd6d884085886d7d564afe03aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.374373 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 1bf69315f07a470bb2b897ad7c590729 (DEFAULT_TABLE table=default.Table_5fb98b65f51b4d628de6bed5d36b53d1 [id=b82c8f8f0f6b415580c2b590f05c0f7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.374806 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bf69315f07a470bb2b897ad7c590729. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.375689 27363 tablet_service.cc:1468] Processing CreateTablet for tablet 2059ccbd6d884085886d7d564afe03aa (DEFAULT_TABLE table=default.Table_0bbccf098ebf46ae8d825952b89a2357 [id=91abc1158ea047849f398926e3695927]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.376077 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2059ccbd6d884085886d7d564afe03aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.378474 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 512908bc4d6941ec81b145ee5795405f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.382442 27952 consensus_queue.cc:1035] T c0343d2226e345669d42375de2545986 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:14.383958 27226 tablet_service.cc:1468] Processing CreateTablet for tablet 67ca1dcf6fc7435db555392235f9900f (DEFAULT_TABLE table=default.Table_3aff8eb0c058463b831146139686b58e [id=5568f788d1414c83b4359b5adf28632a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.384542 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67ca1dcf6fc7435db555392235f9900f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.389977 27220 tablet_service.cc:1468] Processing CreateTablet for tablet 63a1f26fac3b4b939ebb79c6a07d8f8e (DEFAULT_TABLE table=default.Table_362ac8cb413e426ebf154eae4b6b5c7e [id=10ef25498f7448dbb19cd66d4c6ba713]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.391355 27497 tablet_service.cc:1468] Processing CreateTablet for tablet b7a6d5d8a97a4769a81ec7224d591f8c (DEFAULT_TABLE table=default.Table_b86a1d18f92c4844997ebd323da582b6 [id=21b4fd8022f6461691f87200e040e264]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.392431 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet 673c45f3f2a14da9bd306011d396eaca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:14.392597 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 673c45f3f2a14da9bd306011d396eaca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:14.392827 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 63a1f26fac3b4b939ebb79c6a07d8f8e (DEFAULT_TABLE table=default.Table_362ac8cb413e426ebf154eae4b6b5c7e [id=10ef25498f7448dbb19cd66d4c6ba713]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.393260 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63a1f26fac3b4b939ebb79c6a07d8f8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.393349 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63a1f26fac3b4b939ebb79c6a07d8f8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.397614 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 59a1dd48d76241d7ba817070fbffd853 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:14.398008 27356 tablet_service.cc:1468] Processing CreateTablet for tablet 22bc87d1a4804b6c80f77f9692500d63 (DEFAULT_TABLE table=default.Table_c967f8b0d1314f589eae529cc492493b [id=93b379ddcb834087a42ce197c318f764]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.399231 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet 7cc6cf579e2642e99fefed723b0ee900 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:14.400087 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet 59a1dd48d76241d7ba817070fbffd853 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:14.401232 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7a6d5d8a97a4769a81ec7224d591f8c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:14.401638 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 466e501e424e46bc98e8bbc64f97005e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 466e501e424e46bc98e8bbc64f97005e already in progress: creating tablet
W20250903 20:05:14.401867 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 673c45f3f2a14da9bd306011d396eaca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 673c45f3f2a14da9bd306011d396eaca already in progress: creating tablet
I20250903 20:05:14.402500 27685 raft_consensus.cc:2241] T 4268316c28e8471a9f28807d20dbb395 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:14.402741 27685 raft_consensus.cc:2270] T 4268316c28e8471a9f28807d20dbb395 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:14.403349 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22bc87d1a4804b6c80f77f9692500d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.403259 27500 tablet_service.cc:1468] Processing CreateTablet for tablet 512908bc4d6941ec81b145ee5795405f (DEFAULT_TABLE table=default.Table_d9276aa6946f41d7ac0fabc39338c23d [id=60e7bfe4938540e4a3c34b248356969f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.404965 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 512908bc4d6941ec81b145ee5795405f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.405181 27499 tablet_service.cc:1468] Processing CreateTablet for tablet 22bc87d1a4804b6c80f77f9692500d63 (DEFAULT_TABLE table=default.Table_c967f8b0d1314f589eae529cc492493b [id=93b379ddcb834087a42ce197c318f764]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.405864 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 1bf69315f07a470bb2b897ad7c590729 (DEFAULT_TABLE table=default.Table_5fb98b65f51b4d628de6bed5d36b53d1 [id=b82c8f8f0f6b415580c2b590f05c0f7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.406579 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bf69315f07a470bb2b897ad7c590729. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.406805 27685 ts_tablet_manager.cc:1905] T 4268316c28e8471a9f28807d20dbb395 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:14.402192 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 59a1dd48d76241d7ba817070fbffd853 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59a1dd48d76241d7ba817070fbffd853 already in progress: creating tablet
I20250903 20:05:14.410815 27058 master_service.cc:948] ReplaceTablet: received request to replace tablet 1bcb9d5f010f4297bab78c0183889238 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:14.410966 28051 tablet.cc:1722] T 9367b3a22f624c4087848099de32c080 P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:14.412617 27065 master_service.cc:948] ReplaceTablet: received request to replace tablet 8023fa7537234ce783c2bf90b575b00c from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:14.410136 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 8945085bdc174e6db22a5a9477810297 (table default.Table_a0582ccb94a64603a5fbdd0e531a9edf [id=6f7fece7a8d54d4786f1bab10da9129c]) successfully deleted
I20250903 20:05:14.414952 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet c6d67cd0c0304c77ba571b414878dad2
I20250903 20:05:14.415444 27059 catalog_manager.cc:6652] ReplaceTablet: tablet c6d67cd0c0304c77ba571b414878dad2 deleted and replaced by tablet 886b8783aa9d4618aad498d4a1115843
I20250903 20:05:14.415759 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ab7e78ea930c4d88bf2e62caa36ca317"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:14.416338 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab7e78ea930c4d88bf2e62caa36ca317 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:14.417614 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 5f27f525fdab407280aa23df1ae71b43
I20250903 20:05:14.418329 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1e6e2159113540b7944963d2140ab460" } new_table_name: "default.Table_b9a8db9323cd4f2286256cf26bcacbac" modify_external_catalogs: true
I20250903 20:05:14.419620 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22bc87d1a4804b6c80f77f9692500d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.419657 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c005844d740343c0a67ba9977747f574" } new_table_name: "default.Table_bd1102072e4a421d9231b7dee60b6232" modify_external_catalogs: true
I20250903 20:05:14.420683 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 2059ccbd6d884085886d7d564afe03aa (DEFAULT_TABLE table=default.Table_0bbccf098ebf46ae8d825952b89a2357 [id=91abc1158ea047849f398926e3695927]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.421139 27496 tablet_service.cc:1515] Processing DeleteTablet for tablet 466e501e424e46bc98e8bbc64f97005e with delete_type TABLET_DATA_DELETED (replaced by tablet 06a79706f1184fa49d10182ba669b0ab) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:14.422024 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_f038423efb3f4fd5bba9911713436b97"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:14.422117 27501 tablet_service.cc:1468] Processing CreateTablet for tablet 120ca61b3da742dc9a2efa483ead7e97 (DEFAULT_TABLE table=default.Table_1742550430f54b36b962c92f5c81f08a [id=a10d0a5b11484d078240afb2ad212bff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.422415 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_279a72f111474b7b86aa09094dc82ac6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:14.422605 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f038423efb3f4fd5bba9911713436b97 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:14.422873 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_279a72f111474b7b86aa09094dc82ac6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:14.423931 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_d9105c35c28247f1a71866f2096d93f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:14.424419 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9105c35c28247f1a71866f2096d93f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:14.424566 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 607c9a9cbc5540e5bd0fa436fcba564c
I20250903 20:05:14.425244 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_16b62656e76142339c0a42ca5a7f5e6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:14.425786 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_16b62656e76142339c0a42ca5a7f5e6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:14.429945 27496 tablet_service.cc:1468] Processing CreateTablet for tablet 67ca1dcf6fc7435db555392235f9900f (DEFAULT_TABLE table=default.Table_3aff8eb0c058463b831146139686b58e [id=5568f788d1414c83b4359b5adf28632a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.430439 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2059ccbd6d884085886d7d564afe03aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.432561 27363 tablet_service.cc:1515] Processing DeleteTablet for tablet 7cc6cf579e2642e99fefed723b0ee900 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:14.433571 27498 tablet_service.cc:1515] Processing DeleteTablet for tablet 673c45f3f2a14da9bd306011d396eaca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:14.435572 27063 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_a9a6eefd24394081aad7a7b56c2613f5 [id=428657b402d748c0a9e3942d13342b4e] alter complete (version 2)
I20250903 20:05:14.436935 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 8023fa7537234ce783c2bf90b575b00c
W20250903 20:05:14.437059 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8d2d64f103fc44d7b390559b92cbf4b4 consensus state has no leader
I20250903 20:05:14.437446 27065 catalog_manager.cc:6652] ReplaceTablet: tablet 8023fa7537234ce783c2bf90b575b00c deleted and replaced by tablet 950a6fcd35f64f2d889831fe9cdf503a
I20250903 20:05:14.437342 27366 tablet_service.cc:1468] Processing CreateTablet for tablet 67ca1dcf6fc7435db555392235f9900f (DEFAULT_TABLE table=default.Table_3aff8eb0c058463b831146139686b58e [id=5568f788d1414c83b4359b5adf28632a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.438529 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67ca1dcf6fc7435db555392235f9900f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.440033 27503 tablet_service.cc:1515] Processing DeleteTablet for tablet 59a1dd48d76241d7ba817070fbffd853 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:14.442807 27693 tablet_bootstrap.cc:492] T 7ba18e3f8cb148d484010cb4e72510b2 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:14.454239 27502 tablet_service.cc:1468] Processing CreateTablet for tablet 63a1f26fac3b4b939ebb79c6a07d8f8e (DEFAULT_TABLE table=default.Table_362ac8cb413e426ebf154eae4b6b5c7e [id=10ef25498f7448dbb19cd66d4c6ba713]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.454668 28030 raft_consensus.cc:491] T f7ab722c37264973b204cd4f1f95052d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:14.455379 28030 raft_consensus.cc:513] T f7ab722c37264973b204cd4f1f95052d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.454244 27503 tablet_service.cc:1515] Processing DeleteTablet for tablet 7cc6cf579e2642e99fefed723b0ee900 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:14.456054 28030 leader_election.cc:290] T f7ab722c37264973b204cd4f1f95052d P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:14.456617 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67ca1dcf6fc7435db555392235f9900f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:14.458115 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 7cc6cf579e2642e99fefed723b0ee900 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7cc6cf579e2642e99fefed723b0ee900 already in progress: creating tablet
I20250903 20:05:14.458921 27363 tablet_service.cc:1515] Processing DeleteTablet for tablet 607c9a9cbc5540e5bd0fa436fcba564c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:14.459731 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7ab722c37264973b204cd4f1f95052d" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:14.460211 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet 8023fa7537234ce783c2bf90b575b00c with delete_type TABLET_DATA_DELETED (replaced by tablet 950a6fcd35f64f2d889831fe9cdf503a) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:14.460256 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 607c9a9cbc5540e5bd0fa436fcba564c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 607c9a9cbc5540e5bd0fa436fcba564c already in progress: creating tablet
I20250903 20:05:14.460788 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63a1f26fac3b4b939ebb79c6a07d8f8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.463125 27503 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f27f525fdab407280aa23df1ae71b43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:14.463290 27498 tablet_service.cc:1515] Processing DeleteTablet for tablet c6d67cd0c0304c77ba571b414878dad2 with delete_type TABLET_DATA_DELETED (replaced by tablet 886b8783aa9d4618aad498d4a1115843) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:14.466284 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7ab722c37264973b204cd4f1f95052d" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:14.467283 27501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 120ca61b3da742dc9a2efa483ead7e97. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:14.467830 27183 leader_election.cc:343] T f7ab722c37264973b204cd4f1f95052d P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:14.470297 27224 tablet_service.cc:1515] Processing DeleteTablet for tablet 607c9a9cbc5540e5bd0fa436fcba564c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:14.470534 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f27f525fdab407280aa23df1ae71b43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:14.470669 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet c6d67cd0c0304c77ba571b414878dad2 with delete_type TABLET_DATA_DELETED (replaced by tablet 886b8783aa9d4618aad498d4a1115843) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:14.471297 27224 tablet_service.cc:1515] Processing DeleteTablet for tablet 8023fa7537234ce783c2bf90b575b00c with delete_type TABLET_DATA_DELETED (replaced by tablet 950a6fcd35f64f2d889831fe9cdf503a) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:14.471649 27184 leader_election.cc:343] T f7ab722c37264973b204cd4f1f95052d P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:14.471771 27184 leader_election.cc:304] T f7ab722c37264973b204cd4f1f95052d P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:14.472329 28030 raft_consensus.cc:2747] T f7ab722c37264973b204cd4f1f95052d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:14.472414 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f27f525fdab407280aa23df1ae71b43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:14.472648 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet c9c870fb14e3469fb2979d8462b49121 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:14.472906 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet 44db427ee53a4d0a907dfd63282d6d4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:14.473327 28030 raft_consensus.cc:491] T c9c870fb14e3469fb2979d8462b49121 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:14.473501 28030 raft_consensus.cc:513] T c9c870fb14e3469fb2979d8462b49121 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.474282 28030 leader_election.cc:290] T c9c870fb14e3469fb2979d8462b49121 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:14.476677 27693 tablet_bootstrap.cc:654] T 7ba18e3f8cb148d484010cb4e72510b2 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:14.479857 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9c870fb14e3469fb2979d8462b49121" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:14.480481 27184 leader_election.cc:343] T c9c870fb14e3469fb2979d8462b49121 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:14.480986 27503 tablet_service.cc:1515] Processing DeleteTablet for tablet 8023fa7537234ce783c2bf90b575b00c with delete_type TABLET_DATA_DELETED (replaced by tablet 950a6fcd35f64f2d889831fe9cdf503a) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:14.481204 27498 tablet_service.cc:1515] Processing DeleteTablet for tablet 607c9a9cbc5540e5bd0fa436fcba564c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:14.481213 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 8023fa7537234ce783c2bf90b575b00c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8023fa7537234ce783c2bf90b575b00c already in progress: creating tablet
W20250903 20:05:14.481603 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 5f27f525fdab407280aa23df1ae71b43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f27f525fdab407280aa23df1ae71b43 already in progress: creating tablet
I20250903 20:05:14.483794 27695 tablet_bootstrap.cc:492] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:14.484007 27695 ts_tablet_manager.cc:1397] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.173s user 0.008s sys 0.012s
I20250903 20:05:14.484915 27695 raft_consensus.cc:357] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.485235 27695 raft_consensus.cc:383] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:14.485347 27695 raft_consensus.cc:738] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:14.485410 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet c6d67cd0c0304c77ba571b414878dad2 with delete_type TABLET_DATA_DELETED (replaced by tablet 886b8783aa9d4618aad498d4a1115843) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:14.485735 27695 consensus_queue.cc:260] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:14.486511 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a223f75045345ae833f52bcc8f239f9 consensus state has no leader
I20250903 20:05:14.487002 27695 ts_tablet_manager.cc:1428] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250903 20:05:14.487429 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9c870fb14e3469fb2979d8462b49121" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:14.488039 27183 leader_election.cc:343] T c9c870fb14e3469fb2979d8462b49121 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:14.488184 27183 leader_election.cc:304] T c9c870fb14e3469fb2979d8462b49121 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:14.494122 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_2746c6354cbc45c296e45155908acb77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:14.494750 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2746c6354cbc45c296e45155908acb77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:14.495343 27062 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 9367b3a22f624c4087848099de32c080 (table default.Table_08918e8b515e4487bf4aee0a33234bd5 [id=891b259c1c6b44e2a82dc982c24e6cb6]). Expected version 1 got 0
I20250903 20:05:14.496114 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_60d0c49d795c4cad954d2b7327485eb9" } new_table_name: "default.Table_a98f50198fb2489fad7fa669a4250ee7" modify_external_catalogs: true
I20250903 20:05:14.497354 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_36a3bbdf545442dcacd498d928fa996b" } new_table_name: "default.Table_75f638a7c40e48e584a02949e6f864fd" modify_external_catalogs: true
I20250903 20:05:14.497748 28030 raft_consensus.cc:2747] T c9c870fb14e3469fb2979d8462b49121 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:14.498078 27695 tablet_bootstrap.cc:492] T 73f65e7515994b898eec2ea23903aff4 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:14.498529 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b88ff2c2e61049d4a11af5676d28327a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:14.499058 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b88ff2c2e61049d4a11af5676d28327a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:14.499773 27695 tablet_bootstrap.cc:654] T 73f65e7515994b898eec2ea23903aff4 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:14.501756 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet 29581a9dba46479884a0e6e01b7923de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:14.502334 27687 tablet_replica.cc:331] T 29581a9dba46479884a0e6e01b7923de P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
W20250903 20:05:14.502313 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet c6d67cd0c0304c77ba571b414878dad2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6d67cd0c0304c77ba571b414878dad2 already in progress: creating tablet
I20250903 20:05:14.502559 27687 raft_consensus.cc:2241] T 29581a9dba46479884a0e6e01b7923de P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:14.502782 27687 raft_consensus.cc:2270] T 29581a9dba46479884a0e6e01b7923de P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:14.502974 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_0c416f6e2e1b4ae4941c2e5613f16bda"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:14.503454 27687 ts_tablet_manager.cc:1905] T 29581a9dba46479884a0e6e01b7923de P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:14.503486 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c416f6e2e1b4ae4941c2e5613f16bda in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:14.510098 27462 consensus_peers.cc:489] T c0343d2226e345669d42375de2545986 P e72ef1dbcc5143cf9f04a808f101c80f -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:14.511078 27695 tablet_bootstrap.cc:492] T 73f65e7515994b898eec2ea23903aff4 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:14.511268 27695 ts_tablet_manager.cc:1397] T 73f65e7515994b898eec2ea23903aff4 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.004s
I20250903 20:05:14.512156 27695 raft_consensus.cc:357] T 73f65e7515994b898eec2ea23903aff4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:14.512441 27695 raft_consensus.cc:383] T 73f65e7515994b898eec2ea23903aff4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:14.512538 27695 raft_consensus.cc:738] T 73f65e7515994b898eec2ea23903aff4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:14.512849 27695 consensus_queue.cc:260] T 73f65e7515994b898eec2ea23903aff4 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:14.514676 27695 ts_tablet_manager.cc:1428] T 73f65e7515994b898eec2ea23903aff4 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:14.515000 27695 tablet_bootstrap.cc:492] T 31342583c026438cbbae14106e75bbe5 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:14.516767 27695 tablet_bootstrap.cc:654] T 31342583c026438cbbae14106e75bbe5 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:14.520573 27064 master_service.cc:948] ReplaceTablet: received request to replace tablet 8d2d64f103fc44d7b390559b92cbf4b4 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:14.522886 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_14e02c1be9c74bd99f4184faa6753f52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:14.523610 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14e02c1be9c74bd99f4184faa6753f52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:14.529522 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1bad3c509460422396a17a3c4ae3d1e8 consensus state has no leader
W20250903 20:05:14.532424 27059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 90baf83357494a0aae325908c118b471 consensus state has no leader
I20250903 20:05:14.533215 27357 tablet_service.cc:1468] Processing CreateTablet for tablet a0f79831866b40c49c5b27ce2014c3f6 (DEFAULT_TABLE table=default.Table_3d20da97fa86497e9376f8aa69f1c3fd [id=35b0c6c60b9b43b5ad6c376284f45da5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.533914 27062 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 2a5c74a281b44019872a6442d41bc373 (table default.Table_5c08431ef8e04a6aab9b96ab78615d2f [id=a9ddfed19dab46a7bb4e049483e3996f]). Expected version 2 got 0
I20250903 20:05:14.539562 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet 73f65e7515994b898eec2ea23903aff4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:14.542889 27220 tablet_service.cc:1468] Processing CreateTablet for tablet a0f79831866b40c49c5b27ce2014c3f6 (DEFAULT_TABLE table=default.Table_3d20da97fa86497e9376f8aa69f1c3fd [id=35b0c6c60b9b43b5ad6c376284f45da5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.543366 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0f79831866b40c49c5b27ce2014c3f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.543809 27364 tablet_service.cc:1468] Processing CreateTablet for tablet 567630e2a93645e58f9bf05ffff87b32 (DEFAULT_TABLE table=default.Table_f2d0d1d8bddc492dbccd2c5c0a4e1a16 [id=3348ac5023a64fe3b19aed6c9471d580]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.544932 27692 tablet_bootstrap.cc:492] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:14.545112 27692 ts_tablet_manager.cc:1397] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.294s user 0.008s sys 0.021s
I20250903 20:05:14.546018 27692 raft_consensus.cc:357] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.546205 27692 raft_consensus.cc:383] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:14.546327 27692 raft_consensus.cc:738] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:14.546849 27226 tablet_service.cc:1468] Processing CreateTablet for tablet 567630e2a93645e58f9bf05ffff87b32 (DEFAULT_TABLE table=default.Table_f2d0d1d8bddc492dbccd2c5c0a4e1a16 [id=3348ac5023a64fe3b19aed6c9471d580]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.547255 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 567630e2a93645e58f9bf05ffff87b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.547370 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 541278ca9ffb4e41b341139c2faa337f (DEFAULT_TABLE table=default.Table_01215977307c453e9236e5bd72bc8427 [id=7ae71054c4d34d66a210e7c8f7460d4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.548034 27685 ts_tablet_manager.cc:1918] T 4268316c28e8471a9f28807d20dbb395 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:14.548158 27685 log.cc:1199] T 4268316c28e8471a9f28807d20dbb395 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/4268316c28e8471a9f28807d20dbb395
I20250903 20:05:14.548614 27685 ts_tablet_manager.cc:1939] T 4268316c28e8471a9f28807d20dbb395 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:14.549048 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 1bcb9d5f010f4297bab78c0183889238
I20250903 20:05:14.549541 27058 catalog_manager.cc:6652] ReplaceTablet: tablet 1bcb9d5f010f4297bab78c0183889238 deleted and replaced by tablet 7a9e55f0811345f7afa71645f38ebdaf
I20250903 20:05:14.549731 27685 tablet_replica.cc:331] T e20fcd28c3ae4355a033d2b8429f84e2 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:14.549932 27685 raft_consensus.cc:2241] T e20fcd28c3ae4355a033d2b8429f84e2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:14.550083 27685 raft_consensus.cc:2270] T e20fcd28c3ae4355a033d2b8429f84e2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:14.550177 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 4268316c28e8471a9f28807d20dbb395 (table default.Table_cdb258f01fd94ff48ddd75c368e68d6c [id=f70d7ea93e154e6199f6766316353862]) successfully deleted
W20250903 20:05:14.550500 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet f2cc35f4244a42eeb4a65de749824c11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2cc35f4244a42eeb4a65de749824c11 already in progress: creating tablet
I20250903 20:05:14.550930 27693 tablet_bootstrap.cc:492] T 7ba18e3f8cb148d484010cb4e72510b2 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:14.551182 27693 ts_tablet_manager.cc:1397] T 7ba18e3f8cb148d484010cb4e72510b2 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.108s user 0.005s sys 0.000s
I20250903 20:05:14.551288 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bcb9d5f010f4297bab78c0183889238 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a9e55f0811345f7afa71645f38ebdaf) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:14.551762 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 8d2d64f103fc44d7b390559b92cbf4b4
I20250903 20:05:14.552218 27064 catalog_manager.cc:6652] ReplaceTablet: tablet 8d2d64f103fc44d7b390559b92cbf4b4 deleted and replaced by tablet a38f65e992884635aae9942681c0dc4f
I20250903 20:05:14.552139 27693 raft_consensus.cc:357] T 7ba18e3f8cb148d484010cb4e72510b2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.552412 27693 raft_consensus.cc:383] T 7ba18e3f8cb148d484010cb4e72510b2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:14.553488 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d2d64f103fc44d7b390559b92cbf4b4 with delete_type TABLET_DATA_DELETED (replaced by tablet a38f65e992884635aae9942681c0dc4f) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:14.553077 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d2d64f103fc44d7b390559b92cbf4b4 with delete_type TABLET_DATA_DELETED (replaced by tablet a38f65e992884635aae9942681c0dc4f) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:14.552502 27693 raft_consensus.cc:738] T 7ba18e3f8cb148d484010cb4e72510b2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:14.554494 27693 consensus_queue.cc:260] T 7ba18e3f8cb148d484010cb4e72510b2 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.557160 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet b5bdc27275cd4156afe767a1fb33a8bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:14.557453 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 46b5d491888147bbb101963641fd05f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:14.557721 27363 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bcb9d5f010f4297bab78c0183889238 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a9e55f0811345f7afa71645f38ebdaf) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:14.559166 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 541278ca9ffb4e41b341139c2faa337f (DEFAULT_TABLE table=default.Table_01215977307c453e9236e5bd72bc8427 [id=7ae71054c4d34d66a210e7c8f7460d4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.559674 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 541278ca9ffb4e41b341139c2faa337f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.559682 27693 ts_tablet_manager.cc:1428] T 7ba18e3f8cb148d484010cb4e72510b2 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250903 20:05:14.563262 27693 tablet_bootstrap.cc:492] T c0343d2226e345669d42375de2545986 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:14.573132 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0f79831866b40c49c5b27ce2014c3f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.581813 27695 tablet_bootstrap.cc:492] T 31342583c026438cbbae14106e75bbe5 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:14.581992 27695 ts_tablet_manager.cc:1397] T 31342583c026438cbbae14106e75bbe5 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.067s user 0.006s sys 0.012s
W20250903 20:05:14.585466 27053 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 673c45f3f2a14da9bd306011d396eaca (table default.Table_43ea308240ac4628a72434608a22b975 [id=8de1ad04c15648d9bcccfc56efc9e7fb], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 673c45f3f2a14da9bd306011d396eaca consensus state has no leader
I20250903 20:05:14.585880 27687 ts_tablet_manager.cc:1918] T 29581a9dba46479884a0e6e01b7923de P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:14.586045 27687 log.cc:1199] T 29581a9dba46479884a0e6e01b7923de P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/29581a9dba46479884a0e6e01b7923de
I20250903 20:05:14.553321 27507 tablet_service.cc:1468] Processing CreateTablet for tablet a0f79831866b40c49c5b27ce2014c3f6 (DEFAULT_TABLE table=default.Table_3d20da97fa86497e9376f8aa69f1c3fd [id=35b0c6c60b9b43b5ad6c376284f45da5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.586670 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0f79831866b40c49c5b27ce2014c3f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.589165 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 541278ca9ffb4e41b341139c2faa337f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.596015 27692 consensus_queue.cc:260] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.598423 27220 tablet_service.cc:1468] Processing CreateTablet for tablet 8fff36d5a80e45af8a256c3908555e08 (DEFAULT_TABLE table=default.Table_644507bb34724729854f122d8191cf82 [id=5616ab4744594f75b25cc4b7db7c3797]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.578807 27363 tablet_service.cc:1468] Processing CreateTablet for tablet 8fff36d5a80e45af8a256c3908555e08 (DEFAULT_TABLE table=default.Table_644507bb34724729854f122d8191cf82 [id=5616ab4744594f75b25cc4b7db7c3797]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.599185 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fff36d5a80e45af8a256c3908555e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.599190 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fff36d5a80e45af8a256c3908555e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.580068 27356 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ba18e3f8cb148d484010cb4e72510b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:14.602429 28005 catalog_manager.cc:1261] Loaded cluster ID: 07b8834eda114abdb992ba4c63cadc24
I20250903 20:05:14.603578 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 567630e2a93645e58f9bf05ffff87b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.607870 28005 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2: loading cluster ID for follower catalog manager: success
I20250903 20:05:14.562868 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bcb9d5f010f4297bab78c0183889238 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a9e55f0811345f7afa71645f38ebdaf) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:14.563570 27496 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d2d64f103fc44d7b390559b92cbf4b4 with delete_type TABLET_DATA_DELETED (replaced by tablet a38f65e992884635aae9942681c0dc4f) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:14.564662 27684 ts_tablet_manager.cc:1918] T 9509364ba3ec4e9bad09db09406a978a P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:14.568749 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 541278ca9ffb4e41b341139c2faa337f (DEFAULT_TABLE table=default.Table_01215977307c453e9236e5bd72bc8427 [id=7ae71054c4d34d66a210e7c8f7460d4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.611110 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 541278ca9ffb4e41b341139c2faa337f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.612252 27693 tablet_bootstrap.cc:654] T c0343d2226e345669d42375de2545986 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:14.614670 27684 log.cc:1199] T 9509364ba3ec4e9bad09db09406a978a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/9509364ba3ec4e9bad09db09406a978a
I20250903 20:05:14.615176 27684 ts_tablet_manager.cc:1939] T 9509364ba3ec4e9bad09db09406a978a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:14.616514 27684 tablet_replica.cc:331] T 710e3bc3c4194efbbeef6bc76ee64f24 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:14.616830 27684 raft_consensus.cc:2241] T 710e3bc3c4194efbbeef6bc76ee64f24 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:14.617125 27684 raft_consensus.cc:2270] T 710e3bc3c4194efbbeef6bc76ee64f24 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:14.597569 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 8fff36d5a80e45af8a256c3908555e08 (DEFAULT_TABLE table=default.Table_644507bb34724729854f122d8191cf82 [id=5616ab4744594f75b25cc4b7db7c3797]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.553807 27495 tablet_service.cc:1468] Processing CreateTablet for tablet 567630e2a93645e58f9bf05ffff87b32 (DEFAULT_TABLE table=default.Table_f2d0d1d8bddc492dbccd2c5c0a4e1a16 [id=3348ac5023a64fe3b19aed6c9471d580]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.622241 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 567630e2a93645e58f9bf05ffff87b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.621763 27693 tablet_bootstrap.cc:492] T c0343d2226e345669d42375de2545986 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:14.622731 27693 ts_tablet_manager.cc:1397] T c0343d2226e345669d42375de2545986 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.060s user 0.006s sys 0.004s
I20250903 20:05:14.623780 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fff36d5a80e45af8a256c3908555e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.625263 27685 ts_tablet_manager.cc:1905] T e20fcd28c3ae4355a033d2b8429f84e2 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:14.625555 27693 raft_consensus.cc:357] T c0343d2226e345669d42375de2545986 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.625831 27693 raft_consensus.cc:383] T c0343d2226e345669d42375de2545986 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:14.625964 27693 raft_consensus.cc:738] T c0343d2226e345669d42375de2545986 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:14.622238 27684 ts_tablet_manager.cc:1905] T 710e3bc3c4194efbbeef6bc76ee64f24 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:14.626365 27693 consensus_queue.cc:260] T c0343d2226e345669d42375de2545986 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.634002 27693 ts_tablet_manager.cc:1428] T c0343d2226e345669d42375de2545986 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250903 20:05:14.637081 27693 tablet_bootstrap.cc:492] T 46e3a65118b744298b5536f88c22f1db P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:14.638849 27692 ts_tablet_manager.cc:1428] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.094s user 0.002s sys 0.000s
I20250903 20:05:14.639169 27693 tablet_bootstrap.cc:654] T 46e3a65118b744298b5536f88c22f1db P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:14.639719 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 9509364ba3ec4e9bad09db09406a978a (table default.Table_03ce0e151fcc4259ba52504c9c5ddfcd [id=f261e84232234244824d0232efbb4399]) successfully deleted
I20250903 20:05:14.643074 27366 tablet_service.cc:1468] Processing CreateTablet for tablet 976322efa9464ed7a2175d3e3de51dab (DEFAULT_TABLE table=default.Table_2f6f89e1dd48493f9e989d4526caffed [id=4e05a1e86a4d4319b51f2ff36f3db76b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.643710 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 976322efa9464ed7a2175d3e3de51dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.644493 27692 tablet_bootstrap.cc:492] T cfa8a1d9011e460681443a6b77b95958 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:14.646382 27692 tablet_bootstrap.cc:654] T cfa8a1d9011e460681443a6b77b95958 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:14.648104 27693 tablet_bootstrap.cc:492] T 46e3a65118b744298b5536f88c22f1db P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:14.648299 27693 ts_tablet_manager.cc:1397] T 46e3a65118b744298b5536f88c22f1db P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20250903 20:05:14.649187 27693 raft_consensus.cc:357] T 46e3a65118b744298b5536f88c22f1db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.649423 27693 raft_consensus.cc:383] T 46e3a65118b744298b5536f88c22f1db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:14.649519 27693 raft_consensus.cc:738] T 46e3a65118b744298b5536f88c22f1db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:14.649845 27693 consensus_queue.cc:260] T 46e3a65118b744298b5536f88c22f1db P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.651758 27693 ts_tablet_manager.cc:1428] T 46e3a65118b744298b5536f88c22f1db P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250903 20:05:14.652123 27693 tablet_bootstrap.cc:492] T 76742d0df44c43729197f7e09f96cad8 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:14.657425 27693 tablet_bootstrap.cc:654] T 76742d0df44c43729197f7e09f96cad8 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:14.658023 27695 raft_consensus.cc:357] T 31342583c026438cbbae14106e75bbe5 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.658782 27695 raft_consensus.cc:383] T 31342583c026438cbbae14106e75bbe5 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:14.658890 27695 raft_consensus.cc:738] T 31342583c026438cbbae14106e75bbe5 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:14.659510 27695 consensus_queue.cc:260] T 31342583c026438cbbae14106e75bbe5 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.660096 27695 ts_tablet_manager.cc:1428] T 31342583c026438cbbae14106e75bbe5 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.078s user 0.003s sys 0.000s
I20250903 20:05:14.660808 28030 raft_consensus.cc:491] T 8023fa7537234ce783c2bf90b575b00c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:14.661358 28030 raft_consensus.cc:513] T 8023fa7537234ce783c2bf90b575b00c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.662055 27692 tablet_bootstrap.cc:492] T cfa8a1d9011e460681443a6b77b95958 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:14.662313 27692 ts_tablet_manager.cc:1397] T cfa8a1d9011e460681443a6b77b95958 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.001s
I20250903 20:05:14.662735 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8023fa7537234ce783c2bf90b575b00c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:14.663403 27184 leader_election.cc:343] T 8023fa7537234ce783c2bf90b575b00c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:14.663501 27692 raft_consensus.cc:357] T cfa8a1d9011e460681443a6b77b95958 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.664672 27692 raft_consensus.cc:383] T cfa8a1d9011e460681443a6b77b95958 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:14.664778 27692 raft_consensus.cc:738] T cfa8a1d9011e460681443a6b77b95958 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:14.665076 27692 consensus_queue.cc:260] T cfa8a1d9011e460681443a6b77b95958 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.666646 27692 ts_tablet_manager.cc:1428] T cfa8a1d9011e460681443a6b77b95958 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:14.667014 27692 tablet_bootstrap.cc:492] T 68cff0e416a54a5eb37071f5a12f07ab P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:14.668525 27357 tablet_service.cc:1468] Processing CreateTablet for tablet a253e3399622409e92299069b19d3698 (DEFAULT_TABLE table=default.Table_83be2fe3517249a39f2aa821588442b9 [id=2218f11e07d243adaa3a4d24e521b1b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.668998 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a253e3399622409e92299069b19d3698. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.669145 27692 tablet_bootstrap.cc:654] T 68cff0e416a54a5eb37071f5a12f07ab P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:14.669725 27364 tablet_service.cc:1468] Processing CreateTablet for tablet 5002b7e8118c49b6a247cb8dc4ca1aea (DEFAULT_TABLE table=default.Table_b20633c3b14e463aac0b7bd917c3ab9b [id=c6a148f02792496e8d8f2bb2dfc5e69b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.670136 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5002b7e8118c49b6a247cb8dc4ca1aea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.673499 28030 leader_election.cc:290] T 8023fa7537234ce783c2bf90b575b00c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:14.674980 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8023fa7537234ce783c2bf90b575b00c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:14.676108 27687 ts_tablet_manager.cc:1939] T 29581a9dba46479884a0e6e01b7923de P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:14.677429 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 29581a9dba46479884a0e6e01b7923de (table default.Table_7e75298df3b54e3794ab39c8b0fbffc7 [id=cf5354e1f11f4a228ea25ec2baa569a4]) successfully deleted
I20250903 20:05:14.677554 27687 tablet_replica.cc:331] T b5bdc27275cd4156afe767a1fb33a8bd P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:14.677774 27687 raft_consensus.cc:2241] T b5bdc27275cd4156afe767a1fb33a8bd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:14.677959 27687 raft_consensus.cc:2270] T b5bdc27275cd4156afe767a1fb33a8bd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:14.678066 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 8d2d64f103fc44d7b390559b92cbf4b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d2d64f103fc44d7b390559b92cbf4b4 already in progress: creating tablet
I20250903 20:05:14.678889 27687 ts_tablet_manager.cc:1905] T b5bdc27275cd4156afe767a1fb33a8bd P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:14.682562 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_4810ae9efbde43bc8c237c6e135aa5aa" } modify_external_catalogs: true
I20250903 20:05:14.682824 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_4810ae9efbde43bc8c237c6e135aa5aa" } modify_external_catalogs: true
I20250903 20:05:14.682895 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_5bc4b150c12940db90c7800f5d07b9ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:14.683532 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bc4b150c12940db90c7800f5d07b9ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:14.684393 27693 tablet_bootstrap.cc:492] T 76742d0df44c43729197f7e09f96cad8 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:14.684558 27693 ts_tablet_manager.cc:1397] T 76742d0df44c43729197f7e09f96cad8 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.000s
I20250903 20:05:14.685305 27693 raft_consensus.cc:357] T 76742d0df44c43729197f7e09f96cad8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.685549 27693 raft_consensus.cc:383] T 76742d0df44c43729197f7e09f96cad8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:14.685635 27693 raft_consensus.cc:738] T 76742d0df44c43729197f7e09f96cad8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:14.685843 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_37d794e122f2457294c1eb6115be9472" } modify_external_catalogs: true
I20250903 20:05:14.686012 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_37d794e122f2457294c1eb6115be9472" } modify_external_catalogs: true
I20250903 20:05:14.685990 27693 consensus_queue.cc:260] T 76742d0df44c43729197f7e09f96cad8 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.688026 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_943bea896c744410a2a4d80ffb9ca439" } modify_external_catalogs: true
I20250903 20:05:14.688215 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_943bea896c744410a2a4d80ffb9ca439" } modify_external_catalogs: true
I20250903 20:05:14.688359 27065 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_644507bb34724729854f122d8191cf82" } modify_external_catalogs: true
I20250903 20:05:14.688537 27065 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_644507bb34724729854f122d8191cf82" } modify_external_catalogs: true
I20250903 20:05:14.690001 27059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f2d0d1d8bddc492dbccd2c5c0a4e1a16" } modify_external_catalogs: true
I20250903 20:05:14.690254 27059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f2d0d1d8bddc492dbccd2c5c0a4e1a16" } modify_external_catalogs: true
I20250903 20:05:14.691617 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_cdb258f01fd94ff48ddd75c368e68d6c" } new_table_name: "default.Table_6ab02d2c339e463fbed5c3429c7c4ec9" modify_external_catalogs: true
I20250903 20:05:14.693030 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b5110c69d0364c50a92330c1824b8416"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:14.693593 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5110c69d0364c50a92330c1824b8416 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:14.694720 28005 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2: acquiring CA information for follower catalog manager: success
I20250903 20:05:14.695226 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3aff8eb0c058463b831146139686b58e" } new_table_name: "default.Table_bd23bbbddf114a4286a815ee4468be81" modify_external_catalogs: true
I20250903 20:05:14.696159 27693 ts_tablet_manager.cc:1428] T 76742d0df44c43729197f7e09f96cad8 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250903 20:05:14.697383 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 976322efa9464ed7a2175d3e3de51dab (DEFAULT_TABLE table=default.Table_2f6f89e1dd48493f9e989d4526caffed [id=4e05a1e86a4d4319b51f2ff36f3db76b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.697873 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 976322efa9464ed7a2175d3e3de51dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.699245 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 283a3afed7414a958b2dc4176f00bb01
I20250903 20:05:14.700060 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_61ee151cc2184644b05a60d8feb9e3c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:14.700680 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61ee151cc2184644b05a60d8feb9e3c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:14.701287 27693 tablet_bootstrap.cc:492] T 710e3bc3c4194efbbeef6bc76ee64f24 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:14.703333 27693 tablet_bootstrap.cc:654] T 710e3bc3c4194efbbeef6bc76ee64f24 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:14.703951 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet 283a3afed7414a958b2dc4176f00bb01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:14.704409 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 07565ea2b5f34dcfb407ff3bfde17243
I20250903 20:05:14.705296 27692 tablet_bootstrap.cc:492] T 68cff0e416a54a5eb37071f5a12f07ab P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:14.705452 27692 ts_tablet_manager.cc:1397] T 68cff0e416a54a5eb37071f5a12f07ab P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.002s
I20250903 20:05:14.705720 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet 07565ea2b5f34dcfb407ff3bfde17243 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:14.707041 27692 raft_consensus.cc:357] T 68cff0e416a54a5eb37071f5a12f07ab P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.707232 27692 raft_consensus.cc:383] T 68cff0e416a54a5eb37071f5a12f07ab P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:14.707321 27692 raft_consensus.cc:738] T 68cff0e416a54a5eb37071f5a12f07ab P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:14.710282 27687 ts_tablet_manager.cc:1918] T b5bdc27275cd4156afe767a1fb33a8bd P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:14.710438 27687 log.cc:1199] T b5bdc27275cd4156afe767a1fb33a8bd P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/b5bdc27275cd4156afe767a1fb33a8bd
I20250903 20:05:14.710494 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 8fff36d5a80e45af8a256c3908555e08
I20250903 20:05:14.710780 27687 ts_tablet_manager.cc:1939] T b5bdc27275cd4156afe767a1fb33a8bd P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:14.711966 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 06a79706f1184fa49d10182ba669b0ab (DEFAULT_TABLE table=default.Table_83ea6dfcee144a4fbc86931567b676bb [id=f321a32c1ac043238c2fc92bfd5ec1ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.712170 27356 tablet_service.cc:1515] Processing DeleteTablet for tablet 8fff36d5a80e45af8a256c3908555e08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:14.712407 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06a79706f1184fa49d10182ba669b0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.712550 27365 tablet_service.cc:1468] Processing CreateTablet for tablet d1c96296a579440c887ba6c2a93f7204 (DEFAULT_TABLE table=default.Table_66babebee1c9462ea057c8489d43846d [id=a4d648e314504b68bc297abdbccad46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.712913 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1c96296a579440c887ba6c2a93f7204. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.714514 27363 tablet_service.cc:1468] Processing CreateTablet for tablet 7345305820be4d32a3ceb227f045f009 (DEFAULT_TABLE table=default.Table_0ac35f89d4d94e8fa94f9c3f853bf275 [id=52f969e50b9d466cbd14de6f252b7706]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.714851 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7345305820be4d32a3ceb227f045f009. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.715619 27501 tablet_service.cc:1468] Processing CreateTablet for tablet a253e3399622409e92299069b19d3698 (DEFAULT_TABLE table=default.Table_83be2fe3517249a39f2aa821588442b9 [id=2218f11e07d243adaa3a4d24e521b1b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.715986 27501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a253e3399622409e92299069b19d3698. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.719571 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet b5bdc27275cd4156afe767a1fb33a8bd (table default.Table_c557d48ec717410fb860dfdc7ce9957b [id=31f68383b3734a8691dfe0317fc60345]) successfully deleted
I20250903 20:05:14.720203 27359 tablet_service.cc:1468] Processing CreateTablet for tablet 02b3e262dd7841dfbb0520377eb94743 (DEFAULT_TABLE table=default.Table_ff22ae81dcc340ca897c8e8667aa83d0 [id=a30b474b4c8b494db74d62e576e509a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.720589 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02b3e262dd7841dfbb0520377eb94743. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.722864 28005 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 8857b5d4ea1e4f45aafb8e029837cca2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250903 20:05:14.723685 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 283a3afed7414a958b2dc4176f00bb01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:14.727952 27357 tablet_service.cc:1468] Processing CreateTablet for tablet fc634116a78f45dd833034ddda90e9bf (DEFAULT_TABLE table=default.Table_594fcf5c34de4f7287d7b7f605271d6c [id=fb43441ca8f2415d99a628e81ef1642f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.735014 27754 raft_consensus.cc:491] T c0343d2226e345669d42375de2545986 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:14.735014 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc634116a78f45dd833034ddda90e9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.735706 27754 raft_consensus.cc:513] T c0343d2226e345669d42375de2545986 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.736411 27754 leader_election.cc:290] T c0343d2226e345669d42375de2545986 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:14.742363 27695 tablet_bootstrap.cc:492] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
W20250903 20:05:14.745677 27183 leader_election.cc:343] T 8023fa7537234ce783c2bf90b575b00c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:14.746764 27687 tablet_replica.cc:331] T 46b5d491888147bbb101963641fd05f8 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:14.747094 27687 raft_consensus.cc:2241] T 46b5d491888147bbb101963641fd05f8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:14.747464 27687 raft_consensus.cc:2270] T 46b5d491888147bbb101963641fd05f8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:14.748234 27687 ts_tablet_manager.cc:1905] T 46b5d491888147bbb101963641fd05f8 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:14.748596 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 976322efa9464ed7a2175d3e3de51dab (DEFAULT_TABLE table=default.Table_2f6f89e1dd48493f9e989d4526caffed [id=4e05a1e86a4d4319b51f2ff36f3db76b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.750589 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_61b231b7d4df47138eae97d28eb75962"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:14.751246 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61b231b7d4df47138eae97d28eb75962 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:14.751873 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 07565ea2b5f34dcfb407ff3bfde17243 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:14.754819 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet 8fff36d5a80e45af8a256c3908555e08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:14.755206 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 567630e2a93645e58f9bf05ffff87b32
I20250903 20:05:14.756024 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_19fd54d3e8d8446a8cbe4956067e56d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:14.756639 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19fd54d3e8d8446a8cbe4956067e56d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:14.757321 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_fdf0ad621407452eb74b6a9a56be2e85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:14.757812 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fdf0ad621407452eb74b6a9a56be2e85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:14.757973 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a9f43781299e47a8aecc37a0b73d7a64 consensus state has no leader
I20250903 20:05:14.758373 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6f20b7d359084b4faef6f2eb3fdbab70" } new_table_name: "default.Table_7e582ad4ca9446c68da4850af02126ea" modify_external_catalogs: true
I20250903 20:05:14.759439 27692 consensus_queue.cc:260] T 68cff0e416a54a5eb37071f5a12f07ab P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.761296 27363 tablet_service.cc:1515] Processing DeleteTablet for tablet 76742d0df44c43729197f7e09f96cad8 with delete_type TABLET_DATA_DELETED (replaced by tablet 0940159eaea740f99be2b67d5abacd2d) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:14.761663 27692 ts_tablet_manager.cc:1428] T 68cff0e416a54a5eb37071f5a12f07ab P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.055s user 0.002s sys 0.000s
I20250903 20:05:14.761829 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_f92253d7e7aa42949cfa903765b446ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:14.762418 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f92253d7e7aa42949cfa903765b446ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:14.763113 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 5002b7e8118c49b6a247cb8dc4ca1aea (DEFAULT_TABLE table=default.Table_b20633c3b14e463aac0b7bd917c3ab9b [id=c6a148f02792496e8d8f2bb2dfc5e69b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.763406 27687 ts_tablet_manager.cc:1918] T 46b5d491888147bbb101963641fd05f8 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:14.763538 27687 log.cc:1199] T 46b5d491888147bbb101963641fd05f8 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/46b5d491888147bbb101963641fd05f8
I20250903 20:05:14.763620 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8
I20250903 20:05:14.763926 27687 ts_tablet_manager.cc:1939] T 46b5d491888147bbb101963641fd05f8 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:14.763876 27496 tablet_service.cc:1468] Processing CreateTablet for tablet 7345305820be4d32a3ceb227f045f009 (DEFAULT_TABLE table=default.Table_0ac35f89d4d94e8fa94f9c3f853bf275 [id=52f969e50b9d466cbd14de6f252b7706]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.764645 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5002b7e8118c49b6a247cb8dc4ca1aea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.764763 27363 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:14.764969 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 46b5d491888147bbb101963641fd05f8 (table default.Table_b88b7b0fe749402e90c3826d01e8cc8e [id=3b241b63c32f42c0966e9af3dc06d701]) successfully deleted
I20250903 20:05:14.765282 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a2b477ce2adf4ea0bd1b8685c6483f66" } new_table_name: "default.Table_03a48bb2af164512844bcc33bef7f521" modify_external_catalogs: true
I20250903 20:05:14.762836 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0343d2226e345669d42375de2545986" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:14.769143 27499 tablet_service.cc:1468] Processing CreateTablet for tablet d1c96296a579440c887ba6c2a93f7204 (DEFAULT_TABLE table=default.Table_66babebee1c9462ea057c8489d43846d [id=a4d648e314504b68bc297abdbccad46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.770437 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7345305820be4d32a3ceb227f045f009. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:14.770692 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 1bcb9d5f010f4297bab78c0183889238 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1bcb9d5f010f4297bab78c0183889238 already in progress: creating tablet
I20250903 20:05:14.771232 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 07565ea2b5f34dcfb407ff3bfde17243 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:14.771800 27687 tablet_replica.cc:331] T 7ba18e3f8cb148d484010cb4e72510b2 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:14.772100 27687 raft_consensus.cc:2241] T 7ba18e3f8cb148d484010cb4e72510b2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:14.772330 27687 raft_consensus.cc:2270] T 7ba18e3f8cb148d484010cb4e72510b2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:14.772819 27183 leader_election.cc:304] T 8023fa7537234ce783c2bf90b575b00c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:14.773202 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0343d2226e345669d42375de2545986" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:14.773315 27693 tablet_bootstrap.cc:492] T 710e3bc3c4194efbbeef6bc76ee64f24 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:14.773515 27693 ts_tablet_manager.cc:1397] T 710e3bc3c4194efbbeef6bc76ee64f24 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.072s user 0.004s sys 0.016s
I20250903 20:05:14.774672 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 06a79706f1184fa49d10182ba669b0ab (DEFAULT_TABLE table=default.Table_83ea6dfcee144a4fbc86931567b676bb [id=f321a32c1ac043238c2fc92bfd5ec1ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.774662 27693 raft_consensus.cc:357] T 710e3bc3c4194efbbeef6bc76ee64f24 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.774782 27321 leader_election.cc:304] T c0343d2226e345669d42375de2545986 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:14.775146 27226 tablet_service.cc:1468] Processing CreateTablet for tablet a253e3399622409e92299069b19d3698 (DEFAULT_TABLE table=default.Table_83be2fe3517249a39f2aa821588442b9 [id=2218f11e07d243adaa3a4d24e521b1b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.775278 27754 raft_consensus.cc:2747] T c0343d2226e345669d42375de2545986 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:14.775391 27221 tablet_service.cc:1515] Processing DeleteTablet for tablet 8fff36d5a80e45af8a256c3908555e08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:14.778921 27497 tablet_service.cc:1468] Processing CreateTablet for tablet 02b3e262dd7841dfbb0520377eb94743 (DEFAULT_TABLE table=default.Table_ff22ae81dcc340ca897c8e8667aa83d0 [id=a30b474b4c8b494db74d62e576e509a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.779805 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02b3e262dd7841dfbb0520377eb94743. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.780113 27213 tablet_service.cc:1468] Processing CreateTablet for tablet fc634116a78f45dd833034ddda90e9bf (DEFAULT_TABLE table=default.Table_594fcf5c34de4f7287d7b7f605271d6c [id=fb43441ca8f2415d99a628e81ef1642f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.781715 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 02b3e262dd7841dfbb0520377eb94743 (DEFAULT_TABLE table=default.Table_ff22ae81dcc340ca897c8e8667aa83d0 [id=a30b474b4c8b494db74d62e576e509a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.781989 27223 tablet_service.cc:1515] Processing DeleteTablet for tablet 567630e2a93645e58f9bf05ffff87b32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:14.782311 27218 tablet_service.cc:1468] Processing CreateTablet for tablet d1c96296a579440c887ba6c2a93f7204 (DEFAULT_TABLE table=default.Table_66babebee1c9462ea057c8489d43846d [id=a4d648e314504b68bc297abdbccad46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.783294 27687 ts_tablet_manager.cc:1905] T 7ba18e3f8cb148d484010cb4e72510b2 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:14.783401 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 5002b7e8118c49b6a247cb8dc4ca1aea (DEFAULT_TABLE table=default.Table_b20633c3b14e463aac0b7bd917c3ab9b [id=c6a148f02792496e8d8f2bb2dfc5e69b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.781085 27692 tablet_bootstrap.cc:492] T 4268316c28e8471a9f28807d20dbb395 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:14.784219 27224 tablet_service.cc:1515] Processing DeleteTablet for tablet 283a3afed7414a958b2dc4176f00bb01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:14.784709 27223 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:14.785307 27183 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 97503.3us
I20250903 20:05:14.786525 27221 tablet_service.cc:1468] Processing CreateTablet for tablet 7345305820be4d32a3ceb227f045f009 (DEFAULT_TABLE table=default.Table_0ac35f89d4d94e8fa94f9c3f853bf275 [id=52f969e50b9d466cbd14de6f252b7706]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.786693 27840 raft_consensus.cc:491] T 31342583c026438cbbae14106e75bbe5 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:14.787691 27840 raft_consensus.cc:513] T 31342583c026438cbbae14106e75bbe5 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.788371 27840 leader_election.cc:290] T 31342583c026438cbbae14106e75bbe5 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:14.788519 28030 raft_consensus.cc:2747] T 8023fa7537234ce783c2bf90b575b00c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:14.774935 27693 raft_consensus.cc:383] T 710e3bc3c4194efbbeef6bc76ee64f24 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:14.790307 27693 raft_consensus.cc:738] T 710e3bc3c4194efbbeef6bc76ee64f24 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:14.790673 27693 consensus_queue.cc:260] T 710e3bc3c4194efbbeef6bc76ee64f24 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.791399 27693 ts_tablet_manager.cc:1428] T 710e3bc3c4194efbbeef6bc76ee64f24 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20250903 20:05:14.791785 27693 tablet_bootstrap.cc:492] T d8ed7b3e292241cb9525745a797ff13a P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:14.793536 27693 tablet_bootstrap.cc:654] T d8ed7b3e292241cb9525745a797ff13a P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:14.794226 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31342583c026438cbbae14106e75bbe5" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:14.801286 27685 ts_tablet_manager.cc:1918] T e20fcd28c3ae4355a033d2b8429f84e2 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:14.803028 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1c96296a579440c887ba6c2a93f7204. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.805657 27685 log.cc:1199] T e20fcd28c3ae4355a033d2b8429f84e2 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/e20fcd28c3ae4355a033d2b8429f84e2
I20250903 20:05:14.805727 27687 ts_tablet_manager.cc:1918] T 7ba18e3f8cb148d484010cb4e72510b2 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:14.805934 27687 log.cc:1199] T 7ba18e3f8cb148d484010cb4e72510b2 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/7ba18e3f8cb148d484010cb4e72510b2
I20250903 20:05:14.806104 27685 ts_tablet_manager.cc:1939] T e20fcd28c3ae4355a033d2b8429f84e2 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
W20250903 20:05:14.807006 27184 leader_election.cc:343] T 31342583c026438cbbae14106e75bbe5 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:14.807461 27223 tablet_service.cc:1515] Processing DeleteTablet for tablet 31342583c026438cbbae14106e75bbe5 with delete_type TABLET_DATA_DELETED (replaced by tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:14.809218 27695 tablet_bootstrap.cc:654] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:14.822311 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 976322efa9464ed7a2175d3e3de51dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.824313 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc634116a78f45dd833034ddda90e9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.826342 27693 tablet_bootstrap.cc:492] T d8ed7b3e292241cb9525745a797ff13a P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:14.826534 27693 ts_tablet_manager.cc:1397] T d8ed7b3e292241cb9525745a797ff13a P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.004s
I20250903 20:05:14.827657 27693 raft_consensus.cc:357] T d8ed7b3e292241cb9525745a797ff13a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.827970 27693 raft_consensus.cc:383] T d8ed7b3e292241cb9525745a797ff13a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:14.828106 27693 raft_consensus.cc:738] T d8ed7b3e292241cb9525745a797ff13a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:14.828444 27693 consensus_queue.cc:260] T d8ed7b3e292241cb9525745a797ff13a P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.828681 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02b3e262dd7841dfbb0520377eb94743. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.829104 27687 ts_tablet_manager.cc:1939] T 7ba18e3f8cb148d484010cb4e72510b2 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:14.829823 27693 ts_tablet_manager.cc:1428] T d8ed7b3e292241cb9525745a797ff13a P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:14.830184 27693 tablet_bootstrap.cc:492] T c25aaa6ff310432bb27a0695394ec420 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:14.833105 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 7ba18e3f8cb148d484010cb4e72510b2 (table default.Table_9691d111178f4b9c9024a0a66fc3f03b [id=83d0f963c409436faa6c9a78bc0167cd]) successfully deleted
I20250903 20:05:14.833452 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a253e3399622409e92299069b19d3698. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:14.833415 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 283a3afed7414a958b2dc4176f00bb01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 283a3afed7414a958b2dc4176f00bb01 already in progress: creating tablet
W20250903 20:05:14.833683 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 07565ea2b5f34dcfb407ff3bfde17243 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07565ea2b5f34dcfb407ff3bfde17243 already in progress: creating tablet
I20250903 20:05:14.835578 28030 raft_consensus.cc:491] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:14.835721 28030 raft_consensus.cc:513] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.836301 27687 tablet_replica.cc:331] T 76742d0df44c43729197f7e09f96cad8 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:14.836318 28030 leader_election.cc:290] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:14.836488 27687 raft_consensus.cc:2241] T 76742d0df44c43729197f7e09f96cad8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:14.836660 27687 raft_consensus.cc:2270] T 76742d0df44c43729197f7e09f96cad8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:14.836647 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 8fff36d5a80e45af8a256c3908555e08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8fff36d5a80e45af8a256c3908555e08 already in progress: creating tablet
I20250903 20:05:14.837224 27693 tablet_bootstrap.cc:654] T c25aaa6ff310432bb27a0695394ec420 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:14.837828 27495 tablet_service.cc:1468] Processing CreateTablet for tablet fc634116a78f45dd833034ddda90e9bf (DEFAULT_TABLE table=default.Table_594fcf5c34de4f7287d7b7f605271d6c [id=fb43441ca8f2415d99a628e81ef1642f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.838063 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31342583c026438cbbae14106e75bbe5" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:14.838160 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2a5c74a281b44019872a6442d41bc373" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:14.838289 27687 ts_tablet_manager.cc:1905] T 76742d0df44c43729197f7e09f96cad8 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:14.838490 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2a5c74a281b44019872a6442d41bc373" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:14.838960 27183 leader_election.cc:343] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:14.839565 27183 leader_election.cc:343] T 31342583c026438cbbae14106e75bbe5 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:14.839682 27183 leader_election.cc:304] T 31342583c026438cbbae14106e75bbe5 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:14.839746 27184 leader_election.cc:343] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:14.840121 28030 raft_consensus.cc:2747] T 31342583c026438cbbae14106e75bbe5 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:14.839893 27184 leader_election.cc:304] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:14.840956 28030 raft_consensus.cc:2747] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:14.845118 27687 ts_tablet_manager.cc:1918] T 76742d0df44c43729197f7e09f96cad8 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:14.845247 27687 log.cc:1199] T 76742d0df44c43729197f7e09f96cad8 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/76742d0df44c43729197f7e09f96cad8
I20250903 20:05:14.845636 27687 ts_tablet_manager.cc:1939] T 76742d0df44c43729197f7e09f96cad8 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:14.846297 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 06a79706f1184fa49d10182ba669b0ab (DEFAULT_TABLE table=default.Table_83ea6dfcee144a4fbc86931567b676bb [id=f321a32c1ac043238c2fc92bfd5ec1ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:14.846920 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 76742d0df44c43729197f7e09f96cad8 (table default.Table_3055928e679e4dd8b8559b363b580e13 [id=9545d6a8d12e4937bb2b4cdbfc6933de]) successfully deleted
W20250903 20:05:14.847488 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8 already in progress: creating tablet
I20250903 20:05:14.852186 27685 tablet_replica.cc:331] T f7ab722c37264973b204cd4f1f95052d P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:14.852396 27685 raft_consensus.cc:2241] T f7ab722c37264973b204cd4f1f95052d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:14.852595 27685 raft_consensus.cc:2270] T f7ab722c37264973b204cd4f1f95052d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:14.853147 27685 ts_tablet_manager.cc:1905] T f7ab722c37264973b204cd4f1f95052d P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:14.854534 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet e20fcd28c3ae4355a033d2b8429f84e2 (table default.Table_53cc54536434460ba6637a083afaddb2 [id=0e611bffbb74451d9e24eed4a553c2ca]) successfully deleted
I20250903 20:05:14.854676 27692 tablet_bootstrap.cc:654] T 4268316c28e8471a9f28807d20dbb395 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:14.855609 27684 ts_tablet_manager.cc:1918] T 710e3bc3c4194efbbeef6bc76ee64f24 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:14.855744 27684 log.cc:1199] T 710e3bc3c4194efbbeef6bc76ee64f24 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/710e3bc3c4194efbbeef6bc76ee64f24
I20250903 20:05:14.856113 27684 ts_tablet_manager.cc:1939] T 710e3bc3c4194efbbeef6bc76ee64f24 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:14.858479 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5002b7e8118c49b6a247cb8dc4ca1aea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.861019 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06a79706f1184fa49d10182ba669b0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.862445 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1c96296a579440c887ba6c2a93f7204. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.864159 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc634116a78f45dd833034ddda90e9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.866451 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7345305820be4d32a3ceb227f045f009. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.870539 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06a79706f1184fa49d10182ba669b0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:14.886415 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet 567630e2a93645e58f9bf05ffff87b32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:14.887008 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a71ff73faf3a43dbb95677820a0561b6" } new_table_name: "default.Table_a88f7945592647468a4f4b0da26f5a84" modify_external_catalogs: true
W20250903 20:05:14.887459 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 567630e2a93645e58f9bf05ffff87b32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 567630e2a93645e58f9bf05ffff87b32 already in progress: creating tablet
W20250903 20:05:14.889719 26890 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 61253.2us
I20250903 20:05:14.903877 27692 tablet_bootstrap.cc:492] T 4268316c28e8471a9f28807d20dbb395 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:14.904175 27692 ts_tablet_manager.cc:1397] T 4268316c28e8471a9f28807d20dbb395 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.140s user 0.001s sys 0.007s
I20250903 20:05:14.905093 27692 raft_consensus.cc:357] T 4268316c28e8471a9f28807d20dbb395 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:14.907550 27066 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 67ca1dcf6fc7435db555392235f9900f consensus state has no leader
I20250903 20:05:14.909128 27692 raft_consensus.cc:383] T 4268316c28e8471a9f28807d20dbb395 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:14.909228 27692 raft_consensus.cc:738] T 4268316c28e8471a9f28807d20dbb395 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:14.909510 27692 consensus_queue.cc:260] T 4268316c28e8471a9f28807d20dbb395 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.913542 28030 raft_consensus.cc:491] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:14.913707 28030 raft_consensus.cc:513] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.914681 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:14.915678 28064 raft_consensus.cc:491] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:14.915798 28064 raft_consensus.cc:513] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.916445 28064 leader_election.cc:290] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:14.918324 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46e3a65118b744298b5536f88c22f1db" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:14.918550 27248 raft_consensus.cc:2466] T 46e3a65118b744298b5536f88c22f1db P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:14.919005 27462 leader_election.cc:304] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:14.919760 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46e3a65118b744298b5536f88c22f1db" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:14.919981 27386 raft_consensus.cc:2466] T 46e3a65118b744298b5536f88c22f1db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:14.920858 28030 leader_election.cc:290] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:14.921432 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68cff0e416a54a5eb37071f5a12f07ab" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:14.921620 27525 raft_consensus.cc:2466] T 68cff0e416a54a5eb37071f5a12f07ab P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:14.922132 27183 leader_election.cc:304] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:14.923008 28064 raft_consensus.cc:2802] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:14.923125 28064 raft_consensus.cc:491] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:14.923213 28064 raft_consensus.cc:3058] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:14.926004 28064 raft_consensus.cc:513] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.926476 28030 raft_consensus.cc:2802] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:14.926589 28030 raft_consensus.cc:491] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:14.926663 28030 raft_consensus.cc:3058] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:14.929962 28030 raft_consensus.cc:513] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.931516 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68cff0e416a54a5eb37071f5a12f07ab" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:14.931756 27525 raft_consensus.cc:3058] T 68cff0e416a54a5eb37071f5a12f07ab P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:14.934357 28030 leader_election.cc:290] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:14.934800 27525 raft_consensus.cc:2466] T 68cff0e416a54a5eb37071f5a12f07ab P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:14.935310 27183 leader_election.cc:304] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:14.936539 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46e3a65118b744298b5536f88c22f1db" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:14.936755 27386 raft_consensus.cc:3058] T 46e3a65118b744298b5536f88c22f1db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:14.939579 27386 raft_consensus.cc:2466] T 46e3a65118b744298b5536f88c22f1db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:14.940510 27462 leader_election.cc:304] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:14.940892 27952 raft_consensus.cc:2802] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:14.941051 27952 raft_consensus.cc:695] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:14.941359 27952 consensus_queue.cc:237] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.942478 27695 tablet_bootstrap.cc:492] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:14.942632 27695 ts_tablet_manager.cc:1397] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.282s user 0.007s sys 0.001s
I20250903 20:05:14.943572 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46e3a65118b744298b5536f88c22f1db" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:14.943507 27695 raft_consensus.cc:357] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.943754 27248 raft_consensus.cc:3058] T 46e3a65118b744298b5536f88c22f1db P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:14.944065 28064 leader_election.cc:290] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:14.944960 28030 raft_consensus.cc:2802] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:14.945858 28030 raft_consensus.cc:695] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:14.946161 28030 consensus_queue.cc:237] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.947491 27248 raft_consensus.cc:2466] T 46e3a65118b744298b5536f88c22f1db P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:14.943758 27695 raft_consensus.cc:383] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:14.947932 27695 raft_consensus.cc:738] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:14.948205 27695 consensus_queue.cc:260] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.948683 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet cfa8a1d9011e460681443a6b77b95958 with delete_type TABLET_DATA_DELETED (replaced by tablet 4af8c03c9d834405b4e25d35441db961) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:14.948825 27695 ts_tablet_manager.cc:1428] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250903 20:05:14.949162 27695 tablet_bootstrap.cc:492] T 262a3549867a473aaaa7dfcc0b5872cd P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:14.950518 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68cff0e416a54a5eb37071f5a12f07ab" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:14.951329 27695 tablet_bootstrap.cc:654] T 262a3549867a473aaaa7dfcc0b5872cd P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:14.951773 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68cff0e416a54a5eb37071f5a12f07ab" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:14.926126 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 710e3bc3c4194efbbeef6bc76ee64f24 (table default.Table_452fdb47678d4520828b9a3db6ddb237 [id=d945a830c86244bca66cb2652e64966d]) successfully deleted
W20250903 20:05:14.952442 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 466e501e424e46bc98e8bbc64f97005e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 466e501e424e46bc98e8bbc64f97005e already in progress: creating tablet
W20250903 20:05:14.952718 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 673c45f3f2a14da9bd306011d396eaca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 673c45f3f2a14da9bd306011d396eaca already in progress: creating tablet
W20250903 20:05:14.952950 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 59a1dd48d76241d7ba817070fbffd853 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59a1dd48d76241d7ba817070fbffd853 already in progress: creating tablet
W20250903 20:05:14.953204 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 7cc6cf579e2642e99fefed723b0ee900 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7cc6cf579e2642e99fefed723b0ee900 already in progress: creating tablet
W20250903 20:05:14.952778 27184 leader_election.cc:343] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:14.953567 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 5f27f525fdab407280aa23df1ae71b43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f27f525fdab407280aa23df1ae71b43 already in progress: creating tablet
W20250903 20:05:14.953809 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet c6d67cd0c0304c77ba571b414878dad2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6d67cd0c0304c77ba571b414878dad2 already in progress: creating tablet
W20250903 20:05:14.954012 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 8023fa7537234ce783c2bf90b575b00c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8023fa7537234ce783c2bf90b575b00c already in progress: creating tablet
W20250903 20:05:14.954394 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 607c9a9cbc5540e5bd0fa436fcba564c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 607c9a9cbc5540e5bd0fa436fcba564c already in progress: creating tablet
W20250903 20:05:14.954610 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 1bcb9d5f010f4297bab78c0183889238 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1bcb9d5f010f4297bab78c0183889238 already in progress: creating tablet
W20250903 20:05:14.954852 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 8d2d64f103fc44d7b390559b92cbf4b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d2d64f103fc44d7b390559b92cbf4b4 already in progress: creating tablet
I20250903 20:05:14.954996 27501 tablet_service.cc:1515] Processing DeleteTablet for tablet 567630e2a93645e58f9bf05ffff87b32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:14.955053 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 283a3afed7414a958b2dc4176f00bb01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 283a3afed7414a958b2dc4176f00bb01 already in progress: creating tablet
W20250903 20:05:14.954639 27184 leader_election.cc:343] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:14.955289 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 07565ea2b5f34dcfb407ff3bfde17243 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07565ea2b5f34dcfb407ff3bfde17243 already in progress: creating tablet
W20250903 20:05:14.955492 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 8fff36d5a80e45af8a256c3908555e08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8fff36d5a80e45af8a256c3908555e08 already in progress: creating tablet
I20250903 20:05:14.956673 27692 ts_tablet_manager.cc:1428] T 4268316c28e8471a9f28807d20dbb395 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.052s user 0.000s sys 0.002s
I20250903 20:05:14.957921 27692 tablet_bootstrap.cc:492] T 8023fa7537234ce783c2bf90b575b00c P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:14.957448 27684 tablet_replica.cc:331] T cfa8a1d9011e460681443a6b77b95958 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:14.962491 27684 raft_consensus.cc:2241] T cfa8a1d9011e460681443a6b77b95958 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:14.962772 27684 raft_consensus.cc:2270] T cfa8a1d9011e460681443a6b77b95958 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:14.964192 27684 ts_tablet_manager.cc:1905] T cfa8a1d9011e460681443a6b77b95958 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:14.969527 27692 tablet_bootstrap.cc:654] T 8023fa7537234ce783c2bf90b575b00c P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:14.970108 26890 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 76444.6us
W20250903 20:05:14.972469 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dbfdab4332234b2a93fd3c901a9cb645 consensus state has no leader
I20250903 20:05:14.975952 27693 tablet_bootstrap.cc:492] T c25aaa6ff310432bb27a0695394ec420 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:14.976166 27693 ts_tablet_manager.cc:1397] T c25aaa6ff310432bb27a0695394ec420 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.146s user 0.024s sys 0.029s
I20250903 20:05:14.976161 27684 ts_tablet_manager.cc:1918] T cfa8a1d9011e460681443a6b77b95958 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:14.976300 27684 log.cc:1199] T cfa8a1d9011e460681443a6b77b95958 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/cfa8a1d9011e460681443a6b77b95958
I20250903 20:05:14.976748 27684 ts_tablet_manager.cc:1939] T cfa8a1d9011e460681443a6b77b95958 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:14.977664 27693 raft_consensus.cc:357] T c25aaa6ff310432bb27a0695394ec420 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.977881 27693 raft_consensus.cc:383] T c25aaa6ff310432bb27a0695394ec420 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:14.977962 27693 raft_consensus.cc:738] T c25aaa6ff310432bb27a0695394ec420 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:14.978271 27693 consensus_queue.cc:260] T c25aaa6ff310432bb27a0695394ec420 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.978811 27693 ts_tablet_manager.cc:1428] T c25aaa6ff310432bb27a0695394ec420 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250903 20:05:14.979152 27693 tablet_bootstrap.cc:492] T 9509364ba3ec4e9bad09db09406a978a P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:14.981048 27693 tablet_bootstrap.cc:654] T 9509364ba3ec4e9bad09db09406a978a P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:14.986066 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_bd2731bed3d34406a8867671f5f461a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:14.986804 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd2731bed3d34406a8867671f5f461a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:14.987897 27693 tablet_bootstrap.cc:492] T 9509364ba3ec4e9bad09db09406a978a P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:14.988070 27693 ts_tablet_manager.cc:1397] T 9509364ba3ec4e9bad09db09406a978a P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20250903 20:05:14.989063 27693 raft_consensus.cc:357] T 9509364ba3ec4e9bad09db09406a978a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.989259 27693 raft_consensus.cc:383] T 9509364ba3ec4e9bad09db09406a978a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:14.989354 27693 raft_consensus.cc:738] T 9509364ba3ec4e9bad09db09406a978a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:14.989603 27693 consensus_queue.cc:260] T 9509364ba3ec4e9bad09db09406a978a P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:14.990540 27693 ts_tablet_manager.cc:1428] T 9509364ba3ec4e9bad09db09406a978a P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20250903 20:05:14.990876 27693 tablet_bootstrap.cc:492] T e5136d5388bc4c2ba71e09be40e8f412 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:14.993533 27065 master_service.cc:948] ReplaceTablet: received request to replace tablet 9685cba2f73b412382a69dab2c7f50c1 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:14.993992 27057 master_service.cc:948] ReplaceTablet: received request to replace tablet 7c398634d6304a12b35c0579fba4c813 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:14.995052 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_fcaf18430cde44f3b4dec80ebf611ce5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:14.995220 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_3a0bcaebe62e46e79afba5b118eeb855"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:14.996013 27061 master_service.cc:948] ReplaceTablet: received request to replace tablet 57a1167457b84ea8a493be232b1d088a from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:14.996735 27057 master_service.cc:948] ReplaceTablet: received request to replace tablet fd499b81344c4b58bf77f3622422763c from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:14.998606 27692 tablet_bootstrap.cc:492] T 8023fa7537234ce783c2bf90b575b00c P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:14.998977 27692 ts_tablet_manager.cc:1397] T 8023fa7537234ce783c2bf90b575b00c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.000s
W20250903 20:05:14.999145 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0f5b87c1c4a24f8abdbfd6194f3b2c4a consensus state has no leader
I20250903 20:05:14.999580 27693 tablet_bootstrap.cc:654] T e5136d5388bc4c2ba71e09be40e8f412 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:14.999861 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d859c0222c3647dab3dddd105e4cf0fc consensus state has no leader
I20250903 20:05:15.000937 27064 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c967f8b0d1314f589eae529cc492493b" } modify_external_catalogs: true
I20250903 20:05:15.001169 27064 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c967f8b0d1314f589eae529cc492493b" } modify_external_catalogs: true
I20250903 20:05:15.002302 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_a22a6e16b90a461ba0c890fb7c5791e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:15.005288 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a6608fc0162446a480d8e8ecb07db28f" } modify_external_catalogs: true
I20250903 20:05:15.005499 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a6608fc0162446a480d8e8ecb07db28f" } modify_external_catalogs: true
I20250903 20:05:15.006170 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_16b62656e76142339c0a42ca5a7f5e6b" } modify_external_catalogs: true
I20250903 20:05:15.006371 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_16b62656e76142339c0a42ca5a7f5e6b" } modify_external_catalogs: true
I20250903 20:05:15.009728 27952 raft_consensus.cc:491] T 4268316c28e8471a9f28807d20dbb395 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:15.010171 27952 raft_consensus.cc:513] T 4268316c28e8471a9f28807d20dbb395 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.011936 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4268316c28e8471a9f28807d20dbb395" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:15.012538 27462 leader_election.cc:343] T 4268316c28e8471a9f28807d20dbb395 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:15.013633 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a22a6e16b90a461ba0c890fb7c5791e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:15.013718 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a0bcaebe62e46e79afba5b118eeb855 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:15.014969 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fcaf18430cde44f3b4dec80ebf611ce5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:15.014997 27952 leader_election.cc:290] T 4268316c28e8471a9f28807d20dbb395 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:15.024668 27462 leader_election.cc:343] T 4268316c28e8471a9f28807d20dbb395 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 4268316c28e8471a9f28807d20dbb395
I20250903 20:05:15.024945 27462 leader_election.cc:304] T 4268316c28e8471a9f28807d20dbb395 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:15.001421 27692 raft_consensus.cc:357] T 8023fa7537234ce783c2bf90b575b00c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.025960 27952 raft_consensus.cc:2747] T 4268316c28e8471a9f28807d20dbb395 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:15.026701 27692 raft_consensus.cc:383] T 8023fa7537234ce783c2bf90b575b00c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:15.027030 27692 raft_consensus.cc:738] T 8023fa7537234ce783c2bf90b575b00c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:15.027670 27695 tablet_bootstrap.cc:492] T 262a3549867a473aaaa7dfcc0b5872cd P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:15.027865 27695 ts_tablet_manager.cc:1397] T 262a3549867a473aaaa7dfcc0b5872cd P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.079s user 0.004s sys 0.000s
I20250903 20:05:15.028187 27685 ts_tablet_manager.cc:1918] T f7ab722c37264973b204cd4f1f95052d P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:15.028170 27692 consensus_queue.cc:260] T 8023fa7537234ce783c2bf90b575b00c P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.028375 27685 log.cc:1199] T f7ab722c37264973b204cd4f1f95052d P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/f7ab722c37264973b204cd4f1f95052d
I20250903 20:05:15.029664 27685 ts_tablet_manager.cc:1939] T f7ab722c37264973b204cd4f1f95052d P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:15.031893 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet fd499b81344c4b58bf77f3622422763c
I20250903 20:05:15.031889 27695 raft_consensus.cc:357] T 262a3549867a473aaaa7dfcc0b5872cd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.032124 27695 raft_consensus.cc:383] T 262a3549867a473aaaa7dfcc0b5872cd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:15.032208 27695 raft_consensus.cc:738] T 262a3549867a473aaaa7dfcc0b5872cd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:15.032423 27057 catalog_manager.cc:6652] ReplaceTablet: tablet fd499b81344c4b58bf77f3622422763c deleted and replaced by tablet cb2f50683e904dd39ee6eb128717d38c
I20250903 20:05:15.032480 27695 consensus_queue.cc:260] T 262a3549867a473aaaa7dfcc0b5872cd P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.032976 27695 ts_tablet_manager.cc:1428] T 262a3549867a473aaaa7dfcc0b5872cd P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250903 20:05:15.033406 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet f7ab722c37264973b204cd4f1f95052d (table default.Table_b2a82a2c310140caaeac5d0dcb4bb61a [id=e2548ad469e54a41af7c0181357d2467]) successfully deleted
I20250903 20:05:15.033972 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet fd499b81344c4b58bf77f3622422763c with delete_type TABLET_DATA_DELETED (replaced by tablet cb2f50683e904dd39ee6eb128717d38c) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:15.034289 27687 tablet_replica.cc:331] T fd499b81344c4b58bf77f3622422763c P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:15.034518 27687 raft_consensus.cc:2241] T fd499b81344c4b58bf77f3622422763c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:15.034732 27687 raft_consensus.cc:2270] T fd499b81344c4b58bf77f3622422763c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:15.035399 27687 ts_tablet_manager.cc:1905] T fd499b81344c4b58bf77f3622422763c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:15.036679 27692 ts_tablet_manager.cc:1428] T 8023fa7537234ce783c2bf90b575b00c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.037s user 0.001s sys 0.001s
I20250903 20:05:15.039283 27692 tablet_bootstrap.cc:492] T f7ab722c37264973b204cd4f1f95052d P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:15.041193 27685 tablet_replica.cc:331] T cfa8a1d9011e460681443a6b77b95958 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:15.041924 27695 tablet_bootstrap.cc:492] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:15.044555 27687 ts_tablet_manager.cc:1918] T fd499b81344c4b58bf77f3622422763c P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 20:05:15.044725 27687 log.cc:1199] T fd499b81344c4b58bf77f3622422763c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/fd499b81344c4b58bf77f3622422763c
I20250903 20:05:15.045625 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet fd499b81344c4b58bf77f3622422763c with delete_type TABLET_DATA_DELETED (replaced by tablet cb2f50683e904dd39ee6eb128717d38c) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:15.046372 27687 ts_tablet_manager.cc:1939] T fd499b81344c4b58bf77f3622422763c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:15.046689 27695 tablet_bootstrap.cc:654] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:15.047050 27692 tablet_bootstrap.cc:654] T f7ab722c37264973b204cd4f1f95052d P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:15.047608 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet fd499b81344c4b58bf77f3622422763c (table default.Table_e914faf335d44d019f91bda71758c473 [id=374b9abdf6a54b38876a96d8ec4a0368]) successfully deleted
I20250903 20:05:15.047700 27685 raft_consensus.cc:2241] T cfa8a1d9011e460681443a6b77b95958 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:15.047876 27685 raft_consensus.cc:2270] T cfa8a1d9011e460681443a6b77b95958 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:15.050042 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8 already in progress: creating tablet
I20250903 20:05:15.052423 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet cfa8a1d9011e460681443a6b77b95958 (table default.Table_6d201f66980a4a7ab430058673f8d513 [id=058a64daaf104dfdace7d05d52388b79]) successfully deleted
W20250903 20:05:15.052734 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 567630e2a93645e58f9bf05ffff87b32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 567630e2a93645e58f9bf05ffff87b32 already in progress: creating tablet
I20250903 20:05:15.055364 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet fd499b81344c4b58bf77f3622422763c with delete_type TABLET_DATA_DELETED (replaced by tablet cb2f50683e904dd39ee6eb128717d38c) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:15.055733 27684 tablet_replica.cc:331] T fd499b81344c4b58bf77f3622422763c P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:15.055976 27684 raft_consensus.cc:2241] T fd499b81344c4b58bf77f3622422763c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:15.056226 27684 raft_consensus.cc:2270] T fd499b81344c4b58bf77f3622422763c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:15.056840 27684 ts_tablet_manager.cc:1905] T fd499b81344c4b58bf77f3622422763c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:15.057420 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet 710e3bc3c4194efbbeef6bc76ee64f24 with delete_type TABLET_DATA_DELETED (replaced by tablet 4be9bdc390de4a74a396003c9c3c2b71) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:15.057668 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet d8ed7b3e292241cb9525745a797ff13a with delete_type TABLET_DATA_DELETED (replaced by tablet 639ecb9caa4a423899ab8eb35bb716d7) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:15.058496 27687 tablet_replica.cc:331] T 710e3bc3c4194efbbeef6bc76ee64f24 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:15.058697 27687 raft_consensus.cc:2241] T 710e3bc3c4194efbbeef6bc76ee64f24 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:15.058877 27687 raft_consensus.cc:2270] T 710e3bc3c4194efbbeef6bc76ee64f24 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:15.059888 27687 ts_tablet_manager.cc:1905] T 710e3bc3c4194efbbeef6bc76ee64f24 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:15.063413 27693 tablet_bootstrap.cc:492] T e5136d5388bc4c2ba71e09be40e8f412 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:15.063624 27693 ts_tablet_manager.cc:1397] T e5136d5388bc4c2ba71e09be40e8f412 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.073s user 0.013s sys 0.021s
I20250903 20:05:15.063675 27685 ts_tablet_manager.cc:1905] T cfa8a1d9011e460681443a6b77b95958 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:15.064488 27693 raft_consensus.cc:357] T e5136d5388bc4c2ba71e09be40e8f412 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:15.064718 27693 raft_consensus.cc:383] T e5136d5388bc4c2ba71e09be40e8f412 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:15.064812 27693 raft_consensus.cc:738] T e5136d5388bc4c2ba71e09be40e8f412 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:15.065135 27693 consensus_queue.cc:260] T e5136d5388bc4c2ba71e09be40e8f412 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:15.065651 27684 ts_tablet_manager.cc:1918] T fd499b81344c4b58bf77f3622422763c P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 20:05:15.065742 27693 ts_tablet_manager.cc:1428] T e5136d5388bc4c2ba71e09be40e8f412 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:15.065907 27684 log.cc:1199] T fd499b81344c4b58bf77f3622422763c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/fd499b81344c4b58bf77f3622422763c
I20250903 20:05:15.066450 27684 ts_tablet_manager.cc:1939] T fd499b81344c4b58bf77f3622422763c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:15.066602 27693 tablet_bootstrap.cc:492] T 8945085bdc174e6db22a5a9477810297 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:15.069222 27692 tablet_bootstrap.cc:492] T f7ab722c37264973b204cd4f1f95052d P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:15.074040 27695 tablet_bootstrap.cc:492] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:15.074237 27695 ts_tablet_manager.cc:1397] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20250903 20:05:15.075685 27695 raft_consensus.cc:357] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.075877 27695 raft_consensus.cc:383] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:15.075969 27695 raft_consensus.cc:738] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:15.076217 27695 consensus_queue.cc:260] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.077764 27062 master_service.cc:948] ReplaceTablet: received request to replace tablet 369c750ff5bf449e989d95ffd89497d4 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:15.078296 27695 ts_tablet_manager.cc:1428] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:15.080083 27692 ts_tablet_manager.cc:1397] T f7ab722c37264973b204cd4f1f95052d P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.002s
I20250903 20:05:15.081125 27692 raft_consensus.cc:357] T f7ab722c37264973b204cd4f1f95052d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.081358 27692 raft_consensus.cc:383] T f7ab722c37264973b204cd4f1f95052d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:15.081446 27692 raft_consensus.cc:738] T f7ab722c37264973b204cd4f1f95052d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:15.081738 27692 consensus_queue.cc:260] T f7ab722c37264973b204cd4f1f95052d P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.081775 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet fd499b81344c4b58bf77f3622422763c (table default.Table_e914faf335d44d019f91bda71758c473 [id=374b9abdf6a54b38876a96d8ec4a0368]) successfully deleted
I20250903 20:05:15.083696 27692 ts_tablet_manager.cc:1428] T f7ab722c37264973b204cd4f1f95052d P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:15.084045 27692 tablet_bootstrap.cc:492] T e20fcd28c3ae4355a033d2b8429f84e2 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:15.085304 27685 ts_tablet_manager.cc:1918] T cfa8a1d9011e460681443a6b77b95958 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:15.085479 27685 log.cc:1199] T cfa8a1d9011e460681443a6b77b95958 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/cfa8a1d9011e460681443a6b77b95958
I20250903 20:05:15.085873 27685 ts_tablet_manager.cc:1939] T cfa8a1d9011e460681443a6b77b95958 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:15.089560 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet c25aaa6ff310432bb27a0695394ec420 with delete_type TABLET_DATA_DELETED (replaced by tablet 3f78a951bb024cf290116b0fe438ddb4) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:15.089643 27687 ts_tablet_manager.cc:1918] T 710e3bc3c4194efbbeef6bc76ee64f24 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:15.089771 27687 log.cc:1199] T 710e3bc3c4194efbbeef6bc76ee64f24 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/710e3bc3c4194efbbeef6bc76ee64f24
I20250903 20:05:15.089910 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet e5136d5388bc4c2ba71e09be40e8f412 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:15.090124 27363 tablet_service.cc:1515] Processing DeleteTablet for tablet 9509364ba3ec4e9bad09db09406a978a with delete_type TABLET_DATA_DELETED (replaced by tablet 1fbe54dc40dd47daa109ceeb07546b0a) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:15.090173 27687 ts_tablet_manager.cc:1939] T 710e3bc3c4194efbbeef6bc76ee64f24 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:15.090277 27692 tablet_bootstrap.cc:654] T e20fcd28c3ae4355a033d2b8429f84e2 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:15.091735 27693 tablet_bootstrap.cc:654] T 8945085bdc174e6db22a5a9477810297 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:15.103101 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet cfa8a1d9011e460681443a6b77b95958 (table default.Table_6d201f66980a4a7ab430058673f8d513 [id=058a64daaf104dfdace7d05d52388b79]) successfully deleted
W20250903 20:05:15.103971 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 466e501e424e46bc98e8bbc64f97005e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 466e501e424e46bc98e8bbc64f97005e already in progress: creating tablet
W20250903 20:05:15.104923 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 673c45f3f2a14da9bd306011d396eaca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 673c45f3f2a14da9bd306011d396eaca already in progress: creating tablet
W20250903 20:05:15.105585 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 7cc6cf579e2642e99fefed723b0ee900 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7cc6cf579e2642e99fefed723b0ee900 already in progress: creating tablet
W20250903 20:05:15.106190 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 59a1dd48d76241d7ba817070fbffd853 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59a1dd48d76241d7ba817070fbffd853 already in progress: creating tablet
I20250903 20:05:15.108510 27695 tablet_bootstrap.cc:492] T 691cb69759654fdc9223844049a06d02 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
W20250903 20:05:15.109093 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 607c9a9cbc5540e5bd0fa436fcba564c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 607c9a9cbc5540e5bd0fa436fcba564c already in progress: creating tablet
W20250903 20:05:15.109769 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 5f27f525fdab407280aa23df1ae71b43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f27f525fdab407280aa23df1ae71b43 already in progress: creating tablet
W20250903 20:05:15.110625 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet c6d67cd0c0304c77ba571b414878dad2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6d67cd0c0304c77ba571b414878dad2 already in progress: creating tablet
I20250903 20:05:15.111159 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 22bc87d1a4804b6c80f77f9692500d63
I20250903 20:05:15.111379 27685 tablet_replica.cc:331] T 8023fa7537234ce783c2bf90b575b00c P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:15.111583 27685 raft_consensus.cc:2241] T 8023fa7537234ce783c2bf90b575b00c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:15.111774 27685 raft_consensus.cc:2270] T 8023fa7537234ce783c2bf90b575b00c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:15.112351 27685 ts_tablet_manager.cc:1905] T 8023fa7537234ce783c2bf90b575b00c P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:15.113710 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 22bc87d1a4804b6c80f77f9692500d63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:15.114295 27363 tablet_service.cc:1515] Processing DeleteTablet for tablet 22bc87d1a4804b6c80f77f9692500d63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:15.115221 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 4c18a98d5c0f4a43ae91c9130047b681
I20250903 20:05:15.115317 27695 tablet_bootstrap.cc:654] T 691cb69759654fdc9223844049a06d02 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:15.116429 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c18a98d5c0f4a43ae91c9130047b681 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:15.120610 27687 tablet_replica.cc:331] T d8ed7b3e292241cb9525745a797ff13a P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:15.123833 27692 tablet_bootstrap.cc:492] T e20fcd28c3ae4355a033d2b8429f84e2 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:15.124150 27687 raft_consensus.cc:2241] T d8ed7b3e292241cb9525745a797ff13a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:15.124423 27687 raft_consensus.cc:2270] T d8ed7b3e292241cb9525745a797ff13a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:15.125085 27363 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c18a98d5c0f4a43ae91c9130047b681 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:15.125886 27692 ts_tablet_manager.cc:1397] T e20fcd28c3ae4355a033d2b8429f84e2 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.002s
I20250903 20:05:15.131460 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 710e3bc3c4194efbbeef6bc76ee64f24 (table default.Table_452fdb47678d4520828b9a3db6ddb237 [id=d945a830c86244bca66cb2652e64966d]) successfully deleted
I20250903 20:05:15.132375 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet 22bc87d1a4804b6c80f77f9692500d63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:15.133121 27687 ts_tablet_manager.cc:1905] T d8ed7b3e292241cb9525745a797ff13a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:15.134687 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c18a98d5c0f4a43ae91c9130047b681 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:15.136689 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_459903503b5148d09d3cb8f2f5519fab" } modify_external_catalogs: true
I20250903 20:05:15.136891 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_459903503b5148d09d3cb8f2f5519fab" } modify_external_catalogs: true
I20250903 20:05:15.137704 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d4f82c96b77d430caad8a6b1bdec1a2f" } new_table_name: "default.Table_ac1cd4ca0afa4c3494fa9953c49c838e" modify_external_catalogs: true
I20250903 20:05:15.141117 27692 raft_consensus.cc:357] T e20fcd28c3ae4355a033d2b8429f84e2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:15.142341 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 22bc87d1a4804b6c80f77f9692500d63 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22bc87d1a4804b6c80f77f9692500d63 already in progress: creating tablet
I20250903 20:05:15.142529 27754 raft_consensus.cc:491] T e5136d5388bc4c2ba71e09be40e8f412 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250903 20:05:15.142692 27051 catalog_manager.cc:4895] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 4c18a98d5c0f4a43ae91c9130047b681 because the tablet was not found. No further retry: Not found: Tablet not found: 4c18a98d5c0f4a43ae91c9130047b681
I20250903 20:05:15.142714 27754 raft_consensus.cc:513] T e5136d5388bc4c2ba71e09be40e8f412 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:15.143178 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3d20da97fa86497e9376f8aa69f1c3fd" } modify_external_catalogs: true
I20250903 20:05:15.143347 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3d20da97fa86497e9376f8aa69f1c3fd" } modify_external_catalogs: true
I20250903 20:05:15.143388 27754 leader_election.cc:290] T e5136d5388bc4c2ba71e09be40e8f412 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:15.143864 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b0d2811b8e4a40f4a34d3be594183ac6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:15.144459 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0d2811b8e4a40f4a34d3be594183ac6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:15.145392 27687 ts_tablet_manager.cc:1918] T d8ed7b3e292241cb9525745a797ff13a P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:15.145546 27687 log.cc:1199] T d8ed7b3e292241cb9525745a797ff13a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/d8ed7b3e292241cb9525745a797ff13a
I20250903 20:05:15.145897 27687 ts_tablet_manager.cc:1939] T d8ed7b3e292241cb9525745a797ff13a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:15.146893 27687 tablet_replica.cc:331] T c25aaa6ff310432bb27a0695394ec420 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:15.147081 27687 raft_consensus.cc:2241] T c25aaa6ff310432bb27a0695394ec420 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:15.147261 27687 raft_consensus.cc:2270] T c25aaa6ff310432bb27a0695394ec420 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:15.148763 27320 leader_election.cc:343] T e5136d5388bc4c2ba71e09be40e8f412 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: e5136d5388bc4c2ba71e09be40e8f412
W20250903 20:05:15.149458 27321 leader_election.cc:343] T e5136d5388bc4c2ba71e09be40e8f412 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: e5136d5388bc4c2ba71e09be40e8f412
I20250903 20:05:15.149588 27321 leader_election.cc:304] T e5136d5388bc4c2ba71e09be40e8f412 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:15.150002 27754 raft_consensus.cc:2747] T e5136d5388bc4c2ba71e09be40e8f412 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:15.152551 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 57a1167457b84ea8a493be232b1d088a
I20250903 20:05:15.152989 27061 catalog_manager.cc:6652] ReplaceTablet: tablet 57a1167457b84ea8a493be232b1d088a deleted and replaced by tablet f326ec776ba446be8d5dda5c1dc5b6ec
I20250903 20:05:15.156293 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet d8ed7b3e292241cb9525745a797ff13a (table default.Table_3da0f03ae8114e348baf44301196b2e5 [id=ad8b3a9bedef4c188b89640ab5e2473a]) successfully deleted
I20250903 20:05:15.156814 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 57a1167457b84ea8a493be232b1d088a with delete_type TABLET_DATA_DELETED (replaced by tablet f326ec776ba446be8d5dda5c1dc5b6ec) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:15.157294 27363 tablet_service.cc:1515] Processing DeleteTablet for tablet 57a1167457b84ea8a493be232b1d088a with delete_type TABLET_DATA_DELETED (replaced by tablet f326ec776ba446be8d5dda5c1dc5b6ec) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:15.158286 27692 raft_consensus.cc:383] T e20fcd28c3ae4355a033d2b8429f84e2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:15.158423 27692 raft_consensus.cc:738] T e20fcd28c3ae4355a033d2b8429f84e2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:15.158681 27692 consensus_queue.cc:260] T e20fcd28c3ae4355a033d2b8429f84e2 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.158996 27057 catalog_manager.cc:5582] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a reported cstate change: term changed from 0 to 1, leader changed from <none> to 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65). New cstate: current_term: 1 leader_uuid: "89f8f0ee68b940db9ce250c0604f934a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: HEALTHY } } }
I20250903 20:05:15.159435 27057 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 68cff0e416a54a5eb37071f5a12f07ab (table default.Table_1e3f794ae5ec4ad6a9b4351b4bda7b15 [id=71d850a0cea14b0ba337a822ad99d513]). Expected version 3 got 0
I20250903 20:05:15.160748 27692 ts_tablet_manager.cc:1428] T e20fcd28c3ae4355a033d2b8429f84e2 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.035s user 0.002s sys 0.000s
I20250903 20:05:15.161063 27692 tablet_bootstrap.cc:492] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:15.162983 27692 tablet_bootstrap.cc:654] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:15.166823 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet 57a1167457b84ea8a493be232b1d088a with delete_type TABLET_DATA_DELETED (replaced by tablet f326ec776ba446be8d5dda5c1dc5b6ec) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:15.167506 27695 tablet_bootstrap.cc:492] T 691cb69759654fdc9223844049a06d02 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:15.167701 27695 ts_tablet_manager.cc:1397] T 691cb69759654fdc9223844049a06d02 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.000s
I20250903 20:05:15.168515 27695 raft_consensus.cc:357] T 691cb69759654fdc9223844049a06d02 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.168733 27695 raft_consensus.cc:383] T 691cb69759654fdc9223844049a06d02 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:15.168813 27695 raft_consensus.cc:738] T 691cb69759654fdc9223844049a06d02 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:15.169098 27695 consensus_queue.cc:260] T 691cb69759654fdc9223844049a06d02 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.171245 27695 ts_tablet_manager.cc:1428] T 691cb69759654fdc9223844049a06d02 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250903 20:05:15.171592 27695 tablet_bootstrap.cc:492] T 264127511a234e9fbf409f3aa2860ccc P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:15.173497 27695 tablet_bootstrap.cc:654] T 264127511a234e9fbf409f3aa2860ccc P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:15.174880 27687 ts_tablet_manager.cc:1905] T c25aaa6ff310432bb27a0695394ec420 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:15.175729 27136 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 4c18a98d5c0f4a43ae91c9130047b681
I20250903 20:05:15.176934 27363 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c18a98d5c0f4a43ae91c9130047b681 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:15.179808 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c18a98d5c0f4a43ae91c9130047b681 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:15.180667 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 57a1167457b84ea8a493be232b1d088a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57a1167457b84ea8a493be232b1d088a already in progress: creating tablet
I20250903 20:05:15.191922 27685 ts_tablet_manager.cc:1918] T 8023fa7537234ce783c2bf90b575b00c P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:15.192169 27685 log.cc:1199] T 8023fa7537234ce783c2bf90b575b00c P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/8023fa7537234ce783c2bf90b575b00c
I20250903 20:05:15.192905 27685 ts_tablet_manager.cc:1939] T 8023fa7537234ce783c2bf90b575b00c P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
W20250903 20:05:15.193356 27051 catalog_manager.cc:4895] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 4c18a98d5c0f4a43ae91c9130047b681 because the tablet was not found. No further retry: Not found: Tablet not found: 4c18a98d5c0f4a43ae91c9130047b681
I20250903 20:05:15.194118 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 8023fa7537234ce783c2bf90b575b00c (table default.Table_9ce3e6d31c314cd0bdbd59ea1260d46a [id=7032fe3b789041d380f5db114f241e29]) successfully deleted
I20250903 20:05:15.194571 27685 tablet_replica.cc:331] T c9c870fb14e3469fb2979d8462b49121 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:15.194860 27685 raft_consensus.cc:2241] T c9c870fb14e3469fb2979d8462b49121 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:15.194995 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c18a98d5c0f4a43ae91c9130047b681 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:15.195122 27685 raft_consensus.cc:2270] T c9c870fb14e3469fb2979d8462b49121 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:15.195678 27685 ts_tablet_manager.cc:1905] T c9c870fb14e3469fb2979d8462b49121 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:15.196523 27363 tablet_service.cc:1468] Processing CreateTablet for tablet 200c06f296b94eab94f35f312c5a443b (DEFAULT_TABLE table=default.Table_0c416f6e2e1b4ae4941c2e5613f16bda [id=37960c5e67734caa951c9a151b0ece51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.197017 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 200c06f296b94eab94f35f312c5a443b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.199115 27692 tablet_bootstrap.cc:492] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:15.199177 27693 tablet_bootstrap.cc:492] T 8945085bdc174e6db22a5a9477810297 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:15.199368 27693 ts_tablet_manager.cc:1397] T 8945085bdc174e6db22a5a9477810297 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.133s user 0.013s sys 0.022s
I20250903 20:05:15.199471 27692 ts_tablet_manager.cc:1397] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20250903 20:05:15.200242 27692 raft_consensus.cc:357] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.200330 27693 raft_consensus.cc:357] T 8945085bdc174e6db22a5a9477810297 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.200559 27692 raft_consensus.cc:383] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:15.200552 27693 raft_consensus.cc:383] T 8945085bdc174e6db22a5a9477810297 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:15.200667 27693 raft_consensus.cc:738] T 8945085bdc174e6db22a5a9477810297 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:15.200671 27692 raft_consensus.cc:738] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:15.200973 27692 consensus_queue.cc:260] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.201084 27693 consensus_queue.cc:260] T 8945085bdc174e6db22a5a9477810297 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.201709 27693 ts_tablet_manager.cc:1428] T 8945085bdc174e6db22a5a9477810297 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:15.202992 27693 tablet_bootstrap.cc:492] T 925c0ab4873041da82702cd9377ec7bc P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:15.207646 27495 tablet_service.cc:1468] Processing CreateTablet for tablet 200c06f296b94eab94f35f312c5a443b (DEFAULT_TABLE table=default.Table_0c416f6e2e1b4ae4941c2e5613f16bda [id=37960c5e67734caa951c9a151b0ece51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.208092 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 200c06f296b94eab94f35f312c5a443b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.209894 27692 ts_tablet_manager.cc:1428] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20250903 20:05:15.212674 27692 tablet_bootstrap.cc:492] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:15.215745 27693 tablet_bootstrap.cc:654] T 925c0ab4873041da82702cd9377ec7bc P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:15.217661 27695 tablet_bootstrap.cc:492] T 264127511a234e9fbf409f3aa2860ccc P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:15.217831 27695 ts_tablet_manager.cc:1397] T 264127511a234e9fbf409f3aa2860ccc P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.001s
I20250903 20:05:15.219087 27695 raft_consensus.cc:357] T 264127511a234e9fbf409f3aa2860ccc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:15.219275 27695 raft_consensus.cc:383] T 264127511a234e9fbf409f3aa2860ccc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:15.219349 27695 raft_consensus.cc:738] T 264127511a234e9fbf409f3aa2860ccc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:15.219580 27695 consensus_queue.cc:260] T 264127511a234e9fbf409f3aa2860ccc P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:15.237480 27695 ts_tablet_manager.cc:1428] T 264127511a234e9fbf409f3aa2860ccc P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20250903 20:05:15.237574 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 200c06f296b94eab94f35f312c5a443b (DEFAULT_TABLE table=default.Table_0c416f6e2e1b4ae4941c2e5613f16bda [id=37960c5e67734caa951c9a151b0ece51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.238173 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 200c06f296b94eab94f35f312c5a443b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.242166 27695 tablet_bootstrap.cc:492] T eced730bdbc14a8c9f6b7b7e53435fd0 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:15.244118 27695 tablet_bootstrap.cc:654] T eced730bdbc14a8c9f6b7b7e53435fd0 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:15.248878 27687 ts_tablet_manager.cc:1918] T c25aaa6ff310432bb27a0695394ec420 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:15.249048 27687 log.cc:1199] T c25aaa6ff310432bb27a0695394ec420 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/c25aaa6ff310432bb27a0695394ec420
I20250903 20:05:15.249543 27687 ts_tablet_manager.cc:1939] T c25aaa6ff310432bb27a0695394ec420 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
W20250903 20:05:15.251032 27183 consensus_peers.cc:489] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd499b81344c4b58bf77f3622422763c. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:15.260731 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_bd1102072e4a421d9231b7dee60b6232" } new_table_name: "default.Table_04bd610b6ae24395b30dc33c4d118258" modify_external_catalogs: true
I20250903 20:05:15.262482 27692 tablet_bootstrap.cc:654] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:15.268787 27059 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 68cff0e416a54a5eb37071f5a12f07ab (table default.Table_1e3f794ae5ec4ad6a9b4351b4bda7b15 [id=71d850a0cea14b0ba337a822ad99d513]). Expected version 3 got 0
I20250903 20:05:15.277459 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet 262a3549867a473aaaa7dfcc0b5872cd with delete_type TABLET_DATA_DELETED (replaced by tablet 7c398634d6304a12b35c0579fba4c813) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:15.277774 27221 tablet_service.cc:1515] Processing DeleteTablet for tablet 79a0553c912049cca5971b1ad6f7e11c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:15.278008 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 264127511a234e9fbf409f3aa2860ccc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:15.278304 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet 691cb69759654fdc9223844049a06d02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:15.279596 27225 tablet_service.cc:1468] Processing CreateTablet for tablet bf459db3327640f88574fca7b5f2ca98 (DEFAULT_TABLE table=default.Table_b88ff2c2e61049d4a11af5676d28327a [id=b6932927b05f46be967419e342532b3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.280104 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf459db3327640f88574fca7b5f2ca98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.283001 27687 tablet_replica.cc:331] T e5136d5388bc4c2ba71e09be40e8f412 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:15.283232 27687 raft_consensus.cc:2241] T e5136d5388bc4c2ba71e09be40e8f412 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:15.283388 27687 raft_consensus.cc:2270] T e5136d5388bc4c2ba71e09be40e8f412 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:15.286731 27505 tablet_service.cc:1515] Processing DeleteTablet for tablet 4268316c28e8471a9f28807d20dbb395 with delete_type TABLET_DATA_DELETED (replaced by tablet a9f43781299e47a8aecc37a0b73d7a64) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:15.287056 27684 tablet_replica.cc:331] T 4268316c28e8471a9f28807d20dbb395 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:15.287284 27684 raft_consensus.cc:2241] T 4268316c28e8471a9f28807d20dbb395 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:15.287482 27684 raft_consensus.cc:2270] T 4268316c28e8471a9f28807d20dbb395 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:15.288254 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_bc654adac8a24afab120af90f0a73f36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:15.288513 27684 ts_tablet_manager.cc:1905] T 4268316c28e8471a9f28807d20dbb395 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:15.288962 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc654adac8a24afab120af90f0a73f36 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:15.290299 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_19fd54d3e8d8446a8cbe4956067e56d6" } new_table_name: "default.Table_b613060864734552b4f204a8844f7ee8" modify_external_catalogs: true
I20250903 20:05:15.290264 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ee1ad081d0c846ca9fdc08acfb8318e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:15.290853 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee1ad081d0c846ca9fdc08acfb8318e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:15.294922 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_478904b0f29b45aaaa87750ddf9ec041"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:15.295491 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_478904b0f29b45aaaa87750ddf9ec041 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:15.295755 27692 tablet_bootstrap.cc:492] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:15.295941 27692 ts_tablet_manager.cc:1397] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.083s user 0.007s sys 0.001s
I20250903 20:05:15.297066 27692 raft_consensus.cc:357] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.297346 27692 raft_consensus.cc:383] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:15.297477 27692 raft_consensus.cc:738] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:15.297777 27692 consensus_queue.cc:260] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.297879 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 4bcd65022b7b4502983b6b58809047e3 (DEFAULT_TABLE table=default.Table_2746c6354cbc45c296e45155908acb77 [id=7ee5ea1a352941d7ba044bf48f21848a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.298401 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bcd65022b7b4502983b6b58809047e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.299260 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet c25aaa6ff310432bb27a0695394ec420 (table default.Table_835ced1f8b194192809c3b6fbf4d1d96 [id=23f448924ec04ef08a541a14bd2ce5df]) successfully deleted
I20250903 20:05:15.300309 27357 tablet_service.cc:1468] Processing CreateTablet for tablet 4bcd65022b7b4502983b6b58809047e3 (DEFAULT_TABLE table=default.Table_2746c6354cbc45c296e45155908acb77 [id=7ee5ea1a352941d7ba044bf48f21848a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.300736 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bcd65022b7b4502983b6b58809047e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.300766 27695 tablet_bootstrap.cc:492] T eced730bdbc14a8c9f6b7b7e53435fd0 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:15.300917 27695 ts_tablet_manager.cc:1397] T eced730bdbc14a8c9f6b7b7e53435fd0 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.000s
I20250903 20:05:15.301690 27695 raft_consensus.cc:357] T eced730bdbc14a8c9f6b7b7e53435fd0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.301879 27695 raft_consensus.cc:383] T eced730bdbc14a8c9f6b7b7e53435fd0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:15.301941 27695 raft_consensus.cc:738] T eced730bdbc14a8c9f6b7b7e53435fd0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:15.302197 27695 consensus_queue.cc:260] T eced730bdbc14a8c9f6b7b7e53435fd0 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.304514 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 4c18a98d5c0f4a43ae91c9130047b681 (DEFAULT_TABLE table=default.Table_16b62656e76142339c0a42ca5a7f5e6b [id=189c3642e92c43759dee94f23ef1aec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.304881 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c18a98d5c0f4a43ae91c9130047b681. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.305163 28102 tablet.cc:1722] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250903 20:05:15.306664 27695 ts_tablet_manager.cc:1428] T eced730bdbc14a8c9f6b7b7e53435fd0 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250903 20:05:15.306991 27685 ts_tablet_manager.cc:1918] T c9c870fb14e3469fb2979d8462b49121 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:15.307185 27685 log.cc:1199] T c9c870fb14e3469fb2979d8462b49121 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/c9c870fb14e3469fb2979d8462b49121
I20250903 20:05:15.308534 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 4c18a98d5c0f4a43ae91c9130047b681 (DEFAULT_TABLE table=default.Table_16b62656e76142339c0a42ca5a7f5e6b [id=189c3642e92c43759dee94f23ef1aec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.308912 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c18a98d5c0f4a43ae91c9130047b681. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.310551 27365 tablet_service.cc:1468] Processing CreateTablet for tablet e877d56bbbfa4c5a92d025d0311ce5c6 (DEFAULT_TABLE table=default.Table_d9105c35c28247f1a71866f2096d93f2 [id=33b22e921daa4ccf9406155bac438d08]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.310921 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e877d56bbbfa4c5a92d025d0311ce5c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.313380 27692 ts_tablet_manager.cc:1428] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.017s user 0.001s sys 0.002s
I20250903 20:05:15.313578 27685 ts_tablet_manager.cc:1939] T c9c870fb14e3469fb2979d8462b49121 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:15.313805 27695 tablet_bootstrap.cc:492] T fdc77810f27a40758e063b8b1150ab71 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:15.313776 27501 tablet_service.cc:1468] Processing CreateTablet for tablet e877d56bbbfa4c5a92d025d0311ce5c6 (DEFAULT_TABLE table=default.Table_d9105c35c28247f1a71866f2096d93f2 [id=33b22e921daa4ccf9406155bac438d08]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.314100 27685 tablet_replica.cc:331] T 44db427ee53a4d0a907dfd63282d6d4f P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:15.314177 27501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e877d56bbbfa4c5a92d025d0311ce5c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.314323 27685 raft_consensus.cc:2241] T 44db427ee53a4d0a907dfd63282d6d4f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:15.314540 27685 raft_consensus.cc:2270] T 44db427ee53a4d0a907dfd63282d6d4f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:15.314582 27356 tablet_service.cc:1468] Processing CreateTablet for tablet 45f7068bafaf48d290ded9d1272c0e46 (DEFAULT_TABLE table=default.Table_279a72f111474b7b86aa09094dc82ac6 [id=e660a48b356a4b0784016e527303f1f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.314941 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45f7068bafaf48d290ded9d1272c0e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.315130 27685 ts_tablet_manager.cc:1905] T 44db427ee53a4d0a907dfd63282d6d4f P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:15.316491 27692 tablet_bootstrap.cc:492] T 2a5c74a281b44019872a6442d41bc373 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:15.319984 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 4bcd65022b7b4502983b6b58809047e3 (DEFAULT_TABLE table=default.Table_2746c6354cbc45c296e45155908acb77 [id=7ee5ea1a352941d7ba044bf48f21848a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.320654 27221 tablet_service.cc:1468] Processing CreateTablet for tablet 4c18a98d5c0f4a43ae91c9130047b681 (DEFAULT_TABLE table=default.Table_16b62656e76142339c0a42ca5a7f5e6b [id=189c3642e92c43759dee94f23ef1aec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.321218 27695 tablet_bootstrap.cc:654] T fdc77810f27a40758e063b8b1150ab71 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:15.321173 27218 tablet_service.cc:1468] Processing CreateTablet for tablet e877d56bbbfa4c5a92d025d0311ce5c6 (DEFAULT_TABLE table=default.Table_d9105c35c28247f1a71866f2096d93f2 [id=33b22e921daa4ccf9406155bac438d08]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.331462 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet c9c870fb14e3469fb2979d8462b49121 (table default.Table_8b2ba396ac1b424493dbc17297d2d133 [id=3403a412096a4ef98878426a43899d7c]) successfully deleted
I20250903 20:05:15.334241 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ee23ae4ae2834af98c0a3bef517f0e60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:15.334941 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee23ae4ae2834af98c0a3bef517f0e60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:15.344754 28030 raft_consensus.cc:491] T 73f65e7515994b898eec2ea23903aff4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:15.344992 28030 raft_consensus.cc:513] T 73f65e7515994b898eec2ea23903aff4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:15.345757 28030 leader_election.cc:290] T 73f65e7515994b898eec2ea23903aff4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:15.346243 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e877d56bbbfa4c5a92d025d0311ce5c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.352409 27695 tablet_bootstrap.cc:492] T fdc77810f27a40758e063b8b1150ab71 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:15.352582 27695 ts_tablet_manager.cc:1397] T fdc77810f27a40758e063b8b1150ab71 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.000s
I20250903 20:05:15.352571 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 726f4d920f3e47fea41b4e12c021c988
I20250903 20:05:15.353367 27695 raft_consensus.cc:357] T fdc77810f27a40758e063b8b1150ab71 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.353549 27695 raft_consensus.cc:383] T fdc77810f27a40758e063b8b1150ab71 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:15.353623 27695 raft_consensus.cc:738] T fdc77810f27a40758e063b8b1150ab71 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:15.353837 27695 consensus_queue.cc:260] T fdc77810f27a40758e063b8b1150ab71 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.353976 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 9685cba2f73b412382a69dab2c7f50c1
I20250903 20:05:15.354468 27065 catalog_manager.cc:6652] ReplaceTablet: tablet 9685cba2f73b412382a69dab2c7f50c1 deleted and replaced by tablet 47c1287bff1145a5b2ab6d14dbd608c7
I20250903 20:05:15.355647 27695 ts_tablet_manager.cc:1428] T fdc77810f27a40758e063b8b1150ab71 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:15.355667 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 961ade36f7d14eba91bb16d5eae2e254
I20250903 20:05:15.355973 27695 tablet_bootstrap.cc:492] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:15.357749 27695 tablet_bootstrap.cc:654] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:15.357877 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bcd65022b7b4502983b6b58809047e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.359619 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "73f65e7515994b898eec2ea23903aff4" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:15.361363 27184 leader_election.cc:343] T 73f65e7515994b898eec2ea23903aff4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:15.362937 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 45f7068bafaf48d290ded9d1272c0e46 (DEFAULT_TABLE table=default.Table_279a72f111474b7b86aa09094dc82ac6 [id=e660a48b356a4b0784016e527303f1f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.363502 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45f7068bafaf48d290ded9d1272c0e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.364840 28103 tablet.cc:1722] T b6b4713936134681aaef1732d46c6ab2 P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250903 20:05:15.365643 27754 raft_consensus.cc:491] T 9509364ba3ec4e9bad09db09406a978a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:15.365895 27754 raft_consensus.cc:513] T 9509364ba3ec4e9bad09db09406a978a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.367377 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "73f65e7515994b898eec2ea23903aff4" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:15.368176 27754 leader_election.cc:290] T 9509364ba3ec4e9bad09db09406a978a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:15.369339 27320 leader_election.cc:343] T 9509364ba3ec4e9bad09db09406a978a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 9509364ba3ec4e9bad09db09406a978a
I20250903 20:05:15.372228 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet 726f4d920f3e47fea41b4e12c021c988 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:15.372313 27366 tablet_service.cc:1515] Processing DeleteTablet for tablet 9685cba2f73b412382a69dab2c7f50c1 with delete_type TABLET_DATA_DELETED (replaced by tablet 47c1287bff1145a5b2ab6d14dbd608c7) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:15.372468 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 961ade36f7d14eba91bb16d5eae2e254 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:15.380137 27183 leader_election.cc:343] T 73f65e7515994b898eec2ea23903aff4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:15.380138 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c18a98d5c0f4a43ae91c9130047b681. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:15.381085 27321 leader_election.cc:343] T 9509364ba3ec4e9bad09db09406a978a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 9509364ba3ec4e9bad09db09406a978a
I20250903 20:05:15.381269 27321 leader_election.cc:304] T 9509364ba3ec4e9bad09db09406a978a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:15.381834 27754 raft_consensus.cc:2747] T 9509364ba3ec4e9bad09db09406a978a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:15.381896 27183 leader_election.cc:304] T 73f65e7515994b898eec2ea23903aff4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:15.382962 27840 raft_consensus.cc:2747] T 73f65e7515994b898eec2ea23903aff4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:15.385519 27495 tablet_service.cc:1468] Processing CreateTablet for tablet bf459db3327640f88574fca7b5f2ca98 (DEFAULT_TABLE table=default.Table_b88ff2c2e61049d4a11af5676d28327a [id=b6932927b05f46be967419e342532b3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.385946 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf459db3327640f88574fca7b5f2ca98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.387382 27692 tablet_bootstrap.cc:654] T 2a5c74a281b44019872a6442d41bc373 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:15.384048 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet 9685cba2f73b412382a69dab2c7f50c1 with delete_type TABLET_DATA_DELETED (replaced by tablet 47c1287bff1145a5b2ab6d14dbd608c7) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:15.389892 27213 tablet_service.cc:1515] Processing DeleteTablet for tablet 961ade36f7d14eba91bb16d5eae2e254 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:15.390112 28030 raft_consensus.cc:491] T 691cb69759654fdc9223844049a06d02 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:15.391880 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet 726f4d920f3e47fea41b4e12c021c988 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:15.394876 28030 raft_consensus.cc:513] T 691cb69759654fdc9223844049a06d02 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.396700 27685 ts_tablet_manager.cc:1918] T 44db427ee53a4d0a907dfd63282d6d4f P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:15.398433 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "691cb69759654fdc9223844049a06d02" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:15.399585 27184 leader_election.cc:343] T 691cb69759654fdc9223844049a06d02 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:15.401000 27695 tablet_bootstrap.cc:492] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:15.401346 27685 log.cc:1199] T 44db427ee53a4d0a907dfd63282d6d4f P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/44db427ee53a4d0a907dfd63282d6d4f
I20250903 20:05:15.401890 27684 ts_tablet_manager.cc:1918] T 4268316c28e8471a9f28807d20dbb395 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:15.402192 27684 log.cc:1199] T 4268316c28e8471a9f28807d20dbb395 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/4268316c28e8471a9f28807d20dbb395
I20250903 20:05:15.402916 27684 ts_tablet_manager.cc:1939] T 4268316c28e8471a9f28807d20dbb395 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:15.403223 27754 raft_consensus.cc:491] T 8945085bdc174e6db22a5a9477810297 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:15.257982 27363 tablet_service.cc:1468] Processing CreateTablet for tablet bf459db3327640f88574fca7b5f2ca98 (DEFAULT_TABLE table=default.Table_b88ff2c2e61049d4a11af5676d28327a [id=b6932927b05f46be967419e342532b3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.260334 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet 8945085bdc174e6db22a5a9477810297 with delete_type TABLET_DATA_DELETED (replaced by tablet f71d3dd9af2543299296f21381d1d563) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:15.272481 27693 tablet_bootstrap.cc:492] T 925c0ab4873041da82702cd9377ec7bc P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:15.403797 27695 ts_tablet_manager.cc:1397] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.000s
I20250903 20:05:15.403857 27693 ts_tablet_manager.cc:1397] T 925c0ab4873041da82702cd9377ec7bc P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.201s user 0.014s sys 0.019s
I20250903 20:05:15.404230 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf459db3327640f88574fca7b5f2ca98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.404794 27695 raft_consensus.cc:357] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.404955 27693 raft_consensus.cc:357] T 925c0ab4873041da82702cd9377ec7bc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:15.405164 27693 raft_consensus.cc:383] T 925c0ab4873041da82702cd9377ec7bc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:15.405242 27693 raft_consensus.cc:738] T 925c0ab4873041da82702cd9377ec7bc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:15.405498 27693 consensus_queue.cc:260] T 925c0ab4873041da82702cd9377ec7bc P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:15.406008 27693 ts_tablet_manager.cc:1428] T 925c0ab4873041da82702cd9377ec7bc P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:15.403530 27754 raft_consensus.cc:513] T 8945085bdc174e6db22a5a9477810297 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.406843 27754 leader_election.cc:290] T 8945085bdc174e6db22a5a9477810297 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:15.407578 28030 leader_election.cc:290] T 691cb69759654fdc9223844049a06d02 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
W20250903 20:05:15.408608 27320 leader_election.cc:343] T 8945085bdc174e6db22a5a9477810297 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 8945085bdc174e6db22a5a9477810297
W20250903 20:05:15.408996 27321 leader_election.cc:343] T 8945085bdc174e6db22a5a9477810297 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 8945085bdc174e6db22a5a9477810297
I20250903 20:05:15.409122 27321 leader_election.cc:304] T 8945085bdc174e6db22a5a9477810297 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:15.409729 27754 raft_consensus.cc:2747] T 8945085bdc174e6db22a5a9477810297 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:15.410341 27693 tablet_bootstrap.cc:492] T e20fcd28c3ae4355a033d2b8429f84e2 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:15.413786 27693 tablet_bootstrap.cc:654] T e20fcd28c3ae4355a033d2b8429f84e2 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:15.405499 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "691cb69759654fdc9223844049a06d02" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:15.420732 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 9685cba2f73b412382a69dab2c7f50c1 with delete_type TABLET_DATA_DELETED (replaced by tablet 47c1287bff1145a5b2ab6d14dbd608c7) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:15.421216 27501 tablet_service.cc:1515] Processing DeleteTablet for tablet 961ade36f7d14eba91bb16d5eae2e254 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:15.427403 27183 leader_election.cc:343] T 691cb69759654fdc9223844049a06d02 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:15.427599 27183 leader_election.cc:304] T 691cb69759654fdc9223844049a06d02 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:15.428172 28030 raft_consensus.cc:2747] T 691cb69759654fdc9223844049a06d02 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:15.420459 27505 tablet_service.cc:1515] Processing DeleteTablet for tablet 726f4d920f3e47fea41b4e12c021c988 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:15.429603 27386 raft_consensus.cc:3058] T c0343d2226e345669d42375de2545986 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:15.432085 27695 raft_consensus.cc:383] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:15.432197 27695 raft_consensus.cc:738] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:15.432436 27695 consensus_queue.cc:260] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.432658 27221 tablet_service.cc:1468] Processing CreateTablet for tablet 45f7068bafaf48d290ded9d1272c0e46 (DEFAULT_TABLE table=default.Table_279a72f111474b7b86aa09094dc82ac6 [id=e660a48b356a4b0784016e527303f1f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.433243 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45f7068bafaf48d290ded9d1272c0e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.436704 27695 ts_tablet_manager.cc:1428] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.033s user 0.002s sys 0.000s
I20250903 20:05:15.437940 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 4268316c28e8471a9f28807d20dbb395 (table default.Table_6ab02d2c339e463fbed5c3429c7c4ec9 [id=f70d7ea93e154e6199f6766316353862]) successfully deleted
W20250903 20:05:15.439302 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 9685cba2f73b412382a69dab2c7f50c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9685cba2f73b412382a69dab2c7f50c1 already in progress: creating tablet
W20250903 20:05:15.440529 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 961ade36f7d14eba91bb16d5eae2e254 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 961ade36f7d14eba91bb16d5eae2e254 already in progress: creating tablet
I20250903 20:05:15.442060 27364 tablet_service.cc:1468] Processing CreateTablet for tablet 950a6fcd35f64f2d889831fe9cdf503a (DEFAULT_TABLE table=default.Table_9ce3e6d31c314cd0bdbd59ea1260d46a [id=7032fe3b789041d380f5db114f241e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.442482 27695 tablet_bootstrap.cc:492] T a10b079e7d5a440580ac00195fe88c65 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:15.442637 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 950a6fcd35f64f2d889831fe9cdf503a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.444376 27695 tablet_bootstrap.cc:654] T a10b079e7d5a440580ac00195fe88c65 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:15.445398 28101 tablet.cc:1722] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250903 20:05:15.445534 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 950a6fcd35f64f2d889831fe9cdf503a (DEFAULT_TABLE table=default.Table_9ce3e6d31c314cd0bdbd59ea1260d46a [id=7032fe3b789041d380f5db114f241e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.445948 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 950a6fcd35f64f2d889831fe9cdf503a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.449883 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 548b50819f694fa3ad508da9f442a99c (DEFAULT_TABLE table=default.Table_ab7e78ea930c4d88bf2e62caa36ca317 [id=1eb6b859ca0049699466cc72e5a2b6f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.450155 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 548b50819f694fa3ad508da9f442a99c (DEFAULT_TABLE table=default.Table_ab7e78ea930c4d88bf2e62caa36ca317 [id=1eb6b859ca0049699466cc72e5a2b6f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.450611 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 548b50819f694fa3ad508da9f442a99c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.450919 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 548b50819f694fa3ad508da9f442a99c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.452255 27952 raft_consensus.cc:491] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:15.452433 27952 raft_consensus.cc:513] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.453120 27952 leader_election.cc:290] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:15.453953 27687 ts_tablet_manager.cc:1905] T e5136d5388bc4c2ba71e09be40e8f412 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:15.454366 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04ccb3e4a8bd4789b9f19edd4a5b6145" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:15.454592 27249 raft_consensus.cc:2466] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:15.453979 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04ccb3e4a8bd4789b9f19edd4a5b6145" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:15.455837 27462 leader_election.cc:343] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:15.456310 27462 leader_election.cc:304] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:15.456831 27952 raft_consensus.cc:2802] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:15.456921 27952 raft_consensus.cc:491] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:15.457006 27952 raft_consensus.cc:3058] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:15.460299 27952 raft_consensus.cc:513] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.461180 27364 tablet_service.cc:1468] Processing CreateTablet for tablet aa9f5e0964b348b7af184f60cd145666 (DEFAULT_TABLE table=default.Table_14e02c1be9c74bd99f4184faa6753f52 [id=a6dc7550a6d643008c92b3a59ba022ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.461539 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04ccb3e4a8bd4789b9f19edd4a5b6145" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:15.461675 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa9f5e0964b348b7af184f60cd145666. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:15.462148 27462 leader_election.cc:343] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:15.463017 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04ccb3e4a8bd4789b9f19edd4a5b6145" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:15.463253 27249 raft_consensus.cc:3058] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:15.463807 27218 tablet_service.cc:1468] Processing CreateTablet for tablet aa9f5e0964b348b7af184f60cd145666 (DEFAULT_TABLE table=default.Table_14e02c1be9c74bd99f4184faa6753f52 [id=a6dc7550a6d643008c92b3a59ba022ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.464244 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa9f5e0964b348b7af184f60cd145666. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.465181 27685 ts_tablet_manager.cc:1939] T 44db427ee53a4d0a907dfd63282d6d4f P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:15.466243 27685 tablet_replica.cc:331] T 73f65e7515994b898eec2ea23903aff4 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:15.466480 27685 raft_consensus.cc:2241] T 73f65e7515994b898eec2ea23903aff4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:15.466522 27952 leader_election.cc:290] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:15.466678 27685 raft_consensus.cc:2270] T 73f65e7515994b898eec2ea23903aff4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:15.466753 27365 tablet_service.cc:1468] Processing CreateTablet for tablet ed096b452ba747f89e91edc15e6e3ac1 (DEFAULT_TABLE table=default.Table_5bc4b150c12940db90c7800f5d07b9ed [id=27548347ece844e3bb55ba894da5c8e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.467096 27249 raft_consensus.cc:2466] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:15.467173 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed096b452ba747f89e91edc15e6e3ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.467347 27685 ts_tablet_manager.cc:1905] T 73f65e7515994b898eec2ea23903aff4 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:15.467734 27462 leader_election.cc:304] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:15.470422 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 04ccb3e4a8bd4789b9f19edd4a5b6145 with delete_type TABLET_DATA_DELETED (replaced by tablet a4c582b441f6467c99ba0b090abcf4cc) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:15.474644 27685 ts_tablet_manager.cc:1918] T 73f65e7515994b898eec2ea23903aff4 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:15.474903 27685 log.cc:1199] T 73f65e7515994b898eec2ea23903aff4 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/73f65e7515994b898eec2ea23903aff4
I20250903 20:05:15.475327 27685 ts_tablet_manager.cc:1939] T 73f65e7515994b898eec2ea23903aff4 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:15.476442 27952 raft_consensus.cc:2802] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:15.478524 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_f02e6857b868499087599a49fe189d58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:15.479123 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f02e6857b868499087599a49fe189d58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:15.479535 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 886b8783aa9d4618aad498d4a1115843 (DEFAULT_TABLE table=default.Table_a540eea45dfb46ec92b02a3f9a2b3aed [id=5fe055478b724744a65fcaa2f25bef0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.479784 27364 tablet_service.cc:1468] Processing CreateTablet for tablet 1223743918814d88b401782ca4b521dd (DEFAULT_TABLE table=default.Table_f038423efb3f4fd5bba9911713436b97 [id=22a7d6c72c2e4dbfadd44a229bb175e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.479969 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 886b8783aa9d4618aad498d4a1115843. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.479558 27357 tablet_service.cc:1468] Processing CreateTablet for tablet 7a9e55f0811345f7afa71645f38ebdaf (DEFAULT_TABLE table=default.Table_05d58d894486468a8a15aaa4b4c0cabb [id=e69321e60a3a4e71a463a0011c0c6d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.480178 27952 raft_consensus.cc:695] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:15.480392 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a9e55f0811345f7afa71645f38ebdaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.480512 27952 consensus_queue.cc:237] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.481468 27360 tablet_service.cc:1468] Processing CreateTablet for tablet a38f65e992884635aae9942681c0dc4f (DEFAULT_TABLE table=default.Table_b9a8db9323cd4f2286256cf26bcacbac [id=f0171ee670294ce58ab84483fd71ab7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.481993 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a38f65e992884635aae9942681c0dc4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.486402 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1223743918814d88b401782ca4b521dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.486748 27356 tablet_service.cc:1468] Processing CreateTablet for tablet 46f49fd718be4a2996d781ab31268217 (DEFAULT_TABLE table=default.Table_b5110c69d0364c50a92330c1824b8416 [id=889259ae7ac44e97aaff581d77e05009]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.487267 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46f49fd718be4a2996d781ab31268217. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.490605 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 44db427ee53a4d0a907dfd63282d6d4f (table default.Table_8e482428f5d8407795dcd3111aefc596 [id=5c647f2080d043f3851e5767b1efd161]) successfully deleted
I20250903 20:05:15.477604 27220 tablet_service.cc:1468] Processing CreateTablet for tablet a38f65e992884635aae9942681c0dc4f (DEFAULT_TABLE table=default.Table_b9a8db9323cd4f2286256cf26bcacbac [id=f0171ee670294ce58ab84483fd71ab7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.478777 27213 tablet_service.cc:1468] Processing CreateTablet for tablet 7a9e55f0811345f7afa71645f38ebdaf (DEFAULT_TABLE table=default.Table_05d58d894486468a8a15aaa4b4c0cabb [id=e69321e60a3a4e71a463a0011c0c6d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.491582 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a38f65e992884635aae9942681c0dc4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.491813 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a9e55f0811345f7afa71645f38ebdaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.491586 27065 catalog_manager.cc:5582] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f reported cstate change: term changed from 0 to 1, leader changed from <none> to e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67). New cstate: current_term: 1 leader_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:15.480459 27226 tablet_service.cc:1468] Processing CreateTablet for tablet 886b8783aa9d4618aad498d4a1115843 (DEFAULT_TABLE table=default.Table_a540eea45dfb46ec92b02a3f9a2b3aed [id=5fe055478b724744a65fcaa2f25bef0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.493786 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 886b8783aa9d4618aad498d4a1115843. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:15.501974 27051 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 61474.7us
I20250903 20:05:15.502303 28061 raft_consensus.cc:491] T 8023fa7537234ce783c2bf90b575b00c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:15.502486 28061 raft_consensus.cc:513] T 8023fa7537234ce783c2bf90b575b00c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.487577 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 46f49fd718be4a2996d781ab31268217 (DEFAULT_TABLE table=default.Table_b5110c69d0364c50a92330c1824b8416 [id=889259ae7ac44e97aaff581d77e05009]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.503192 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46f49fd718be4a2996d781ab31268217. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.503255 28061 leader_election.cc:290] T 8023fa7537234ce783c2bf90b575b00c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:15.487942 27225 tablet_service.cc:1468] Processing CreateTablet for tablet ed096b452ba747f89e91edc15e6e3ac1 (DEFAULT_TABLE table=default.Table_5bc4b150c12940db90c7800f5d07b9ed [id=27548347ece844e3bb55ba894da5c8e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.504745 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed096b452ba747f89e91edc15e6e3ac1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:15.505532 27462 leader_election.cc:343] T 8023fa7537234ce783c2bf90b575b00c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 8023fa7537234ce783c2bf90b575b00c
I20250903 20:05:15.505923 27695 tablet_bootstrap.cc:492] T a10b079e7d5a440580ac00195fe88c65 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
W20250903 20:05:15.505900 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 726f4d920f3e47fea41b4e12c021c988 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 726f4d920f3e47fea41b4e12c021c988 already in progress: creating tablet
I20250903 20:05:15.506114 27695 ts_tablet_manager.cc:1397] T a10b079e7d5a440580ac00195fe88c65 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.000s
I20250903 20:05:15.507164 27695 raft_consensus.cc:357] T a10b079e7d5a440580ac00195fe88c65 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.507361 27695 raft_consensus.cc:383] T a10b079e7d5a440580ac00195fe88c65 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:15.507464 27695 raft_consensus.cc:738] T a10b079e7d5a440580ac00195fe88c65 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:15.507735 27695 consensus_queue.cc:260] T a10b079e7d5a440580ac00195fe88c65 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.509604 27695 ts_tablet_manager.cc:1428] T a10b079e7d5a440580ac00195fe88c65 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250903 20:05:15.509917 27695 tablet_bootstrap.cc:492] T 6ea37668d64d4feeb92a162711609c13 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:15.510000 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet a0f79831866b40c49c5b27ce2014c3f6
I20250903 20:05:15.510851 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1742550430f54b36b962c92f5c81f08a" } new_table_name: "default.Table_47699f82f0074c7eac5b4b83296c71e5" modify_external_catalogs: true
I20250903 20:05:15.512439 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet a0f79831866b40c49c5b27ce2014c3f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:15.513974 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a223f75045345ae833f52bcc8f239f9 consensus state has no leader
I20250903 20:05:15.514170 27695 tablet_bootstrap.cc:654] T 6ea37668d64d4feeb92a162711609c13 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:15.515435 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 1223743918814d88b401782ca4b521dd (DEFAULT_TABLE table=default.Table_f038423efb3f4fd5bba9911713436b97 [id=22a7d6c72c2e4dbfadd44a229bb175e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.515935 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1223743918814d88b401782ca4b521dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.520737 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 950a6fcd35f64f2d889831fe9cdf503a (DEFAULT_TABLE table=default.Table_9ce3e6d31c314cd0bdbd59ea1260d46a [id=7032fe3b789041d380f5db114f241e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.521220 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 950a6fcd35f64f2d889831fe9cdf503a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.522380 27497 tablet_service.cc:1468] Processing CreateTablet for tablet 46f49fd718be4a2996d781ab31268217 (DEFAULT_TABLE table=default.Table_b5110c69d0364c50a92330c1824b8416 [id=889259ae7ac44e97aaff581d77e05009]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.522708 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet a0f79831866b40c49c5b27ce2014c3f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:15.522810 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46f49fd718be4a2996d781ab31268217. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.525489 27501 tablet_service.cc:1468] Processing CreateTablet for tablet 548b50819f694fa3ad508da9f442a99c (DEFAULT_TABLE table=default.Table_ab7e78ea930c4d88bf2e62caa36ca317 [id=1eb6b859ca0049699466cc72e5a2b6f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.525880 27501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 548b50819f694fa3ad508da9f442a99c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.526611 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8023fa7537234ce783c2bf90b575b00c" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:15.527411 27462 leader_election.cc:343] T 8023fa7537234ce783c2bf90b575b00c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:15.527557 27462 leader_election.cc:304] T 8023fa7537234ce783c2bf90b575b00c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:15.528113 27952 raft_consensus.cc:2747] T 8023fa7537234ce783c2bf90b575b00c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:15.528851 27507 tablet_service.cc:1468] Processing CreateTablet for tablet ed096b452ba747f89e91edc15e6e3ac1 (DEFAULT_TABLE table=default.Table_5bc4b150c12940db90c7800f5d07b9ed [id=27548347ece844e3bb55ba894da5c8e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.529228 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed096b452ba747f89e91edc15e6e3ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.532281 27497 tablet_service.cc:1468] Processing CreateTablet for tablet a38f65e992884635aae9942681c0dc4f (DEFAULT_TABLE table=default.Table_b9a8db9323cd4f2286256cf26bcacbac [id=f0171ee670294ce58ab84483fd71ab7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.532661 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a38f65e992884635aae9942681c0dc4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.533262 27499 tablet_service.cc:1468] Processing CreateTablet for tablet 1223743918814d88b401782ca4b521dd (DEFAULT_TABLE table=default.Table_f038423efb3f4fd5bba9911713436b97 [id=22a7d6c72c2e4dbfadd44a229bb175e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.533650 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1223743918814d88b401782ca4b521dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.534801 28030 raft_consensus.cc:491] T 264127511a234e9fbf409f3aa2860ccc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:15.534957 28030 raft_consensus.cc:513] T 264127511a234e9fbf409f3aa2860ccc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:15.535705 28030 leader_election.cc:290] T 264127511a234e9fbf409f3aa2860ccc P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:15.536720 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "264127511a234e9fbf409f3aa2860ccc" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:15.538393 27952 raft_consensus.cc:491] T f7ab722c37264973b204cd4f1f95052d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:15.538547 27952 raft_consensus.cc:513] T f7ab722c37264973b204cd4f1f95052d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.539073 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "264127511a234e9fbf409f3aa2860ccc" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:15.539191 27952 leader_election.cc:290] T f7ab722c37264973b204cd4f1f95052d P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:15.539794 27183 leader_election.cc:343] T 264127511a234e9fbf409f3aa2860ccc P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:15.540042 27184 leader_election.cc:343] T 264127511a234e9fbf409f3aa2860ccc P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:15.540217 27184 leader_election.cc:304] T 264127511a234e9fbf409f3aa2860ccc P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:15.540248 27496 tablet_service.cc:1468] Processing CreateTablet for tablet 7a9e55f0811345f7afa71645f38ebdaf (DEFAULT_TABLE table=default.Table_05d58d894486468a8a15aaa4b4c0cabb [id=e69321e60a3a4e71a463a0011c0c6d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.540704 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a9e55f0811345f7afa71645f38ebdaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.543637 28030 raft_consensus.cc:2747] T 264127511a234e9fbf409f3aa2860ccc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:15.544700 27462 leader_election.cc:343] T f7ab722c37264973b204cd4f1f95052d P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: f7ab722c37264973b204cd4f1f95052d
I20250903 20:05:15.549306 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7ab722c37264973b204cd4f1f95052d" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:15.549991 27462 leader_election.cc:343] T f7ab722c37264973b204cd4f1f95052d P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:15.550176 27462 leader_election.cc:304] T f7ab722c37264973b204cd4f1f95052d P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:15.550506 27952 raft_consensus.cc:2747] T f7ab722c37264973b204cd4f1f95052d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:15.556878 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 302) took 1017 ms. Trace:
I20250903 20:05:15.557030 27685 rpcz_store.cc:276] 0903 20:05:14.539191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:14.539471 (+ 280us) service_pool.cc:225] Handling call
0903 20:05:15.466142 (+926671us) ts_tablet_manager.cc:1145] Deleting tablet 73f65e7515994b898eec2ea23903aff4
0903 20:05:15.466152 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:15.471174 (+ 5022us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:15.474477 (+ 3303us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:15.556795 (+ 82318us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":926411}
I20250903 20:05:15.557808 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 305) took 1006 ms. Trace:
I20250903 20:05:15.557924 27685 rpcz_store.cc:276] 0903 20:05:14.551129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:14.551194 (+ 65us) service_pool.cc:225] Handling call
0903 20:05:15.557595 (+1006401us) ts_tablet_manager.cc:1145] Deleting tablet 1bcb9d5f010f4297bab78c0183889238
0903 20:05:15.557608 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:15.557738 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1006159}
I20250903 20:05:15.558027 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 73f65e7515994b898eec2ea23903aff4 (table default.Table_d5626bea3c88471b846f37ae4ed79715 [id=75e5cb0bab9b46a6b5caec7c99efc7b8]) successfully deleted
W20250903 20:05:15.558357 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 1bcb9d5f010f4297bab78c0183889238 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1bcb9d5f010f4297bab78c0183889238 already in progress: creating tablet
I20250903 20:05:15.558661 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 306) took 1005 ms. Trace:
I20250903 20:05:15.558753 27685 rpcz_store.cc:276] 0903 20:05:14.553309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:14.553375 (+ 66us) service_pool.cc:225] Handling call
0903 20:05:15.558364 (+1004989us) ts_tablet_manager.cc:1145] Deleting tablet 8d2d64f103fc44d7b390559b92cbf4b4
0903 20:05:15.558375 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:15.558600 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1004703}
I20250903 20:05:15.560570 27952 raft_consensus.cc:491] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:15.560766 27952 raft_consensus.cc:513] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.561437 27952 leader_election.cc:290] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:15.561833 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 8d2d64f103fc44d7b390559b92cbf4b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d2d64f103fc44d7b390559b92cbf4b4 already in progress: creating tablet
I20250903 20:05:15.562114 27685 tablet_replica.cc:331] T 31342583c026438cbbae14106e75bbe5 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
W20250903 20:05:15.562166 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 07565ea2b5f34dcfb407ff3bfde17243 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07565ea2b5f34dcfb407ff3bfde17243 already in progress: creating tablet
I20250903 20:05:15.562364 27685 raft_consensus.cc:2241] T 31342583c026438cbbae14106e75bbe5 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
W20250903 20:05:15.562428 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 8fff36d5a80e45af8a256c3908555e08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8fff36d5a80e45af8a256c3908555e08 already in progress: creating tablet
I20250903 20:05:15.562608 27685 raft_consensus.cc:2270] T 31342583c026438cbbae14106e75bbe5 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:15.562618 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 567630e2a93645e58f9bf05ffff87b32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 567630e2a93645e58f9bf05ffff87b32 already in progress: creating tablet
I20250903 20:05:15.562523 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "262a3549867a473aaaa7dfcc0b5872cd" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:15.562804 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 283a3afed7414a958b2dc4176f00bb01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 283a3afed7414a958b2dc4176f00bb01 already in progress: creating tablet
I20250903 20:05:15.562808 27249 raft_consensus.cc:2466] T 262a3549867a473aaaa7dfcc0b5872cd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
W20250903 20:05:15.563006 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8 already in progress: creating tablet
I20250903 20:05:15.563208 27685 ts_tablet_manager.cc:1905] T 31342583c026438cbbae14106e75bbe5 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:15.563735 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "262a3549867a473aaaa7dfcc0b5872cd" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:15.564283 27462 leader_election.cc:304] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
W20250903 20:05:15.564937 27462 leader_election.cc:343] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:15.565388 27952 raft_consensus.cc:2802] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:15.565513 27952 raft_consensus.cc:491] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:15.565639 27952 raft_consensus.cc:3058] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:15.566888 27495 tablet_service.cc:1468] Processing CreateTablet for tablet aa9f5e0964b348b7af184f60cd145666 (DEFAULT_TABLE table=default.Table_14e02c1be9c74bd99f4184faa6753f52 [id=a6dc7550a6d643008c92b3a59ba022ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.567405 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa9f5e0964b348b7af184f60cd145666. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.568897 27952 raft_consensus.cc:513] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.570173 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "262a3549867a473aaaa7dfcc0b5872cd" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
W20250903 20:05:15.570950 27462 leader_election.cc:343] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:15.572105 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "262a3549867a473aaaa7dfcc0b5872cd" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:15.572418 27249 raft_consensus.cc:3058] T 262a3549867a473aaaa7dfcc0b5872cd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:15.576182 28030 raft_consensus.cc:491] T a10b079e7d5a440580ac00195fe88c65 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:15.576457 28030 raft_consensus.cc:513] T a10b079e7d5a440580ac00195fe88c65 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.577415 28030 leader_election.cc:290] T a10b079e7d5a440580ac00195fe88c65 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:15.578452 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a10b079e7d5a440580ac00195fe88c65" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:15.579079 27183 leader_election.cc:343] T a10b079e7d5a440580ac00195fe88c65 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:15.579071 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a10b079e7d5a440580ac00195fe88c65" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:15.579761 27249 raft_consensus.cc:2466] T 262a3549867a473aaaa7dfcc0b5872cd P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:15.580401 27462 leader_election.cc:304] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:15.580776 28061 raft_consensus.cc:2802] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:15.580935 28061 raft_consensus.cc:695] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:15.581235 28061 consensus_queue.cc:237] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.582888 27952 leader_election.cc:290] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:15.579767 27184 leader_election.cc:343] T a10b079e7d5a440580ac00195fe88c65 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:15.583380 27184 leader_election.cc:304] T a10b079e7d5a440580ac00195fe88c65 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:15.584246 28030 raft_consensus.cc:2747] T a10b079e7d5a440580ac00195fe88c65 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:15.586962 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_279a72f111474b7b86aa09094dc82ac6" } new_table_name: "default.Table_d15cadbd10314321848d5f0657333ace" modify_external_catalogs: true
W20250903 20:05:15.587646 27184 consensus_peers.cc:489] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd499b81344c4b58bf77f3622422763c. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:15.588521 27687 ts_tablet_manager.cc:1918] T e5136d5388bc4c2ba71e09be40e8f412 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250903 20:05:15.591156 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24eadc70e10d4d02bcb643d45c9e0e36 has no consensus state
I20250903 20:05:15.591444 27687 log.cc:1199] T e5136d5388bc4c2ba71e09be40e8f412 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/e5136d5388bc4c2ba71e09be40e8f412
I20250903 20:05:15.591483 27061 master_service.cc:948] ReplaceTablet: received request to replace tablet 0a9b7770432942da9373d5100c71bd6a from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:15.591868 27687 ts_tablet_manager.cc:1939] T e5136d5388bc4c2ba71e09be40e8f412 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:15.592913 27687 tablet_replica.cc:331] T 9509364ba3ec4e9bad09db09406a978a P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:15.593103 27687 raft_consensus.cc:2241] T 9509364ba3ec4e9bad09db09406a978a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:15.593281 27687 raft_consensus.cc:2270] T 9509364ba3ec4e9bad09db09406a978a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:15.593875 27687 ts_tablet_manager.cc:1905] T 9509364ba3ec4e9bad09db09406a978a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:15.593842 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_36c6bab093f446f78fe92cf5173f841e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:15.594570 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36c6bab093f446f78fe92cf5173f841e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:15.594951 27693 tablet_bootstrap.cc:492] T e20fcd28c3ae4355a033d2b8429f84e2 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:15.595113 27693 ts_tablet_manager.cc:1397] T e20fcd28c3ae4355a033d2b8429f84e2 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.185s user 0.011s sys 0.002s
I20250903 20:05:15.595204 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_0d7367affcaf4dee8b0a8a1f524a7035"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:15.595780 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d7367affcaf4dee8b0a8a1f524a7035 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:15.595942 27693 raft_consensus.cc:357] T e20fcd28c3ae4355a033d2b8429f84e2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.596139 27693 raft_consensus.cc:383] T e20fcd28c3ae4355a033d2b8429f84e2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:15.596225 27693 raft_consensus.cc:738] T e20fcd28c3ae4355a033d2b8429f84e2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:15.596434 27693 consensus_queue.cc:260] T e20fcd28c3ae4355a033d2b8429f84e2 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.597111 27693 ts_tablet_manager.cc:1428] T e20fcd28c3ae4355a033d2b8429f84e2 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:15.598104 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet e5136d5388bc4c2ba71e09be40e8f412 (table default.Table_9b80dba44fe8402999adc810b230d87a [id=790f57164c7d4183a3df17ba7f230d24]) successfully deleted
W20250903 20:05:15.600863 27051 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 67ca1dcf6fc7435db555392235f9900f (table default.Table_bd23bbbddf114a4286a815ee4468be81 [id=5568f788d1414c83b4359b5adf28632a], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 67ca1dcf6fc7435db555392235f9900f consensus state has no leader
I20250903 20:05:15.606709 27693 tablet_bootstrap.cc:492] T cfa8a1d9011e460681443a6b77b95958 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:15.609416 27693 tablet_bootstrap.cc:654] T cfa8a1d9011e460681443a6b77b95958 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:15.616202 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ed795c89b83a43c68ace2f6652900ca1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:15.616829 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed795c89b83a43c68ace2f6652900ca1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:15.618479 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_fdaf2e3d1322481bb38fbb5298ba40fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:15.618983 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fdaf2e3d1322481bb38fbb5298ba40fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:15.628966 28030 raft_consensus.cc:491] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:15.629163 28030 raft_consensus.cc:513] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.629901 28030 leader_election.cc:290] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
W20250903 20:05:15.630676 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 45f7068bafaf48d290ded9d1272c0e46 consensus state has no leader
W20250903 20:05:15.630708 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 120ca61b3da742dc9a2efa483ead7e97 consensus state has no leader
I20250903 20:05:15.630978 27062 master_service.cc:948] ReplaceTablet: received request to replace tablet 8fff36d5a80e45af8a256c3908555e08 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:15.631254 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e8bc99de4ffe43d58351d170b7c61d8a" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:15.632012 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 0a9b7770432942da9373d5100c71bd6a
W20250903 20:05:15.631999 27184 leader_election.cc:343] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:15.632598 27061 catalog_manager.cc:6652] ReplaceTablet: tablet 0a9b7770432942da9373d5100c71bd6a deleted and replaced by tablet 6800e0b6269045f29682704fd847d5d1
I20250903 20:05:15.633345 27695 tablet_bootstrap.cc:492] T 6ea37668d64d4feeb92a162711609c13 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:15.633538 27695 ts_tablet_manager.cc:1397] T 6ea37668d64d4feeb92a162711609c13 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.124s user 0.026s sys 0.037s
I20250903 20:05:15.634557 27060 master_service.cc:948] ReplaceTablet: received request to replace tablet 1bad3c509460422396a17a3c4ae3d1e8 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:15.634585 27695 raft_consensus.cc:357] T 6ea37668d64d4feeb92a162711609c13 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.634668 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_5e9a58e5be084c1fa563357bdab3d530"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:15.634964 27695 raft_consensus.cc:383] T 6ea37668d64d4feeb92a162711609c13 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:15.635061 27695 raft_consensus.cc:738] T 6ea37668d64d4feeb92a162711609c13 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
W20250903 20:05:15.635336 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e9a58e5be084c1fa563357bdab3d530 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:15.635418 27695 consensus_queue.cc:260] T 6ea37668d64d4feeb92a162711609c13 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.636061 27363 tablet_service.cc:1515] Processing DeleteTablet for tablet 0a9b7770432942da9373d5100c71bd6a with delete_type TABLET_DATA_DELETED (replaced by tablet 6800e0b6269045f29682704fd847d5d1) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:15.638119 27066 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_ac1cd4ca0afa4c3494fa9953c49c838e [id=d0a75545b91c48bba47fb581bb0d1a96] alter complete (version 4)
I20250903 20:05:15.638724 27066 master_service.cc:948] ReplaceTablet: received request to replace tablet 1fbe54dc40dd47daa109ceeb07546b0a from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:15.640671 27065 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_091b5cecb0b04277a78dc078b4f800a6" } modify_external_catalogs: true
I20250903 20:05:15.640702 27064 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5bc4b150c12940db90c7800f5d07b9ed" } modify_external_catalogs: true
I20250903 20:05:15.640892 27064 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5bc4b150c12940db90c7800f5d07b9ed" } modify_external_catalogs: true
I20250903 20:05:15.641872 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_aab58ad5394041179974552994469305" } modify_external_catalogs: true
I20250903 20:05:15.642005 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet 0a9b7770432942da9373d5100c71bd6a with delete_type TABLET_DATA_DELETED (replaced by tablet 6800e0b6269045f29682704fd847d5d1) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:15.642052 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_aab58ad5394041179974552994469305" } modify_external_catalogs: true
I20250903 20:05:15.642882 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet eced730bdbc14a8c9f6b7b7e53435fd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:15.642915 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_118af17580524e4d86fc930e787e92c5" } new_table_name: "default.Table_eaff993b7bb7437580aae238faa74b3e" modify_external_catalogs: true
I20250903 20:05:15.643193 27695 ts_tablet_manager.cc:1428] T 6ea37668d64d4feeb92a162711609c13 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250903 20:05:15.640868 27065 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_091b5cecb0b04277a78dc078b4f800a6" } modify_external_catalogs: true
I20250903 20:05:15.645283 27356 tablet_service.cc:1515] Processing DeleteTablet for tablet e20fcd28c3ae4355a033d2b8429f84e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:15.645845 27363 tablet_service.cc:1515] Processing DeleteTablet for tablet 925c0ab4873041da82702cd9377ec7bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:15.647300 27057 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet e8bc99de4ffe43d58351d170b7c61d8a (table default.Table_c5256f043b61433b8debe9e13625bd0d [id=416ff29f66d14564826a6734dde1c604]). Expected version 2 got 0
I20250903 20:05:15.649677 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ea37668d64d4feeb92a162711609c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:15.650173 27221 tablet_service.cc:1515] Processing DeleteTablet for tablet a10b079e7d5a440580ac00195fe88c65 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b3bfc63aa0b4e56bec189e834dc6a8d) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:15.650458 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet fdc77810f27a40758e063b8b1150ab71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:15.651731 27695 tablet_bootstrap.cc:492] T c9129a3e773a4fecb28ceee27778a294 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:15.654183 27695 tablet_bootstrap.cc:654] T c9129a3e773a4fecb28ceee27778a294 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:15.655961 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b20633c3b14e463aac0b7bd917c3ab9b" } modify_external_catalogs: true
I20250903 20:05:15.656179 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b20633c3b14e463aac0b7bd917c3ab9b" } modify_external_catalogs: true
I20250903 20:05:15.657523 27695 tablet_bootstrap.cc:492] T c9129a3e773a4fecb28ceee27778a294 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:15.657657 27695 ts_tablet_manager.cc:1397] T c9129a3e773a4fecb28ceee27778a294 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20250903 20:05:15.658442 27695 raft_consensus.cc:357] T c9129a3e773a4fecb28ceee27778a294 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.658624 27695 raft_consensus.cc:383] T c9129a3e773a4fecb28ceee27778a294 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:15.658710 27695 raft_consensus.cc:738] T c9129a3e773a4fecb28ceee27778a294 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:15.659386 27695 consensus_queue.cc:260] T c9129a3e773a4fecb28ceee27778a294 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.666350 27693 tablet_bootstrap.cc:492] T cfa8a1d9011e460681443a6b77b95958 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:15.666548 27693 ts_tablet_manager.cc:1397] T cfa8a1d9011e460681443a6b77b95958 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.060s user 0.004s sys 0.019s
I20250903 20:05:15.667084 27695 ts_tablet_manager.cc:1428] T c9129a3e773a4fecb28ceee27778a294 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250903 20:05:15.667413 27693 raft_consensus.cc:357] T cfa8a1d9011e460681443a6b77b95958 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.667621 27693 raft_consensus.cc:383] T cfa8a1d9011e460681443a6b77b95958 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:15.667701 27693 raft_consensus.cc:738] T cfa8a1d9011e460681443a6b77b95958 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:15.667984 27693 consensus_queue.cc:260] T cfa8a1d9011e460681443a6b77b95958 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.668599 27693 ts_tablet_manager.cc:1428] T cfa8a1d9011e460681443a6b77b95958 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:15.669047 27693 tablet_bootstrap.cc:492] T f7ab722c37264973b204cd4f1f95052d P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:15.670625 27063 master_service.cc:948] ReplaceTablet: received request to replace tablet cb4d99139516405bb65c7480ddf259f1 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:15.671280 27693 tablet_bootstrap.cc:654] T f7ab722c37264973b204cd4f1f95052d P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:15.679153 27695 tablet_bootstrap.cc:492] T 5bc3b9771a0042fd87e706204dcbb712 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:15.680989 27695 tablet_bootstrap.cc:654] T 5bc3b9771a0042fd87e706204dcbb712 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:15.683182 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 1bad3c509460422396a17a3c4ae3d1e8
I20250903 20:05:15.683823 27060 catalog_manager.cc:6652] ReplaceTablet: tablet 1bad3c509460422396a17a3c4ae3d1e8 deleted and replaced by tablet 13496efe34944b7280e7a684f180e6e0
I20250903 20:05:15.686335 27695 tablet_bootstrap.cc:492] T 5bc3b9771a0042fd87e706204dcbb712 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:15.686503 27695 ts_tablet_manager.cc:1397] T 5bc3b9771a0042fd87e706204dcbb712 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20250903 20:05:15.686859 27363 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bad3c509460422396a17a3c4ae3d1e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 13496efe34944b7280e7a684f180e6e0) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:15.687163 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bad3c509460422396a17a3c4ae3d1e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 13496efe34944b7280e7a684f180e6e0) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:15.688604 27693 tablet_bootstrap.cc:492] T f7ab722c37264973b204cd4f1f95052d P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:15.688611 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 1fbe54dc40dd47daa109ceeb07546b0a
I20250903 20:05:15.688762 27693 ts_tablet_manager.cc:1397] T f7ab722c37264973b204cd4f1f95052d P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.001s
I20250903 20:05:15.689606 27693 raft_consensus.cc:357] T f7ab722c37264973b204cd4f1f95052d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.689828 27693 raft_consensus.cc:383] T f7ab722c37264973b204cd4f1f95052d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:15.689936 27693 raft_consensus.cc:738] T f7ab722c37264973b204cd4f1f95052d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:15.690193 27693 consensus_queue.cc:260] T f7ab722c37264973b204cd4f1f95052d P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.690469 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet ed096b452ba747f89e91edc15e6e3ac1
I20250903 20:05:15.690770 27693 ts_tablet_manager.cc:1428] T f7ab722c37264973b204cd4f1f95052d P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250903 20:05:15.691059 27693 tablet_bootstrap.cc:492] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
W20250903 20:05:15.693245 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet de8ac9f984be46ec8f021928de12a090 consensus state has no leader
I20250903 20:05:15.693702 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet e9eb24dd5a6243c49a3389320a3f78a0
I20250903 20:05:15.694687 27695 raft_consensus.cc:357] T 5bc3b9771a0042fd87e706204dcbb712 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:15.694909 27695 raft_consensus.cc:383] T 5bc3b9771a0042fd87e706204dcbb712 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:15.694981 27695 raft_consensus.cc:738] T 5bc3b9771a0042fd87e706204dcbb712 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:15.695305 27695 consensus_queue.cc:260] T 5bc3b9771a0042fd87e706204dcbb712 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:15.695981 27363 tablet_service.cc:1515] Processing DeleteTablet for tablet cfa8a1d9011e460681443a6b77b95958 with delete_type TABLET_DATA_DELETED (replaced by tablet 4af8c03c9d834405b4e25d35441db961) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:15.696723 27066 catalog_manager.cc:6652] ReplaceTablet: tablet 1fbe54dc40dd47daa109ceeb07546b0a deleted and replaced by tablet 5b46bb34ac214b1ba23fba1a33f746ac
I20250903 20:05:15.697893 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet ed096b452ba747f89e91edc15e6e3ac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:15.698163 27221 tablet_service.cc:1515] Processing DeleteTablet for tablet e9eb24dd5a6243c49a3389320a3f78a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:15.698452 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet 1fbe54dc40dd47daa109ceeb07546b0a with delete_type TABLET_DATA_DELETED (replaced by tablet 5b46bb34ac214b1ba23fba1a33f746ac) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:15.698734 27356 tablet_service.cc:1515] Processing DeleteTablet for tablet e9eb24dd5a6243c49a3389320a3f78a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:15.699005 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet 1fbe54dc40dd47daa109ceeb07546b0a with delete_type TABLET_DATA_DELETED (replaced by tablet 5b46bb34ac214b1ba23fba1a33f746ac) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:15.699177 27363 tablet_service.cc:1515] Processing DeleteTablet for tablet ed096b452ba747f89e91edc15e6e3ac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:15.701747 27695 ts_tablet_manager.cc:1428] T 5bc3b9771a0042fd87e706204dcbb712 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20250903 20:05:15.702064 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet 5bc3b9771a0042fd87e706204dcbb712 with delete_type TABLET_DATA_DELETED (replaced by tablet 961ade36f7d14eba91bb16d5eae2e254) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:15.706876 27695 tablet_bootstrap.cc:492] T 8b3121fce1c44d16a0b083313dc67b7a P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:15.706951 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet fbfabb6ff1714d2e921f45256b543759
I20250903 20:05:15.707865 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet fbfabb6ff1714d2e921f45256b543759 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:15.707957 27363 tablet_service.cc:1515] Processing DeleteTablet for tablet f7ab722c37264973b204cd4f1f95052d with delete_type TABLET_DATA_DELETED (replaced by tablet 57a1167457b84ea8a493be232b1d088a) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:15.708141 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet fbfabb6ff1714d2e921f45256b543759 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:15.708807 27695 tablet_bootstrap.cc:654] T 8b3121fce1c44d16a0b083313dc67b7a P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:15.709834 27693 tablet_bootstrap.cc:654] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:15.719841 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 5002b7e8118c49b6a247cb8dc4ca1aea
I20250903 20:05:15.720759 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet cb4d99139516405bb65c7480ddf259f1
I20250903 20:05:15.720841 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet 5002b7e8118c49b6a247cb8dc4ca1aea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:15.721478 27063 catalog_manager.cc:6652] ReplaceTablet: tablet cb4d99139516405bb65c7480ddf259f1 deleted and replaced by tablet 6d4b8d4bf29e408cbdf9557c8b0d8a4b
I20250903 20:05:15.721553 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet cb4d99139516405bb65c7480ddf259f1 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d4b8d4bf29e408cbdf9557c8b0d8a4b) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:15.722285 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet 5002b7e8118c49b6a247cb8dc4ca1aea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:15.722278 27221 tablet_service.cc:1515] Processing DeleteTablet for tablet cb4d99139516405bb65c7480ddf259f1 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d4b8d4bf29e408cbdf9557c8b0d8a4b) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:15.722501 27693 tablet_bootstrap.cc:492] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:15.722702 27693 ts_tablet_manager.cc:1397] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.002s
I20250903 20:05:15.723690 27693 raft_consensus.cc:357] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.723965 27693 raft_consensus.cc:383] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:15.724059 27693 raft_consensus.cc:738] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:15.724363 27693 consensus_queue.cc:260] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.730242 27695 tablet_bootstrap.cc:492] T 8b3121fce1c44d16a0b083313dc67b7a P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:15.730486 27695 ts_tablet_manager.cc:1397] T 8b3121fce1c44d16a0b083313dc67b7a P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.009s
I20250903 20:05:15.731421 27695 raft_consensus.cc:357] T 8b3121fce1c44d16a0b083313dc67b7a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.731616 27695 raft_consensus.cc:383] T 8b3121fce1c44d16a0b083313dc67b7a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:15.731714 27695 raft_consensus.cc:738] T 8b3121fce1c44d16a0b083313dc67b7a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:15.731972 27695 consensus_queue.cc:260] T 8b3121fce1c44d16a0b083313dc67b7a P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.732142 27693 ts_tablet_manager.cc:1428] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20250903 20:05:15.732522 27695 ts_tablet_manager.cc:1428] T 8b3121fce1c44d16a0b083313dc67b7a P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:15.732846 27695 tablet_bootstrap.cc:492] T 1fef9400227449ed892daff588999caa P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:15.734794 27695 tablet_bootstrap.cc:654] T 1fef9400227449ed892daff588999caa P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:15.740015 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet 8023fa7537234ce783c2bf90b575b00c with delete_type TABLET_DATA_DELETED (replaced by tablet 950a6fcd35f64f2d889831fe9cdf503a) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:15.745360 27693 tablet_bootstrap.cc:492] T 68cff0e416a54a5eb37071f5a12f07ab P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:15.747879 28061 raft_consensus.cc:491] T e20fcd28c3ae4355a033d2b8429f84e2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:15.748064 28061 raft_consensus.cc:513] T e20fcd28c3ae4355a033d2b8429f84e2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.748298 27221 tablet_service.cc:1515] Processing DeleteTablet for tablet 8b3121fce1c44d16a0b083313dc67b7a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:15.748749 28061 leader_election.cc:290] T e20fcd28c3ae4355a033d2b8429f84e2 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:15.749295 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e8bc99de4ffe43d58351d170b7c61d8a" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:15.751721 27693 tablet_bootstrap.cc:654] T 68cff0e416a54a5eb37071f5a12f07ab P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:15.752466 27692 tablet_bootstrap.cc:492] T 2a5c74a281b44019872a6442d41bc373 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:15.752657 27692 ts_tablet_manager.cc:1397] T 2a5c74a281b44019872a6442d41bc373 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.436s user 0.004s sys 0.009s
I20250903 20:05:15.753517 27692 raft_consensus.cc:357] T 2a5c74a281b44019872a6442d41bc373 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.753726 27692 raft_consensus.cc:383] T 2a5c74a281b44019872a6442d41bc373 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:15.753798 27692 raft_consensus.cc:738] T 2a5c74a281b44019872a6442d41bc373 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:15.754021 27692 consensus_queue.cc:260] T 2a5c74a281b44019872a6442d41bc373 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.754593 27692 ts_tablet_manager.cc:1428] T 2a5c74a281b44019872a6442d41bc373 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:15.754833 27496 tablet_service.cc:1468] Processing CreateTablet for tablet 886b8783aa9d4618aad498d4a1115843 (DEFAULT_TABLE table=default.Table_a540eea45dfb46ec92b02a3f9a2b3aed [id=5fe055478b724744a65fcaa2f25bef0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.755563 27501 tablet_service.cc:1515] Processing DeleteTablet for tablet a0f79831866b40c49c5b27ce2014c3f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:15.758069 27505 tablet_service.cc:1515] Processing DeleteTablet for tablet e20fcd28c3ae4355a033d2b8429f84e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:15.758400 27500 tablet_service.cc:1515] Processing DeleteTablet for tablet 8023fa7537234ce783c2bf90b575b00c with delete_type TABLET_DATA_DELETED (replaced by tablet 950a6fcd35f64f2d889831fe9cdf503a) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:15.758615 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bad3c509460422396a17a3c4ae3d1e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 13496efe34944b7280e7a684f180e6e0) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:15.758783 27498 tablet_service.cc:1515] Processing DeleteTablet for tablet ed096b452ba747f89e91edc15e6e3ac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:15.761119 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet 4268316c28e8471a9f28807d20dbb395 with delete_type TABLET_DATA_DELETED (replaced by tablet a9f43781299e47a8aecc37a0b73d7a64) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:15.762640 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet f7ab722c37264973b204cd4f1f95052d with delete_type TABLET_DATA_DELETED (replaced by tablet 57a1167457b84ea8a493be232b1d088a) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:15.762950 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 0a9b7770432942da9373d5100c71bd6a with delete_type TABLET_DATA_DELETED (replaced by tablet 6800e0b6269045f29682704fd847d5d1) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:15.763587 27503 tablet_service.cc:1515] Processing DeleteTablet for tablet e9eb24dd5a6243c49a3389320a3f78a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:15.764525 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 1fbe54dc40dd47daa109ceeb07546b0a with delete_type TABLET_DATA_DELETED (replaced by tablet 5b46bb34ac214b1ba23fba1a33f746ac) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:15.766099 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20fcd28c3ae4355a033d2b8429f84e2" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:15.766431 27383 raft_consensus.cc:2466] T e20fcd28c3ae4355a033d2b8429f84e2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:15.767189 27492 tablet_service.cc:1515] Processing DeleteTablet for tablet cb4d99139516405bb65c7480ddf259f1 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d4b8d4bf29e408cbdf9557c8b0d8a4b) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:15.767745 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet fbfabb6ff1714d2e921f45256b543759 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:15.768342 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet 5002b7e8118c49b6a247cb8dc4ca1aea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:15.768993 27501 tablet_service.cc:1515] Processing DeleteTablet for tablet 262a3549867a473aaaa7dfcc0b5872cd with delete_type TABLET_DATA_DELETED (replaced by tablet 7c398634d6304a12b35c0579fba4c813) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:15.771612 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 5557dfdd953a41e9bd3a8be4cd689604 (DEFAULT_TABLE table=default.Table_b0d2811b8e4a40f4a34d3be594183ac6 [id=6d5615e3adc24110a36e1486e69cde7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.772130 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5557dfdd953a41e9bd3a8be4cd689604. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.782061 27462 leader_election.cc:304] T e20fcd28c3ae4355a033d2b8429f84e2 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:15.782897 27952 raft_consensus.cc:2802] T e20fcd28c3ae4355a033d2b8429f84e2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:15.784278 27952 raft_consensus.cc:491] T e20fcd28c3ae4355a033d2b8429f84e2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:15.784457 27952 raft_consensus.cc:3058] T e20fcd28c3ae4355a033d2b8429f84e2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:15.786367 27363 tablet_service.cc:1468] Processing CreateTablet for tablet 238f2f30377b42c2852b016018aa3245 (DEFAULT_TABLE table=default.Table_a22a6e16b90a461ba0c890fb7c5791e9 [id=8fe9f7413b124e39bd3c5645312415e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.786877 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 238f2f30377b42c2852b016018aa3245. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.788815 28061 consensus_queue.cc:1035] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20250903 20:05:15.794552 27493 tablet_service.cc:1468] Processing CreateTablet for tablet 5557dfdd953a41e9bd3a8be4cd689604 (DEFAULT_TABLE table=default.Table_b0d2811b8e4a40f4a34d3be594183ac6 [id=6d5615e3adc24110a36e1486e69cde7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.794847 27501 tablet_service.cc:1515] Processing DeleteTablet for tablet 04ccb3e4a8bd4789b9f19edd4a5b6145 with delete_type TABLET_DATA_DELETED (replaced by tablet a4c582b441f6467c99ba0b090abcf4cc) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:15.795619 28100 raft_consensus.cc:491] T 925c0ab4873041da82702cd9377ec7bc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250903 20:05:15.795734 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet a0f79831866b40c49c5b27ce2014c3f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0f79831866b40c49c5b27ce2014c3f6 already in progress: creating tablet
I20250903 20:05:15.795852 28100 raft_consensus.cc:513] T 925c0ab4873041da82702cd9377ec7bc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:15.796890 28100 leader_election.cc:290] T 925c0ab4873041da82702cd9377ec7bc P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
W20250903 20:05:15.798513 27321 leader_election.cc:343] T 925c0ab4873041da82702cd9377ec7bc P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 925c0ab4873041da82702cd9377ec7bc
I20250903 20:05:15.800792 27356 tablet_service.cc:1468] Processing CreateTablet for tablet 8ba2636641654d8596fd2a5b0644d073 (DEFAULT_TABLE table=default.Table_61b231b7d4df47138eae97d28eb75962 [id=4bffa29ea98049718ac3ce35de9813c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.801312 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ba2636641654d8596fd2a5b0644d073. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:15.805182 27320 leader_election.cc:343] T 925c0ab4873041da82702cd9377ec7bc P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 925c0ab4873041da82702cd9377ec7bc
I20250903 20:05:15.805919 27320 leader_election.cc:304] T 925c0ab4873041da82702cd9377ec7bc P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:15.806602 28100 raft_consensus.cc:2747] T 925c0ab4873041da82702cd9377ec7bc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:15.807794 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_be001b7aec594b93a1b45e9313b772ca" } modify_external_catalogs: true
I20250903 20:05:15.807926 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b9a8db9323cd4f2286256cf26bcacbac" } new_table_name: "default.Table_bb8c68be79034a5c8469862c43953131" modify_external_catalogs: true
I20250903 20:05:15.808101 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_be001b7aec594b93a1b45e9313b772ca" } modify_external_catalogs: true
I20250903 20:05:15.809494 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_016e6db78b02447b8777dba09ae06057"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:15.809602 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_58569618f9d04fdfbd005b3d0e15c049"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:15.809844 27692 tablet_bootstrap.cc:492] T 79a0553c912049cca5971b1ad6f7e11c P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
W20250903 20:05:15.810164 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58569618f9d04fdfbd005b3d0e15c049 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:15.810191 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_016e6db78b02447b8777dba09ae06057 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:15.810417 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 886b8783aa9d4618aad498d4a1115843. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.811640 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_fcaf18430cde44f3b4dec80ebf611ce5" } new_table_name: "default.Table_e79f52615fe142b9823a2a99b8b9c406" modify_external_catalogs: true
I20250903 20:05:15.816963 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5557dfdd953a41e9bd3a8be4cd689604. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:15.821985 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a38f65e992884635aae9942681c0dc4f consensus state has no leader
I20250903 20:05:15.824764 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet a4f3230f50ec4c95b528fead8c191224
I20250903 20:05:15.826292 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet a4f3230f50ec4c95b528fead8c191224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:15.827306 27057 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 2a5c74a281b44019872a6442d41bc373 (table default.Table_5c08431ef8e04a6aab9b96ab78615d2f [id=a9ddfed19dab46a7bb4e049483e3996f]). Expected version 2 got 0
I20250903 20:05:15.832221 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 09fa32312c9648fbbd6fd327e9738066 (DEFAULT_TABLE table=default.Table_61ee151cc2184644b05a60d8feb9e3c4 [id=f459194547904acd99011de4c21c4e33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.833618 27501 tablet_service.cc:1468] Processing CreateTablet for tablet 238f2f30377b42c2852b016018aa3245 (DEFAULT_TABLE table=default.Table_a22a6e16b90a461ba0c890fb7c5791e9 [id=8fe9f7413b124e39bd3c5645312415e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.834479 27684 tablet_replica.cc:331] T e20fcd28c3ae4355a033d2b8429f84e2 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:15.835750 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet a4f3230f50ec4c95b528fead8c191224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:15.835932 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09fa32312c9648fbbd6fd327e9738066. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.840090 27363 tablet_service.cc:1468] Processing CreateTablet for tablet f1d22e76ea72441db51f7e9db26eb520 (DEFAULT_TABLE table=default.Table_f92253d7e7aa42949cfa903765b446ca [id=ae5e27c159a9462482d09a79e148f795]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.840659 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1d22e76ea72441db51f7e9db26eb520. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.841063 27501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 238f2f30377b42c2852b016018aa3245. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.844048 27952 raft_consensus.cc:513] T e20fcd28c3ae4355a033d2b8429f84e2 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.846856 27693 tablet_bootstrap.cc:492] T 68cff0e416a54a5eb37071f5a12f07ab P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:15.847030 27693 ts_tablet_manager.cc:1397] T 68cff0e416a54a5eb37071f5a12f07ab P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.102s user 0.025s sys 0.040s
I20250903 20:05:15.847218 27356 tablet_service.cc:1468] Processing CreateTablet for tablet f326ec776ba446be8d5dda5c1dc5b6ec (DEFAULT_TABLE table=default.Table_b2a82a2c310140caaeac5d0dcb4bb61a [id=e2548ad469e54a41af7c0181357d2467]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.847741 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f326ec776ba446be8d5dda5c1dc5b6ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.847965 27693 raft_consensus.cc:357] T 68cff0e416a54a5eb37071f5a12f07ab P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.848225 27693 raft_consensus.cc:383] T 68cff0e416a54a5eb37071f5a12f07ab P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:15.848322 27693 raft_consensus.cc:738] T 68cff0e416a54a5eb37071f5a12f07ab P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:15.848960 27684 raft_consensus.cc:2241] T e20fcd28c3ae4355a033d2b8429f84e2 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:15.850782 27687 ts_tablet_manager.cc:1918] T 9509364ba3ec4e9bad09db09406a978a P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:15.851003 27687 log.cc:1199] T 9509364ba3ec4e9bad09db09406a978a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/9509364ba3ec4e9bad09db09406a978a
I20250903 20:05:15.850927 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e20fcd28c3ae4355a033d2b8429f84e2" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:15.851224 27386 raft_consensus.cc:3058] T e20fcd28c3ae4355a033d2b8429f84e2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:15.851258 27693 consensus_queue.cc:260] T 68cff0e416a54a5eb37071f5a12f07ab P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.851776 27684 raft_consensus.cc:2270] T e20fcd28c3ae4355a033d2b8429f84e2 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:15.851927 27693 ts_tablet_manager.cc:1428] T 68cff0e416a54a5eb37071f5a12f07ab P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250903 20:05:15.852284 27693 tablet_bootstrap.cc:492] T 4268316c28e8471a9f28807d20dbb395 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
W20250903 20:05:15.853014 27462 leader_election.cc:343] T e20fcd28c3ae4355a033d2b8429f84e2 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: e20fcd28c3ae4355a033d2b8429f84e2
I20250903 20:05:15.855062 27386 raft_consensus.cc:2466] T e20fcd28c3ae4355a033d2b8429f84e2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:15.858502 27364 tablet_service.cc:1468] Processing CreateTablet for tablet 24eadc70e10d4d02bcb643d45c9e0e36 (DEFAULT_TABLE table=default.Table_b613060864734552b4f204a8844f7ee8 [id=e9a291cff3f4497cad9393ab4c571bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.859097 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24eadc70e10d4d02bcb643d45c9e0e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.861033 27462 leader_election.cc:304] T e20fcd28c3ae4355a033d2b8429f84e2 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
W20250903 20:05:15.861590 27462 raft_consensus.cc:2684] T e20fcd28c3ae4355a033d2b8429f84e2 P e72ef1dbcc5143cf9f04a808f101c80f: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250903 20:05:15.862004 27692 tablet_bootstrap.cc:654] T 79a0553c912049cca5971b1ad6f7e11c P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:15.862991 27952 leader_election.cc:290] T e20fcd28c3ae4355a033d2b8429f84e2 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:15.864113 27684 ts_tablet_manager.cc:1905] T e20fcd28c3ae4355a033d2b8429f84e2 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:15.874624 27687 ts_tablet_manager.cc:1939] T 9509364ba3ec4e9bad09db09406a978a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:15.876076 27494 tablet_service.cc:1468] Processing CreateTablet for tablet 09fa32312c9648fbbd6fd327e9738066 (DEFAULT_TABLE table=default.Table_61ee151cc2184644b05a60d8feb9e3c4 [id=f459194547904acd99011de4c21c4e33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.876821 27494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09fa32312c9648fbbd6fd327e9738066. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.878096 28030 raft_consensus.cc:491] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:15.878403 28030 raft_consensus.cc:513] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.879307 28030 leader_election.cc:290] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:15.880767 27492 tablet_service.cc:1468] Processing CreateTablet for tablet f1d22e76ea72441db51f7e9db26eb520 (DEFAULT_TABLE table=default.Table_f92253d7e7aa42949cfa903765b446ca [id=ae5e27c159a9462482d09a79e148f795]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.882870 27492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1d22e76ea72441db51f7e9db26eb520. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.883916 27506 tablet_service.cc:1468] Processing CreateTablet for tablet f326ec776ba446be8d5dda5c1dc5b6ec (DEFAULT_TABLE table=default.Table_b2a82a2c310140caaeac5d0dcb4bb61a [id=e2548ad469e54a41af7c0181357d2467]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.885671 27364 tablet_service.cc:1468] Processing CreateTablet for tablet 472f0fb01263468a828ec13795656624 (DEFAULT_TABLE table=default.Table_bd2731bed3d34406a8867671f5f461a2 [id=86709866cc264317a260253a1ffc6afb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.886269 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 472f0fb01263468a828ec13795656624. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.886468 27693 tablet_bootstrap.cc:654] T 4268316c28e8471a9f28807d20dbb395 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:15.886644 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f326ec776ba446be8d5dda5c1dc5b6ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.886497 27497 tablet_service.cc:1468] Processing CreateTablet for tablet 8ba2636641654d8596fd2a5b0644d073 (DEFAULT_TABLE table=default.Table_61b231b7d4df47138eae97d28eb75962 [id=4bffa29ea98049718ac3ce35de9813c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.890028 27693 tablet_bootstrap.cc:492] T 4268316c28e8471a9f28807d20dbb395 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:15.890242 27693 ts_tablet_manager.cc:1397] T 4268316c28e8471a9f28807d20dbb395 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.003s
I20250903 20:05:15.893184 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ba2636641654d8596fd2a5b0644d073. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.893546 27503 tablet_service.cc:1468] Processing CreateTablet for tablet 24eadc70e10d4d02bcb643d45c9e0e36 (DEFAULT_TABLE table=default.Table_b613060864734552b4f204a8844f7ee8 [id=e9a291cff3f4497cad9393ab4c571bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.894193 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24eadc70e10d4d02bcb643d45c9e0e36. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:15.895094 27462 leader_election.cc:343] T e20fcd28c3ae4355a033d2b8429f84e2 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: e20fcd28c3ae4355a033d2b8429f84e2
I20250903 20:05:15.895834 27693 raft_consensus.cc:357] T 4268316c28e8471a9f28807d20dbb395 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.895913 27357 tablet_service.cc:1468] Processing CreateTablet for tablet cb2f50683e904dd39ee6eb128717d38c (DEFAULT_TABLE table=default.Table_e914faf335d44d019f91bda71758c473 [id=374b9abdf6a54b38876a96d8ec4a0368]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.896062 27693 raft_consensus.cc:383] T 4268316c28e8471a9f28807d20dbb395 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:15.896137 27693 raft_consensus.cc:738] T 4268316c28e8471a9f28807d20dbb395 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:15.896387 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb2f50683e904dd39ee6eb128717d38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.904424 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79a0553c912049cca5971b1ad6f7e11c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:15.905159 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 8ba2636641654d8596fd2a5b0644d073 (DEFAULT_TABLE table=default.Table_61b231b7d4df47138eae97d28eb75962 [id=4bffa29ea98049718ac3ce35de9813c5]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:15.905370 27184 leader_election.cc:343] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:15.905910 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 472f0fb01263468a828ec13795656624 (DEFAULT_TABLE table=default.Table_bd2731bed3d34406a8867671f5f461a2 [id=86709866cc264317a260253a1ffc6afb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.906749 27221 tablet_service.cc:1468] Processing CreateTablet for tablet 5557dfdd953a41e9bd3a8be4cd689604 (DEFAULT_TABLE table=default.Table_b0d2811b8e4a40f4a34d3be594183ac6 [id=6d5615e3adc24110a36e1486e69cde7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.907326 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_9ce3e6d31c314cd0bdbd59ea1260d46a" } new_table_name: "default.Table_7dae8ff5ca9744708d223312079a4315" modify_external_catalogs: true
I20250903 20:05:15.908178 27059 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 2a5c74a281b44019872a6442d41bc373 (table default.Table_5c08431ef8e04a6aab9b96ab78615d2f [id=a9ddfed19dab46a7bb4e049483e3996f]). Expected version 2 got 0
I20250903 20:05:15.909044 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_91e306fcee12497e98a9d2da6302ba2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:15.909654 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91e306fcee12497e98a9d2da6302ba2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:15.910053 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_afa82ec7c58343adb623622c56033b9c" } new_table_name: "default.Table_5ca9587a0da045e8a6baf89e4ee8b537" modify_external_catalogs: true
I20250903 20:05:15.910468 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_d9878f04c3404885815ae53a9054de14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:15.911057 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9878f04c3404885815ae53a9054de14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:15.911855 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_9a599b6ff41449a6ba1b8239b7472262"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:15.912364 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_d84f067d5a924a23964078d360422878"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:15.912557 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a599b6ff41449a6ba1b8239b7472262 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:15.912842 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d84f067d5a924a23964078d360422878 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:15.914242 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_49ef7a892d00492d9e3b45f6dc7f2670"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:15.914873 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_49ef7a892d00492d9e3b45f6dc7f2670 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:15.915584 27498 tablet_service.cc:1468] Processing CreateTablet for tablet 472f0fb01263468a828ec13795656624 (DEFAULT_TABLE table=default.Table_bd2731bed3d34406a8867671f5f461a2 [id=86709866cc264317a260253a1ffc6afb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.916124 27498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 472f0fb01263468a828ec13795656624. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.917747 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_4ba98389160f4a7c802792a41f6bd00e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:15.918421 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ba98389160f4a7c802792a41f6bd00e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:15.921483 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 472f0fb01263468a828ec13795656624. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.923527 27213 tablet_service.cc:1468] Processing CreateTablet for tablet 24eadc70e10d4d02bcb643d45c9e0e36 (DEFAULT_TABLE table=default.Table_b613060864734552b4f204a8844f7ee8 [id=e9a291cff3f4497cad9393ab4c571bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.923959 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet a4f3230f50ec4c95b528fead8c191224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:15.924190 27220 tablet_service.cc:1468] Processing CreateTablet for tablet f326ec776ba446be8d5dda5c1dc5b6ec (DEFAULT_TABLE table=default.Table_b2a82a2c310140caaeac5d0dcb4bb61a [id=e2548ad469e54a41af7c0181357d2467]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.924939 27223 tablet_service.cc:1468] Processing CreateTablet for tablet cb2f50683e904dd39ee6eb128717d38c (DEFAULT_TABLE table=default.Table_e914faf335d44d019f91bda71758c473 [id=374b9abdf6a54b38876a96d8ec4a0368]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.925242 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 09fa32312c9648fbbd6fd327e9738066 (DEFAULT_TABLE table=default.Table_61ee151cc2184644b05a60d8feb9e3c4 [id=f459194547904acd99011de4c21c4e33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.925550 27226 tablet_service.cc:1468] Processing CreateTablet for tablet f1d22e76ea72441db51f7e9db26eb520 (DEFAULT_TABLE table=default.Table_f92253d7e7aa42949cfa903765b446ca [id=ae5e27c159a9462482d09a79e148f795]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.924508 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 238f2f30377b42c2852b016018aa3245 (DEFAULT_TABLE table=default.Table_a22a6e16b90a461ba0c890fb7c5791e9 [id=8fe9f7413b124e39bd3c5645312415e0]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:15.926069 27183 leader_election.cc:343] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:15.926930 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5557dfdd953a41e9bd3a8be4cd689604. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.936273 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_64dabe496cb94cfb9061d0d7eefcbd3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:15.937081 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64dabe496cb94cfb9061d0d7eefcbd3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:15.939311 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 950a6fcd35f64f2d889831fe9cdf503a consensus state has no leader
I20250903 20:05:15.939867 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ea59a50e05934c84a8cf4e8f198d7093"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:15.940932 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea59a50e05934c84a8cf4e8f198d7093 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:15.942646 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 516830fc42e84af693b3af52caee30ee consensus state has no leader
I20250903 20:05:15.943044 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_03a48bb2af164512844bcc33bef7f521" } new_table_name: "default.Table_f7800a262084435690aa7507a259155c" modify_external_catalogs: true
I20250903 20:05:15.956259 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1e3f794ae5ec4ad6a9b4351b4bda7b15" } modify_external_catalogs: true
I20250903 20:05:15.956548 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1e3f794ae5ec4ad6a9b4351b4bda7b15" } modify_external_catalogs: true
W20250903 20:05:15.957549 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 793d6157927143d4b9d8dbafd7cf0fcb consensus state has no leader
I20250903 20:05:15.957913 27065 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0cd65e00bd5a40bd9dc9db12fd0bda4d" } modify_external_catalogs: true
I20250903 20:05:15.958108 27065 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0cd65e00bd5a40bd9dc9db12fd0bda4d" } modify_external_catalogs: true
I20250903 20:05:15.963083 27693 consensus_queue.cc:260] T 4268316c28e8471a9f28807d20dbb395 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:15.963804 27693 ts_tablet_manager.cc:1428] T 4268316c28e8471a9f28807d20dbb395 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.073s user 0.000s sys 0.002s
I20250903 20:05:15.967971 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 793d6157927143d4b9d8dbafd7cf0fcb (DEFAULT_TABLE table=default.Table_fcaf18430cde44f3b4dec80ebf611ce5 [id=3432f6ac076b4e49b9fe63bf5fce8591]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.968317 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 9509364ba3ec4e9bad09db09406a978a (table default.Table_03ce0e151fcc4259ba52504c9c5ddfcd [id=f261e84232234244824d0232efbb4399]) successfully deleted
I20250903 20:05:15.968734 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 793d6157927143d4b9d8dbafd7cf0fcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:15.972667 26885 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskH7eH49/build/asan/bin/kudu pid 26961
I20250903 20:05:15.973999 27025 generic_service.cc:196] Checking for leaks (request via RPC)
W20250903 20:05:15.976751 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 22bc87d1a4804b6c80f77f9692500d63 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22bc87d1a4804b6c80f77f9692500d63 already in progress: creating tablet
I20250903 20:05:15.982247 27361 tablet_service.cc:1468] Processing CreateTablet for tablet a3103e1595ec4c38ba65dca26fe1167c (DEFAULT_TABLE table=default.Table_fdf0ad621407452eb74b6a9a56be2e85 [id=e4270da20ece47aaa8897f47a02d86d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:15.986122 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_47699f82f0074c7eac5b4b83296c71e5" } modify_external_catalogs: true
I20250903 20:05:15.986436 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_47699f82f0074c7eac5b4b83296c71e5" } modify_external_catalogs: true
I20250903 20:05:15.987931 27059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a22a6e16b90a461ba0c890fb7c5791e9" } modify_external_catalogs: true
I20250903 20:05:15.988126 27059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a22a6e16b90a461ba0c890fb7c5791e9" } modify_external_catalogs: true
W20250903 20:05:15.989857 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0f5b87c1c4a24f8abdbfd6194f3b2c4a consensus state has no leader
I20250903 20:05:15.991160 27693 tablet_bootstrap.cc:492] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
W20250903 20:05:15.993103 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 4c18a98d5c0f4a43ae91c9130047b681 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c18a98d5c0f4a43ae91c9130047b681 already in progress: creating tablet
I20250903 20:05:15.993468 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3103e1595ec4c38ba65dca26fe1167c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:15.994690 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 57a1167457b84ea8a493be232b1d088a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57a1167457b84ea8a493be232b1d088a already in progress: creating tablet
I20250903 20:05:16.001761 28100 raft_consensus.cc:491] T cfa8a1d9011e460681443a6b77b95958 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:16.011658 28100 raft_consensus.cc:513] T cfa8a1d9011e460681443a6b77b95958 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:16.016218 28100 leader_election.cc:290] T cfa8a1d9011e460681443a6b77b95958 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:16.016433 28100 raft_consensus.cc:491] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:16.016602 28100 raft_consensus.cc:513] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:16.017467 28100 leader_election.cc:290] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:16.015364 27693 tablet_bootstrap.cc:654] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:16.017915 27360 tablet_service.cc:1468] Processing CreateTablet for tablet ee40566e497740789575275d0cdca0db (DEFAULT_TABLE table=default.Table_ee1ad081d0c846ca9fdc08acfb8318e9 [id=1794b6e2ac9443718c798588325d4a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.022189 27363 tablet_service.cc:1468] Processing CreateTablet for tablet 4e34d6d5d68e4395aed78821ea077c1b (DEFAULT_TABLE table=default.Table_3a0bcaebe62e46e79afba5b118eeb855 [id=08f0b93efef74aa59b1ec3c286b3b45b]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:16.023257 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 4c18a98d5c0f4a43ae91c9130047b681 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c18a98d5c0f4a43ae91c9130047b681 already in progress: creating tablet
I20250903 20:05:16.024102 27356 tablet_service.cc:1468] Processing CreateTablet for tablet 47c1287bff1145a5b2ab6d14dbd608c7 (DEFAULT_TABLE table=default.Table_0cd65e00bd5a40bd9dc9db12fd0bda4d [id=15369700921549be8faa450d2ca98949]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:16.024895 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 726f4d920f3e47fea41b4e12c021c988 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 726f4d920f3e47fea41b4e12c021c988 already in progress: creating tablet
I20250903 20:05:16.026118 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47c1287bff1145a5b2ab6d14dbd608c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.030447 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee40566e497740789575275d0cdca0db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.034500 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e34d6d5d68e4395aed78821ea077c1b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:16.038956 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 9685cba2f73b412382a69dab2c7f50c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9685cba2f73b412382a69dab2c7f50c1 already in progress: creating tablet
W20250903 20:05:16.039935 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 961ade36f7d14eba91bb16d5eae2e254 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 961ade36f7d14eba91bb16d5eae2e254 already in progress: creating tablet
I20250903 20:05:16.040380 27687 tablet_replica.cc:331] T 8945085bdc174e6db22a5a9477810297 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:16.040354 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e9cb6a34300a4887b169da952eee37a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:16.040707 27687 raft_consensus.cc:2241] T 8945085bdc174e6db22a5a9477810297 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:16.040899 27687 raft_consensus.cc:2270] T 8945085bdc174e6db22a5a9477810297 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:16.041309 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9cb6a34300a4887b169da952eee37a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.041564 27687 ts_tablet_manager.cc:1905] T 8945085bdc174e6db22a5a9477810297 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:16.043141 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_97be735f442746fe92867a2df71838a0" } new_table_name: "default.Table_5eacd4f8a0974bdbabb297ffd1cd4a98" modify_external_catalogs: true
I20250903 20:05:16.046128 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 68cff0e416a54a5eb37071f5a12f07ab
I20250903 20:05:16.047118 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 47c1287bff1145a5b2ab6d14dbd608c7
I20250903 20:05:16.047782 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 238f2f30377b42c2852b016018aa3245
I20250903 20:05:16.048544 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 120ca61b3da742dc9a2efa483ead7e97
I20250903 20:05:16.048662 28030 raft_consensus.cc:491] T 6ea37668d64d4feeb92a162711609c13 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:16.048890 28030 raft_consensus.cc:513] T 6ea37668d64d4feeb92a162711609c13 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:16.049615 28030 leader_election.cc:290] T 6ea37668d64d4feeb92a162711609c13 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:16.057090 27504 tablet_service.cc:1468] Processing CreateTablet for tablet a3103e1595ec4c38ba65dca26fe1167c (DEFAULT_TABLE table=default.Table_fdf0ad621407452eb74b6a9a56be2e85 [id=e4270da20ece47aaa8897f47a02d86d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.057668 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3103e1595ec4c38ba65dca26fe1167c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.058719 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7f07933eeaf8443384edb1e59396e36e" } modify_external_catalogs: true
I20250903 20:05:16.058924 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7f07933eeaf8443384edb1e59396e36e" } modify_external_catalogs: true
I20250903 20:05:16.059369 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 47c1287bff1145a5b2ab6d14dbd608c7 (DEFAULT_TABLE table=default.Table_0cd65e00bd5a40bd9dc9db12fd0bda4d [id=15369700921549be8faa450d2ca98949]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.059784 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47c1287bff1145a5b2ab6d14dbd608c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.061286 27499 tablet_service.cc:1468] Processing CreateTablet for tablet 4e34d6d5d68e4395aed78821ea077c1b (DEFAULT_TABLE table=default.Table_3a0bcaebe62e46e79afba5b118eeb855 [id=08f0b93efef74aa59b1ec3c286b3b45b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.061697 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e34d6d5d68e4395aed78821ea077c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.063198 28100 raft_consensus.cc:491] T f7ab722c37264973b204cd4f1f95052d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:16.063386 28100 raft_consensus.cc:513] T f7ab722c37264973b204cd4f1f95052d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:16.064124 28100 leader_election.cc:290] T f7ab722c37264973b204cd4f1f95052d P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:16.069231 27500 tablet_service.cc:1468] Processing CreateTablet for tablet cb2f50683e904dd39ee6eb128717d38c (DEFAULT_TABLE table=default.Table_e914faf335d44d019f91bda71758c473 [id=374b9abdf6a54b38876a96d8ec4a0368]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.069710 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb2f50683e904dd39ee6eb128717d38c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:16.070911 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 912eee5b48ef467bbd6bbe34f8aa390c consensus state has no leader
I20250903 20:05:16.078157 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet e78e6d212daf4589a6b4b65711e356cb
I20250903 20:05:16.088574 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet 68cff0e416a54a5eb37071f5a12f07ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:16.088886 27505 tablet_service.cc:1515] Processing DeleteTablet for tablet 47c1287bff1145a5b2ab6d14dbd608c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:16.089382 27491 tablet_service.cc:1515] Processing DeleteTablet for tablet 238f2f30377b42c2852b016018aa3245 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:16.082528 27692 tablet_bootstrap.cc:492] T 79a0553c912049cca5971b1ad6f7e11c P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:16.139757 27692 ts_tablet_manager.cc:1397] T 79a0553c912049cca5971b1ad6f7e11c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.349s user 0.014s sys 0.010s
I20250903 20:05:16.147212 27692 raft_consensus.cc:357] T 79a0553c912049cca5971b1ad6f7e11c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:16.125981 27684 ts_tablet_manager.cc:1918] T e20fcd28c3ae4355a033d2b8429f84e2 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:16.161721 27684 log.cc:1199] T e20fcd28c3ae4355a033d2b8429f84e2 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/e20fcd28c3ae4355a033d2b8429f84e2
I20250903 20:05:16.162087 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_af52d46a98f249a7982d675a93661af0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:16.162374 27684 ts_tablet_manager.cc:1939] T e20fcd28c3ae4355a033d2b8429f84e2 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:16.162451 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ed795c89b83a43c68ace2f6652900ca1" } new_table_name: "default.Table_c9ba436d242a498582b9009fa5d299e4" modify_external_catalogs: true
W20250903 20:05:16.162762 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af52d46a98f249a7982d675a93661af0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.163764 27684 tablet_replica.cc:331] T f7ab722c37264973b204cd4f1f95052d P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:16.164073 27684 raft_consensus.cc:2241] T f7ab722c37264973b204cd4f1f95052d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:16.164220 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2f6f89e1dd48493f9e989d4526caffed" } new_table_name: "default.Table_1ce23fe2aa6d4aceb99b7f02a2e6d04b" modify_external_catalogs: true
I20250903 20:05:16.164309 27684 raft_consensus.cc:2270] T f7ab722c37264973b204cd4f1f95052d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:16.165047 27684 ts_tablet_manager.cc:1905] T f7ab722c37264973b204cd4f1f95052d P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:16.169692 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_07ba95129d584160ae8b768f77aa044d" } new_table_name: "default.Table_876161e56d1d4c4c9d6726471530d965" modify_external_catalogs: true
I20250903 20:05:16.139158 27496 tablet_service.cc:1468] Processing CreateTablet for tablet ee40566e497740789575275d0cdca0db (DEFAULT_TABLE table=default.Table_ee1ad081d0c846ca9fdc08acfb8318e9 [id=1794b6e2ac9443718c798588325d4a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.170586 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_f05d6db7f87643ebbb52602f99530905"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:16.170778 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee40566e497740789575275d0cdca0db. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:16.171139 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f05d6db7f87643ebbb52602f99530905 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.172408 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_8c565fdbc0e641bfaee34e0b42f855ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:16.172950 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c565fdbc0e641bfaee34e0b42f855ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:16.174497 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 17f16e9c8f794e03a2f8b3b6b35f8e84 consensus state has no leader
I20250903 20:05:16.174998 27692 raft_consensus.cc:383] T 79a0553c912049cca5971b1ad6f7e11c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:16.175138 27692 raft_consensus.cc:738] T 79a0553c912049cca5971b1ad6f7e11c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:16.155035 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 0fef9aa7153a4902abce522e3ff0df3f (DEFAULT_TABLE table=default.Table_4ba98389160f4a7c802792a41f6bd00e [id=f8544f28af82478aa3762f84b040a6d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.155836 27494 tablet_service.cc:1468] Processing CreateTablet for tablet e2441aa3c2c748bba5360259fb663c81 (DEFAULT_TABLE table=default.Table_64dabe496cb94cfb9061d0d7eefcbd3a [id=741df86b1206422b9467c06cab14ddce]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:16.177697 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 976322efa9464ed7a2175d3e3de51dab consensus state has no leader
I20250903 20:05:16.177873 27494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2441aa3c2c748bba5360259fb663c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.156205 27492 tablet_service.cc:1515] Processing DeleteTablet for tablet e78e6d212daf4589a6b4b65711e356cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:16.135412 27490 tablet_service.cc:1515] Processing DeleteTablet for tablet 120ca61b3da742dc9a2efa483ead7e97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:16.136507 27498 tablet_service.cc:1468] Processing CreateTablet for tablet de20d18d7f264811a0631aa1144aef2c (DEFAULT_TABLE table=default.Table_ea59a50e05934c84a8cf4e8f198d7093 [id=1fdfd927e5654aad860585fc23ce5f8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.180450 27498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de20d18d7f264811a0631aa1144aef2c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:16.184504 27059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f9709dfd18654c5bb071e39da7c8889f consensus state has no leader
W20250903 20:05:16.135087 27495 consensus_meta.cc:287] T aa9f5e0964b348b7af184f60cd145666 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent flushing consensus metadata: real 0.564s user 0.004s sys 0.000s
I20250903 20:05:16.188735 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fef9aa7153a4902abce522e3ff0df3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.193053 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e1977f95b5174bb2b40623d225aff528"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:16.193842 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1977f95b5174bb2b40623d225aff528 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.198926 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_cc45498ec439442797707c4fdd3692bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:16.199589 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc45498ec439442797707c4fdd3692bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.200894 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_119f28a77b72449f9374b78ff4b7e016"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:16.201433 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_119f28a77b72449f9374b78ff4b7e016 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.202924 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_98cd4cb4df914678b7bb584d97bd0ca9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:16.203423 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98cd4cb4df914678b7bb584d97bd0ca9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.204515 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_4ba98389160f4a7c802792a41f6bd00e" } new_table_name: "default.Table_c304f6eaf4014caba10a5b376c13972d" modify_external_catalogs: true
I20250903 20:05:16.210703 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ec03f537112c436ba9cce72cf2ff59a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:16.211313 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec03f537112c436ba9cce72cf2ff59a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.197394 27692 consensus_queue.cc:260] T 79a0553c912049cca5971b1ad6f7e11c P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:16.198299 27501 tablet_service.cc:1468] Processing CreateTablet for tablet 793d6157927143d4b9d8dbafd7cf0fcb (DEFAULT_TABLE table=default.Table_e79f52615fe142b9823a2a99b8b9c406 [id=3432f6ac076b4e49b9fe63bf5fce8591]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.216521 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_c750811a8d6c4069a3192ca531cf5712"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:16.217177 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c750811a8d6c4069a3192ca531cf5712 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.199702 27492 tablet_service.cc:1468] Processing CreateTablet for tablet 39c2ebe4ada5426590ec3769c8662644 (DEFAULT_TABLE table=default.Table_d84f067d5a924a23964078d360422878 [id=5cb314181f104d63b1aed98c7198d5c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.219367 27492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39c2ebe4ada5426590ec3769c8662644. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:16.221508 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0fef9aa7153a4902abce522e3ff0df3f consensus state has no leader
I20250903 20:05:16.200253 27495 tablet_service.cc:1468] Processing CreateTablet for tablet 30cc0d4ba8d444f1b98232f906ea0c7b (DEFAULT_TABLE table=default.Table_91e306fcee12497e98a9d2da6302ba2b [id=57668609851c4124b73a3797bb6d2eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.212473 27490 tablet_service.cc:1468] Processing CreateTablet for tablet 64c4dd2645034a7eaf243f5474603f10 (DEFAULT_TABLE table=default.Table_d9878f04c3404885815ae53a9054de14 [id=a98d20608a054ad29a951bb47cf61711]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.222934 27490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64c4dd2645034a7eaf243f5474603f10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.230552 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30cc0d4ba8d444f1b98232f906ea0c7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.239813 27501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 793d6157927143d4b9d8dbafd7cf0fcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.246039 27183 leader_election.cc:304] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:16.260926 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b2a82a2c310140caaeac5d0dcb4bb61a" } modify_external_catalogs: true
I20250903 20:05:16.261245 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b2a82a2c310140caaeac5d0dcb4bb61a" } modify_external_catalogs: true
I20250903 20:05:16.262617 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_119f28a77b72449f9374b78ff4b7e016" } modify_external_catalogs: true
I20250903 20:05:16.262827 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_119f28a77b72449f9374b78ff4b7e016" } modify_external_catalogs: true
I20250903 20:05:16.262898 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a540eea45dfb46ec92b02a3f9a2b3aed" } modify_external_catalogs: true
I20250903 20:05:16.263053 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a540eea45dfb46ec92b02a3f9a2b3aed" } modify_external_catalogs: true
I20250903 20:05:16.263671 27065 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7dae8ff5ca9744708d223312079a4315" } modify_external_catalogs: true
I20250903 20:05:16.263862 27065 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7dae8ff5ca9744708d223312079a4315" } modify_external_catalogs: true
I20250903 20:05:16.266618 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 238f2f30377b42c2852b016018aa3245. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:16.271052 27183 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 504467us
I20250903 20:05:16.272933 28030 raft_consensus.cc:2747] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:16.273880 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_62f55f5f45434b01b7b7e2d6d1ddc6fd" } new_table_name: "default.Table_f6b26b9f6ef340989d2250b8d83647c2" modify_external_catalogs: true
I20250903 20:05:16.274439 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_5187f549b3d44cce99c527429ce39a4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:16.275005 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5187f549b3d44cce99c527429ce39a4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.278347 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 793d6157927143d4b9d8dbafd7cf0fcb (DEFAULT_TABLE table=default.Table_e79f52615fe142b9823a2a99b8b9c406 [id=3432f6ac076b4e49b9fe63bf5fce8591]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.280786 28030 raft_consensus.cc:491] T fdc77810f27a40758e063b8b1150ab71 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:16.280961 28030 raft_consensus.cc:513] T fdc77810f27a40758e063b8b1150ab71 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:16.281765 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a0582ccb94a64603a5fbdd0e531a9edf" } modify_external_catalogs: true
I20250903 20:05:16.281978 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a0582ccb94a64603a5fbdd0e531a9edf" } modify_external_catalogs: true
W20250903 20:05:16.286760 27066 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ac9b5d4f39db4f05a15316585040d142 consensus state has no leader
I20250903 20:05:16.287643 28030 leader_election.cc:290] T fdc77810f27a40758e063b8b1150ab71 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:16.293345 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 793d6157927143d4b9d8dbafd7cf0fcb. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:16.299702 27321 leader_election.cc:343] T cfa8a1d9011e460681443a6b77b95958 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: cfa8a1d9011e460681443a6b77b95958
W20250903 20:05:16.300009 27321 leader_election.cc:343] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 8023fa7537234ce783c2bf90b575b00c
W20250903 20:05:16.300192 27321 leader_election.cc:343] T f7ab722c37264973b204cd4f1f95052d P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: f7ab722c37264973b204cd4f1f95052d
I20250903 20:05:16.308415 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 886b8783aa9d4618aad498d4a1115843
I20250903 20:05:16.317345 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet f326ec776ba446be8d5dda5c1dc5b6ec
I20250903 20:05:16.321630 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 9c77bc0e8bb34712bf8109cadbc0fc06
I20250903 20:05:16.322466 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 950a6fcd35f64f2d889831fe9cdf503a
I20250903 20:05:16.323603 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet f71d3dd9af2543299296f21381d1d563
I20250903 20:05:16.330677 27497 tablet_service.cc:1468] Processing CreateTablet for tablet 7824a0bccdc9485abde124572e502791 (DEFAULT_TABLE table=default.Table_9a599b6ff41449a6ba1b8239b7472262 [id=c1afbb6949b9423399ab4fdc2bd49239]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.343519 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 68cff0e416a54a5eb37071f5a12f07ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:16.346427 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 47c1287bff1145a5b2ab6d14dbd608c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:16.347884 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 120ca61b3da742dc9a2efa483ead7e97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:16.348034 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet e78e6d212daf4589a6b4b65711e356cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:16.351954 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 238f2f30377b42c2852b016018aa3245 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:16.352790 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79a0553c912049cca5971b1ad6f7e11c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:16.354280 27358 tablet_service.cc:1468] Processing CreateTablet for tablet 64c4dd2645034a7eaf243f5474603f10 (DEFAULT_TABLE table=default.Table_d9878f04c3404885815ae53a9054de14 [id=a98d20608a054ad29a951bb47cf61711]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.356267 27359 tablet_service.cc:1468] Processing CreateTablet for tablet 30cc0d4ba8d444f1b98232f906ea0c7b (DEFAULT_TABLE table=default.Table_91e306fcee12497e98a9d2da6302ba2b [id=57668609851c4124b73a3797bb6d2eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.356554 27692 ts_tablet_manager.cc:1428] T 79a0553c912049cca5971b1ad6f7e11c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.158s user 0.003s sys 0.000s
I20250903 20:05:16.356810 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64c4dd2645034a7eaf243f5474603f10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.356657 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 0fef9aa7153a4902abce522e3ff0df3f (DEFAULT_TABLE table=default.Table_4ba98389160f4a7c802792a41f6bd00e [id=f8544f28af82478aa3762f84b040a6d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.358234 27362 tablet_service.cc:1468] Processing CreateTablet for tablet 39c2ebe4ada5426590ec3769c8662644 (DEFAULT_TABLE table=default.Table_d84f067d5a924a23964078d360422878 [id=5cb314181f104d63b1aed98c7198d5c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.359875 27350 tablet_service.cc:1468] Processing CreateTablet for tablet 4e9a6a6e05464b7b97ae624f51402cf0 (DEFAULT_TABLE table=default.Table_f02e6857b868499087599a49fe189d58 [id=4eecee99ccf149d49f2f95c27b5b05ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.355365 27352 tablet_service.cc:1468] Processing CreateTablet for tablet 3f303233e6c347289650c6cf83f8127f (DEFAULT_TABLE table=default.Table_58569618f9d04fdfbd005b3d0e15c049 [id=1cbfdd865e51404688e08bc60eec2197]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.355814 27351 tablet_service.cc:1468] Processing CreateTablet for tablet 259bc73ca9df47e9874692725a60bb1c (DEFAULT_TABLE table=default.Table_5e9a58e5be084c1fa563357bdab3d530 [id=60cc385fc1044966b4a12e13fdc7b5bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.362459 27357 tablet_service.cc:1468] Processing CreateTablet for tablet de20d18d7f264811a0631aa1144aef2c (DEFAULT_TABLE table=default.Table_ea59a50e05934c84a8cf4e8f198d7093 [id=1fdfd927e5654aad860585fc23ce5f8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.366616 27490 tablet_service.cc:1468] Processing CreateTablet for tablet 3f303233e6c347289650c6cf83f8127f (DEFAULT_TABLE table=default.Table_58569618f9d04fdfbd005b3d0e15c049 [id=1cbfdd865e51404688e08bc60eec2197]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.364800 27361 tablet_service.cc:1468] Processing CreateTablet for tablet e2441aa3c2c748bba5360259fb663c81 (DEFAULT_TABLE table=default.Table_64dabe496cb94cfb9061d0d7eefcbd3a [id=741df86b1206422b9467c06cab14ddce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.368515 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30cc0d4ba8d444f1b98232f906ea0c7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.368888 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f303233e6c347289650c6cf83f8127f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.369012 27355 tablet_service.cc:1468] Processing CreateTablet for tablet 2385d3006f90486890ebe43e16a53d7a (DEFAULT_TABLE table=default.Table_478904b0f29b45aaaa87750ddf9ec041 [id=05afe1b4c02b403ab3402946b9b13c40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.369557 27355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2385d3006f90486890ebe43e16a53d7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.370107 27366 tablet_service.cc:1468] Processing CreateTablet for tablet 7824a0bccdc9485abde124572e502791 (DEFAULT_TABLE table=default.Table_9a599b6ff41449a6ba1b8239b7472262 [id=c1afbb6949b9423399ab4fdc2bd49239]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.370510 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7824a0bccdc9485abde124572e502791. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.370882 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39c2ebe4ada5426590ec3769c8662644. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.371616 27353 tablet_service.cc:1468] Processing CreateTablet for tablet d4c703dafbca4e669dbb18946ffac699 (DEFAULT_TABLE table=default.Table_bc654adac8a24afab120af90f0a73f36 [id=e6c1c8665bd841b5a932b2d54f9be4e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.371990 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de20d18d7f264811a0631aa1144aef2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.372206 27353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4c703dafbca4e669dbb18946ffac699. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.373800 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e9a6a6e05464b7b97ae624f51402cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.374043 27351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 259bc73ca9df47e9874692725a60bb1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.375072 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fef9aa7153a4902abce522e3ff0df3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.378391 27692 tablet_bootstrap.cc:492] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:16.380846 27057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1480d4baa8e149bb80b62f09d4cf3f90" } new_table_name: "default.Table_6b2c13e18ac747b4bb616391101d313b" modify_external_catalogs: true
I20250903 20:05:16.382481 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_61b231b7d4df47138eae97d28eb75962" } new_table_name: "default.Table_e7148e16606e4cd3908c86b439f75b78" modify_external_catalogs: true
I20250903 20:05:16.378250 27354 tablet_service.cc:1468] Processing CreateTablet for tablet 3919f7f95a954f59b68f56d0051c0109 (DEFAULT_TABLE table=default.Table_36c6bab093f446f78fe92cf5173f841e [id=147416d6b2ea4002953902e413c5a57e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.382673 27347 tablet_service.cc:1468] Processing CreateTablet for tablet 17f16e9c8f794e03a2f8b3b6b35f8e84 (DEFAULT_TABLE table=default.Table_ed795c89b83a43c68ace2f6652900ca1 [id=d7a41e621fde497d8fcfb842c1e6f9e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.382908 27348 tablet_service.cc:1468] Processing CreateTablet for tablet 5b46bb34ac214b1ba23fba1a33f746ac (DEFAULT_TABLE table=default.Table_03ce0e151fcc4259ba52504c9c5ddfcd [id=f261e84232234244824d0232efbb4399]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.385809 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_fc325fc838f44caa818ddcddb486d85f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:16.386473 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc325fc838f44caa818ddcddb486d85f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.386930 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2441aa3c2c748bba5360259fb663c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.388180 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_30721ff3b68e4c08bdb5845184d8ada7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:16.388779 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30721ff3b68e4c08bdb5845184d8ada7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.390103 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7592bead271d4c929d27693318d7c1c3" } new_table_name: "default.Table_24e55a7c820c413f8514bd3ad7bd3269" modify_external_catalogs: true
I20250903 20:05:16.393024 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b46bb34ac214b1ba23fba1a33f746ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.393151 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_5215f44c06fc49d6a9a740f77e9c21fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:16.393678 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5215f44c06fc49d6a9a740f77e9c21fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.393978 27354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3919f7f95a954f59b68f56d0051c0109. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.394829 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17f16e9c8f794e03a2f8b3b6b35f8e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.397039 27693 tablet_bootstrap.cc:492] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:16.397748 27693 ts_tablet_manager.cc:1397] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.430s user 0.010s sys 0.002s
I20250903 20:05:16.396997 27349 tablet_service.cc:1468] Processing CreateTablet for tablet f4d3e1cfdfab4be1a9143b78bf3bc46d (DEFAULT_TABLE table=default.Table_016e6db78b02447b8777dba09ae06057 [id=386b8d4e7c4042d1947c19ac464c75a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.399067 27363 tablet_service.cc:1468] Processing CreateTablet for tablet b4b15d77d07c4fa7856d9c198cf15450 (DEFAULT_TABLE table=default.Table_fdaf2e3d1322481bb38fbb5298ba40fc [id=45cd523acd3c45b994d1b9cc6eb569ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.400094 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24eadc70e10d4d02bcb643d45c9e0e36. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:16.402400 27695 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.606s user 0.075s sys 0.109s
W20250903 20:05:16.404423 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8ba2636641654d8596fd2a5b0644d073 consensus state has no leader
W20250903 20:05:16.405967 27695 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.667s user 0.075s sys 0.109s
I20250903 20:05:16.406454 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ea37668d64d4feeb92a162711609c13" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:16.410058 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0ad64c699b294679b6fec37b5ad23528 consensus state has no leader
I20250903 20:05:16.403445 28030 consensus_queue.cc:1035] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.058s
I20250903 20:05:16.412879 27952 consensus_queue.cc:1035] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.100s
I20250903 20:05:16.415608 27525 raft_consensus.cc:2466] T 79a0553c912049cca5971b1ad6f7e11c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:16.415829 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7824a0bccdc9485abde124572e502791. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.416365 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_4d90c70f5e544f839c056ae573d08f58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:16.416954 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d90c70f5e544f839c056ae573d08f58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.417635 27500 tablet_service.cc:1468] Processing CreateTablet for tablet 259bc73ca9df47e9874692725a60bb1c (DEFAULT_TABLE table=default.Table_5e9a58e5be084c1fa563357bdab3d530 [id=60cc385fc1044966b4a12e13fdc7b5bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.418871 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 259bc73ca9df47e9874692725a60bb1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.420029 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 3919f7f95a954f59b68f56d0051c0109 (DEFAULT_TABLE table=default.Table_36c6bab093f446f78fe92cf5173f841e [id=147416d6b2ea4002953902e413c5a57e]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:16.427783 27057 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 53881ad30bc843839008ace4012a3f70 consensus state has no leader
I20250903 20:05:16.429646 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_dbc7009c4de44b8e88f3caa4b62ec665"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:16.430305 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbc7009c4de44b8e88f3caa4b62ec665 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.431654 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4b15d77d07c4fa7856d9c198cf15450. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.431759 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e4f1c7374cd94b529c3e923355498939"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:16.432235 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e4f1c7374cd94b529c3e923355498939 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.432606 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7ab722c37264973b204cd4f1f95052d" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:16.432883 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_2015884055c845e5a7bed9306978c4ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:16.433318 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2015884055c845e5a7bed9306978c4ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.435527 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_594fcf5c34de4f7287d7b7f605271d6c" } new_table_name: "default.Table_ef539157e90548248025f3570f2bab2d" modify_external_catalogs: true
I20250903 20:05:16.437065 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_00f2d88f1c284af3b6fbcaf8590938d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:16.437537 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00f2d88f1c284af3b6fbcaf8590938d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.440235 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ff2b25e34cab4943949d8b2d514bd98f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:16.440781 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff2b25e34cab4943949d8b2d514bd98f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.458046 28132 raft_consensus.cc:491] T 68cff0e416a54a5eb37071f5a12f07ab P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:16.458281 28132 raft_consensus.cc:513] T 68cff0e416a54a5eb37071f5a12f07ab P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:16.459488 27693 raft_consensus.cc:357] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:16.462332 27499 tablet_service.cc:1468] Processing CreateTablet for tablet 4e9a6a6e05464b7b97ae624f51402cf0 (DEFAULT_TABLE table=default.Table_f02e6857b868499087599a49fe189d58 [id=4eecee99ccf149d49f2f95c27b5b05ca]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:16.463023 27320 leader_election.cc:343] T cfa8a1d9011e460681443a6b77b95958 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: cfa8a1d9011e460681443a6b77b95958
I20250903 20:05:16.417913 27523 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fdc77810f27a40758e063b8b1150ab71" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:16.468335 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 2385d3006f90486890ebe43e16a53d7a (DEFAULT_TABLE table=default.Table_478904b0f29b45aaaa87750ddf9ec041 [id=05afe1b4c02b403ab3402946b9b13c40]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:16.473759 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fc634116a78f45dd833034ddda90e9bf consensus state has no leader
I20250903 20:05:16.418890 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8023fa7537234ce783c2bf90b575b00c" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:16.476954 27522 raft_consensus.cc:2466] T 8023fa7537234ce783c2bf90b575b00c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:16.484475 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_4eafb1a74bd54d2f89763f3c3cadd0f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:16.485349 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4eafb1a74bd54d2f89763f3c3cadd0f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.485809 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4d3e1cfdfab4be1a9143b78bf3bc46d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.488425 27693 raft_consensus.cc:383] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:16.490043 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a98f50198fb2489fad7fa669a4250ee7" } new_table_name: "default.Table_46dac9447e4447f69f428c6c525b0bf2" modify_external_catalogs: true
I20250903 20:05:16.504788 27493 tablet_service.cc:1468] Processing CreateTablet for tablet d4c703dafbca4e669dbb18946ffac699 (DEFAULT_TABLE table=default.Table_bc654adac8a24afab120af90f0a73f36 [id=e6c1c8665bd841b5a932b2d54f9be4e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.508353 27693 raft_consensus.cc:738] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:16.510457 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_187e391a6d6d4d0d96d4bb8a0581face" } modify_external_catalogs: true
I20250903 20:05:16.510694 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_187e391a6d6d4d0d96d4bb8a0581face" } modify_external_catalogs: true
I20250903 20:05:16.514361 27320 leader_election.cc:304] T cfa8a1d9011e460681443a6b77b95958 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:16.514907 28132 leader_election.cc:290] T 68cff0e416a54a5eb37071f5a12f07ab P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:16.515189 28100 raft_consensus.cc:2747] T cfa8a1d9011e460681443a6b77b95958 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:16.515724 27687 ts_tablet_manager.cc:1918] T 8945085bdc174e6db22a5a9477810297 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:16.515856 27355 tablet_service.cc:1468] Processing CreateTablet for tablet 1bf570855b204b96acd070587c434463 (DEFAULT_TABLE table=default.Table_49ef7a892d00492d9e3b45f6dc7f2670 [id=5ff9845a7d804adeaaee6199ac7faee6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.516536 27490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f303233e6c347289650c6cf83f8127f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.519457 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e4f1c7374cd94b529c3e923355498939" } modify_external_catalogs: true
I20250903 20:05:16.519692 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e4f1c7374cd94b529c3e923355498939" } modify_external_catalogs: true
I20250903 20:05:16.519912 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4c703dafbca4e669dbb18946ffac699. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.520205 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3919f7f95a954f59b68f56d0051c0109. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.522735 27491 tablet_service.cc:1468] Processing CreateTablet for tablet 5b46bb34ac214b1ba23fba1a33f746ac (DEFAULT_TABLE table=default.Table_03ce0e151fcc4259ba52504c9c5ddfcd [id=f261e84232234244824d0232efbb4399]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.524338 27065 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ee1ad081d0c846ca9fdc08acfb8318e9" } modify_external_catalogs: true
I20250903 20:05:16.524569 27065 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ee1ad081d0c846ca9fdc08acfb8318e9" } modify_external_catalogs: true
I20250903 20:05:16.525514 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7a6b693ddf594ff6821359a4a7fbee01" } modify_external_catalogs: true
I20250903 20:05:16.525612 27502 tablet_service.cc:1468] Processing CreateTablet for tablet b4b15d77d07c4fa7856d9c198cf15450 (DEFAULT_TABLE table=default.Table_fdaf2e3d1322481bb38fbb5298ba40fc [id=45cd523acd3c45b994d1b9cc6eb569ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.525715 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7a6b693ddf594ff6821359a4a7fbee01" } modify_external_catalogs: true
I20250903 20:05:16.525959 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 17f16e9c8f794e03a2f8b3b6b35f8e84 (DEFAULT_TABLE table=default.Table_ed795c89b83a43c68ace2f6652900ca1 [id=d7a41e621fde497d8fcfb842c1e6f9e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.521189 27503 tablet_service.cc:1468] Processing CreateTablet for tablet f4d3e1cfdfab4be1a9143b78bf3bc46d (DEFAULT_TABLE table=default.Table_016e6db78b02447b8777dba09ae06057 [id=386b8d4e7c4042d1947c19ac464c75a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.528097 27059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e9cb6a34300a4887b169da952eee37a5" } modify_external_catalogs: true
I20250903 20:05:16.528318 27059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e9cb6a34300a4887b169da952eee37a5" } modify_external_catalogs: true
I20250903 20:05:16.529780 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2385d3006f90486890ebe43e16a53d7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.536840 27489 tablet_service.cc:1468] Processing CreateTablet for tablet 1bf570855b204b96acd070587c434463 (DEFAULT_TABLE table=default.Table_49ef7a892d00492d9e3b45f6dc7f2670 [id=5ff9845a7d804adeaaee6199ac7faee6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.541116 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17f16e9c8f794e03a2f8b3b6b35f8e84. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:16.544987 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 13496efe34944b7280e7a684f180e6e0 consensus state has no leader
I20250903 20:05:16.549657 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 5fb2426aa01e47598f268d0a90475271
I20250903 20:05:16.551167 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet e0347eab6c8642beabdc2d7e7bbfdc9b
I20250903 20:05:16.552007 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet ee40566e497740789575275d0cdca0db
I20250903 20:05:16.552695 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 33fb6b357bda43e7abeaed5634183ef6
I20250903 20:05:16.553340 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 60c32370a5274347afafc6a7e1e94270
I20250903 20:05:16.553349 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 4739f10128d64e2c986257b1d9deb90f (DEFAULT_TABLE table=default.Table_cc45498ec439442797707c4fdd3692bc [id=45d7231b92f54017bd1f5eeb837fa166]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:16.554558 27364 consensus_meta.cc:287] T 472f0fb01263468a828ec13795656624 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent flushing consensus metadata: real 0.611s user 0.006s sys 0.003s
W20250903 20:05:16.555500 27320 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 69156.8us
I20250903 20:05:16.557368 27355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bf570855b204b96acd070587c434463. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:16.561167 27183 leader_election.cc:343] T 6ea37668d64d4feeb92a162711609c13 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:16.561353 27183 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 141191us
I20250903 20:05:16.561939 27183 leader_election.cc:304] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:16.564780 28136 raft_consensus.cc:491] T 8b3121fce1c44d16a0b083313dc67b7a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:16.564944 28136 raft_consensus.cc:513] T 8b3121fce1c44d16a0b083313dc67b7a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:16.565603 28136 leader_election.cc:290] T 8b3121fce1c44d16a0b083313dc67b7a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:16.566588 27695 tablet_bootstrap.cc:492] T 1fef9400227449ed892daff588999caa P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:16.566762 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f326ec776ba446be8d5dda5c1dc5b6ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.567528 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4d3e1cfdfab4be1a9143b78bf3bc46d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.568616 27687 log.cc:1199] T 8945085bdc174e6db22a5a9477810297 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/8945085bdc174e6db22a5a9477810297
I20250903 20:05:16.570057 27695 ts_tablet_manager.cc:1397] T 1fef9400227449ed892daff588999caa P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.834s user 0.078s sys 0.109s
I20250903 20:05:16.570930 27695 raft_consensus.cc:357] T 1fef9400227449ed892daff588999caa P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:16.574347 27695 raft_consensus.cc:383] T 1fef9400227449ed892daff588999caa P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:16.574462 27695 raft_consensus.cc:738] T 1fef9400227449ed892daff588999caa P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:16.577869 27498 tablet_service.cc:1468] Processing CreateTablet for tablet 615f42e95bb24c8480f73d371fd63b9c (DEFAULT_TABLE table=default.Table_0d7367affcaf4dee8b0a8a1f524a7035 [id=27dadb1e246b45268355e924682cb3a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.577445 27356 tablet_service.cc:1468] Processing CreateTablet for tablet 6d4b8d4bf29e408cbdf9557c8b0d8a4b (DEFAULT_TABLE table=default.Table_a2fef2e8171c466e86ce74826ba6e281 [id=cbcfc7067aca4796b39e3eb195f38903]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.579756 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4739f10128d64e2c986257b1d9deb90f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.579953 27684 ts_tablet_manager.cc:1918] T f7ab722c37264973b204cd4f1f95052d P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:16.586416 27687 ts_tablet_manager.cc:1939] T 8945085bdc174e6db22a5a9477810297 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:16.577214 27693 consensus_queue.cc:260] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:16.594879 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d4b8d4bf29e408cbdf9557c8b0d8a4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.603638 27059 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 2a5c74a281b44019872a6442d41bc373 (table default.Table_5c08431ef8e04a6aab9b96ab78615d2f [id=a9ddfed19dab46a7bb4e049483e3996f]). Expected version 2 got 0
W20250903 20:05:16.605636 27053 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 53881ad30bc843839008ace4012a3f70 (table default.Table_6b2c13e18ac747b4bb616391101d313b [id=cedbbfab7d0e41369cafd9ce516f2dba], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 53881ad30bc843839008ace4012a3f70 consensus state has no leader
I20250903 20:05:16.609033 27491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b46bb34ac214b1ba23fba1a33f746ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.610527 27695 consensus_queue.cc:260] T 1fef9400227449ed892daff588999caa P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:16.611243 27695 ts_tablet_manager.cc:1428] T 1fef9400227449ed892daff588999caa P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.041s user 0.001s sys 0.001s
I20250903 20:05:16.611620 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68cff0e416a54a5eb37071f5a12f07ab" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:16.611232 27684 log.cc:1199] T f7ab722c37264973b204cd4f1f95052d P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/f7ab722c37264973b204cd4f1f95052d
I20250903 20:05:16.610939 27488 tablet_service.cc:1468] Processing CreateTablet for tablet 6d4b8d4bf29e408cbdf9557c8b0d8a4b (DEFAULT_TABLE table=default.Table_a2fef2e8171c466e86ce74826ba6e281 [id=cbcfc7067aca4796b39e3eb195f38903]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.610095 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 343) took 1327 ms. Trace:
I20250903 20:05:16.617908 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_1eee3233aac9463982bf7a5c2f3e098b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:16.618652 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1eee3233aac9463982bf7a5c2f3e098b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.618784 27489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bf570855b204b96acd070587c434463. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.619165 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_fa8b4f79747046748c7082603e9d2bfa" } new_table_name: "default.Table_c8d55d18c6bb4511ab10f35d2db6d2f9" modify_external_catalogs: true
I20250903 20:05:16.620446 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_3716b922d00f49ff9b1b7cef13737daf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:16.620998 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3716b922d00f49ff9b1b7cef13737daf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.622150 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ff2b25e34cab4943949d8b2d514bd98f" } new_table_name: "default.Table_6688c005609a44fcaf62e7da7d57df49" modify_external_catalogs: true
I20250903 20:05:16.623337 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_da1b078cb62f476c8f2c2454db2713d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:16.623848 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da1b078cb62f476c8f2c2454db2713d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.624285 27684 ts_tablet_manager.cc:1939] T f7ab722c37264973b204cd4f1f95052d P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:16.625351 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4b15d77d07c4fa7856d9c198cf15450. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.626760 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d4b8d4bf29e408cbdf9557c8b0d8a4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.630328 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e9a6a6e05464b7b97ae624f51402cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.630407 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_4eafb1a74bd54d2f89763f3c3cadd0f7" } new_table_name: "default.Table_c5eaadc5a45c45c780ebebbabf4717a0" modify_external_catalogs: true
I20250903 20:05:16.634687 27692 tablet_bootstrap.cc:654] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:16.634847 27362 tablet_service.cc:1468] Processing CreateTablet for tablet 54f1ce5192f547f595dea23a9eabf8e6 (DEFAULT_TABLE table=default.Table_ee23ae4ae2834af98c0a3bef517f0e60 [id=4195535fe553487084ba71bc439c1930]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.635284 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54f1ce5192f547f595dea23a9eabf8e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.629782 27687 rpcz_store.cc:276] 0903 20:05:15.260160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.260222 (+ 62us) service_pool.cc:225] Handling call
0903 20:05:16.040278 (+780056us) ts_tablet_manager.cc:1145] Deleting tablet 8945085bdc174e6db22a5a9477810297
0903 20:05:16.040289 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:16.351415 (+311126us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:16.503943 (+152528us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:16.587220 (+ 83277us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":636537}
I20250903 20:05:16.639741 27351 tablet_service.cc:1468] Processing CreateTablet for tablet 6800e0b6269045f29682704fd847d5d1 (DEFAULT_TABLE table=default.Table_6656b728e12a482fa9e7893ef4842ac7 [id=6788b524a8654e34a4d9ee8cc8fc84d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.640232 27351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6800e0b6269045f29682704fd847d5d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.644248 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_42f95b6c82f949a082559298f94e747c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:16.644886 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42f95b6c82f949a082559298f94e747c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.644814 27359 tablet_service.cc:1468] Processing CreateTablet for tablet 13496efe34944b7280e7a684f180e6e0 (DEFAULT_TABLE table=default.Table_a98f50198fb2489fad7fa669a4250ee7 [id=41f9673d6e604f45a8def10948ce21ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.645715 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 360) took 1132 ms. Trace:
I20250903 20:05:16.645833 27687 rpcz_store.cc:276] 0903 20:05:15.511589 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.511700 (+ 111us) service_pool.cc:225] Handling call
0903 20:05:16.637114 (+1125414us) ts_tablet_manager.cc:1145] Deleting tablet a0f79831866b40c49c5b27ce2014c3f6
0903 20:05:16.637126 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:16.643503 (+ 6377us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1114228}
I20250903 20:05:16.646343 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 361) took 1013 ms. Trace:
I20250903 20:05:16.646456 27687 rpcz_store.cc:276] 0903 20:05:15.632686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.635964 (+ 3278us) service_pool.cc:225] Handling call
0903 20:05:16.646156 (+1010192us) ts_tablet_manager.cc:1145] Deleting tablet 0a9b7770432942da9373d5100c71bd6a
0903 20:05:16.646165 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:16.646288 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1008440}
I20250903 20:05:16.646775 27687 tablet_replica.cc:331] T e20fcd28c3ae4355a033d2b8429f84e2 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
W20250903 20:05:16.647209 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet a0f79831866b40c49c5b27ce2014c3f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0f79831866b40c49c5b27ce2014c3f6 already in progress: creating tablet
W20250903 20:05:16.647521 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 0a9b7770432942da9373d5100c71bd6a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a9b7770432942da9373d5100c71bd6a already in progress: creating tablet
I20250903 20:05:16.648576 27692 tablet_bootstrap.cc:492] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:16.648933 27692 ts_tablet_manager.cc:1397] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.271s user 0.005s sys 0.011s
I20250903 20:05:16.651224 27692 raft_consensus.cc:357] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:16.651711 27692 raft_consensus.cc:383] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:16.651917 27692 raft_consensus.cc:738] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:16.652527 27692 consensus_queue.cc:260] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:16.653784 27692 ts_tablet_manager.cc:1428] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20250903 20:05:16.656651 27350 tablet_service.cc:1468] Processing CreateTablet for tablet f9375fff5f5945078c4412aa21c06f0d (DEFAULT_TABLE table=default.Table_98cd4cb4df914678b7bb584d97bd0ca9 [id=51d8d2e93e264e74948b83bbef771ef1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.663385 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13496efe34944b7280e7a684f180e6e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.664314 27496 tablet_service.cc:1468] Processing CreateTablet for tablet 13496efe34944b7280e7a684f180e6e0 (DEFAULT_TABLE table=default.Table_a98f50198fb2489fad7fa669a4250ee7 [id=41f9673d6e604f45a8def10948ce21ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.664773 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13496efe34944b7280e7a684f180e6e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.667549 27366 tablet_service.cc:1468] Processing CreateTablet for tablet 249362a0b8304719863bcb82c47273b3 (DEFAULT_TABLE table=default.Table_c750811a8d6c4069a3192ca531cf5712 [id=565dc96c38de4efeb4328c01e030eb8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.671361 27687 raft_consensus.cc:2241] T e20fcd28c3ae4355a033d2b8429f84e2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:16.671712 28100 raft_consensus.cc:491] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:16.671845 28100 raft_consensus.cc:513] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:16.673547 27497 tablet_service.cc:1468] Processing CreateTablet for tablet 54f1ce5192f547f595dea23a9eabf8e6 (DEFAULT_TABLE table=default.Table_ee23ae4ae2834af98c0a3bef517f0e60 [id=4195535fe553487084ba71bc439c1930]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.673571 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 6800e0b6269045f29682704fd847d5d1 (DEFAULT_TABLE table=default.Table_6656b728e12a482fa9e7893ef4842ac7 [id=6788b524a8654e34a4d9ee8cc8fc84d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.674021 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54f1ce5192f547f595dea23a9eabf8e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.674479 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9375fff5f5945078c4412aa21c06f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.674659 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6800e0b6269045f29682704fd847d5d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.676816 28100 leader_election.cc:290] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:16.676975 27687 raft_consensus.cc:2270] T e20fcd28c3ae4355a033d2b8429f84e2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:16.677729 27687 ts_tablet_manager.cc:1905] T e20fcd28c3ae4355a033d2b8429f84e2 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:16.678495 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 249362a0b8304719863bcb82c47273b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.681252 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_24fa04bd95714589b18eb17010dbcec2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:16.681545 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_91ae8b84b5b94998a81cf961b29bf4a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:16.681965 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24fa04bd95714589b18eb17010dbcec2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.682185 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_49ef7a892d00492d9e3b45f6dc7f2670" } new_table_name: "default.Table_3dee346162e54d5190c8e658e9483c9a" modify_external_catalogs: true
W20250903 20:05:16.683521 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91ae8b84b5b94998a81cf961b29bf4a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.684432 27358 tablet_service.cc:1468] Processing CreateTablet for tablet ea3f82f4e4244219ab0919f3e341aead (DEFAULT_TABLE table=default.Table_e1977f95b5174bb2b40623d225aff528 [id=e9bd7251cdb64e6bbbdc984babbbe88d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.684916 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea3f82f4e4244219ab0919f3e341aead. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.690028 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet e20fcd28c3ae4355a033d2b8429f84e2 (table default.Table_53cc54536434460ba6637a083afaddb2 [id=0e611bffbb74451d9e24eed4a553c2ca]) successfully deleted
W20250903 20:05:16.690505 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 1bad3c509460422396a17a3c4ae3d1e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1bad3c509460422396a17a3c4ae3d1e8 already in progress: creating tablet
W20250903 20:05:16.690953 27051 catalog_manager.cc:4895] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 4268316c28e8471a9f28807d20dbb395 because the tablet was not found. No further retry: Not found: Tablet not found: 4268316c28e8471a9f28807d20dbb395
I20250903 20:05:16.692266 28138 raft_consensus.cc:491] T 2a5c74a281b44019872a6442d41bc373 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:16.692334 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_d43fc55de38641f6946c7dab57874b91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:16.692466 28138 raft_consensus.cc:513] T 2a5c74a281b44019872a6442d41bc373 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:16.692838 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 0a9b7770432942da9373d5100c71bd6a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a9b7770432942da9373d5100c71bd6a already in progress: creating tablet
W20250903 20:05:16.693017 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d43fc55de38641f6946c7dab57874b91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.694705 27684 tablet_replica.cc:331] T 8023fa7537234ce783c2bf90b575b00c P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:16.693574 27490 tablet_service.cc:1468] Processing CreateTablet for tablet ed4902c002ef4198802375ced39ef2ad (DEFAULT_TABLE table=default.Table_ec03f537112c436ba9cce72cf2ff59a8 [id=a0ae37462e204c88876fa64fccbf77fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.695793 27363 tablet_service.cc:1468] Processing CreateTablet for tablet f4dec89fc6bb436ba6542c267eaf9c15 (DEFAULT_TABLE table=default.Table_8c565fdbc0e641bfaee34e0b42f855ee [id=b0e22c62f5b3455e8b180eb1ae0de718]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.696761 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4dec89fc6bb436ba6542c267eaf9c15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.697013 28138 leader_election.cc:290] T 2a5c74a281b44019872a6442d41bc373 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:16.707690 27684 raft_consensus.cc:2241] T 8023fa7537234ce783c2bf90b575b00c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:16.709144 27684 raft_consensus.cc:2270] T 8023fa7537234ce783c2bf90b575b00c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:16.709792 27684 ts_tablet_manager.cc:1905] T 8023fa7537234ce783c2bf90b575b00c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:16.728170 27504 consensus_meta.cc:287] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent flushing consensus metadata: real 0.514s user 0.006s sys 0.001s
W20250903 20:05:16.745508 27025 rpcz_store.cc:267] Call kudu.server.GenericService.CheckLeaks from 127.0.0.1:57206 (request call id 26) took 771 ms (client timeout 1000 ms). Trace:
W20250903 20:05:16.745757 27025 rpcz_store.cc:269] 0903 20:05:15.973624 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.973708 (+ 84us) service_pool.cc:225] Handling call
0903 20:05:16.745403 (+771695us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250903 20:05:16.752002 26885 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskH7eH49/build/asan/bin/kudu with pid 26961
I20250903 20:05:16.699585 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 7d5a3fbe93e44c228222ab1b1e78c187 (DEFAULT_TABLE table=default.Table_af52d46a98f249a7982d675a93661af0 [id=04c003dc02d74dce9ad96aa42ff9d6dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.706341 27352 tablet_service.cc:1515] Processing DeleteTablet for tablet 886b8783aa9d4618aad498d4a1115843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:16.736454 27365 tablet_service.cc:1468] Processing CreateTablet for tablet ed4902c002ef4198802375ced39ef2ad (DEFAULT_TABLE table=default.Table_ec03f537112c436ba9cce72cf2ff59a8 [id=a0ae37462e204c88876fa64fccbf77fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.756330 27357 tablet_service.cc:1468] Processing CreateTablet for tablet 9c77bc0e8bb34712bf8109cadbc0fc06 (DEFAULT_TABLE table=default.Table_119f28a77b72449f9374b78ff4b7e016 [id=07b954506fb7480f816756d30f35b4d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.756563 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 950a6fcd35f64f2d889831fe9cdf503a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:16.756680 27356 tablet_service.cc:1515] Processing DeleteTablet for tablet f326ec776ba446be8d5dda5c1dc5b6ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:16.758966 27348 tablet_service.cc:1515] Processing DeleteTablet for tablet f71d3dd9af2543299296f21381d1d563 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:16.760218 27354 tablet_service.cc:1468] Processing CreateTablet for tablet 60c32370a5274347afafc6a7e1e94270 (DEFAULT_TABLE table=default.Table_e9cb6a34300a4887b169da952eee37a5 [id=ed3fb47033d145de97ae00dfdf7f8243]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.768781 27684 ts_tablet_manager.cc:1918] T 8023fa7537234ce783c2bf90b575b00c P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:16.770805 27493 tablet_service.cc:1468] Processing CreateTablet for tablet 4739f10128d64e2c986257b1d9deb90f (DEFAULT_TABLE table=default.Table_cc45498ec439442797707c4fdd3692bc [id=45d7231b92f54017bd1f5eeb837fa166]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.771340 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4739f10128d64e2c986257b1d9deb90f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.770809 27501 tablet_service.cc:1468] Processing CreateTablet for tablet f9375fff5f5945078c4412aa21c06f0d (DEFAULT_TABLE table=default.Table_98cd4cb4df914678b7bb584d97bd0ca9 [id=51d8d2e93e264e74948b83bbef771ef1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.772970 27492 tablet_service.cc:1468] Processing CreateTablet for tablet 249362a0b8304719863bcb82c47273b3 (DEFAULT_TABLE table=default.Table_c750811a8d6c4069a3192ca531cf5712 [id=565dc96c38de4efeb4328c01e030eb8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.773324 27492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 249362a0b8304719863bcb82c47273b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.773841 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet 886b8783aa9d4618aad498d4a1115843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:16.774187 27495 tablet_service.cc:1468] Processing CreateTablet for tablet f4dec89fc6bb436ba6542c267eaf9c15 (DEFAULT_TABLE table=default.Table_8c565fdbc0e641bfaee34e0b42f855ee [id=b0e22c62f5b3455e8b180eb1ae0de718]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.774587 27494 tablet_service.cc:1468] Processing CreateTablet for tablet 7d5a3fbe93e44c228222ab1b1e78c187 (DEFAULT_TABLE table=default.Table_af52d46a98f249a7982d675a93661af0 [id=04c003dc02d74dce9ad96aa42ff9d6dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.775221 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 60c32370a5274347afafc6a7e1e94270 (DEFAULT_TABLE table=default.Table_e9cb6a34300a4887b169da952eee37a5 [id=ed3fb47033d145de97ae00dfdf7f8243]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.775848 27503 tablet_service.cc:1468] Processing CreateTablet for tablet 9c77bc0e8bb34712bf8109cadbc0fc06 (DEFAULT_TABLE table=default.Table_119f28a77b72449f9374b78ff4b7e016 [id=07b954506fb7480f816756d30f35b4d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.777463 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 351208baafc94f5eab531101d52e3b3f (DEFAULT_TABLE table=default.Table_f05d6db7f87643ebbb52602f99530905 [id=c393a6de2dbe4ccba47c6d0510a88e3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.775552 27501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9375fff5f5945078c4412aa21c06f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.779306 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet f326ec776ba446be8d5dda5c1dc5b6ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:16.779524 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c77bc0e8bb34712bf8109cadbc0fc06 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:16.779683 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet 950a6fcd35f64f2d889831fe9cdf503a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:16.779898 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet f71d3dd9af2543299296f21381d1d563 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:16.780030 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet 79a0553c912049cca5971b1ad6f7e11c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:16.780365 27502 tablet_service.cc:1468] Processing CreateTablet for tablet 5c42d84a6a5649208b0539a10da21aec (DEFAULT_TABLE table=default.Table_5187f549b3d44cce99c527429ce39a4b [id=bc5ade88feba4061a0f5ad0ffa347bad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.781565 27500 tablet_service.cc:1468] Processing CreateTablet for tablet 314856bf269f4df7aa25f5a30cec0d5b (DEFAULT_TABLE table=default.Table_fc325fc838f44caa818ddcddb486d85f [id=46b4934a16734ec3a7c1570b2d0cfb53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.782943 27491 tablet_service.cc:1468] Processing CreateTablet for tablet 659a127405d341c7b074d3445854256f (DEFAULT_TABLE table=default.Table_30721ff3b68e4c08bdb5845184d8ada7 [id=c807184f006b4eac8c34c84733b6a5fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.785800 27499 tablet_service.cc:1468] Processing CreateTablet for tablet a686ea49a29a4dd7b6e5dce5e1f9935a (DEFAULT_TABLE table=default.Table_00f2d88f1c284af3b6fbcaf8590938d5 [id=374fcca58f844f91a4561f61341feb3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.785063 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 69f8eef52bee431a86f0eef45fd11f9f (DEFAULT_TABLE table=default.Table_5215f44c06fc49d6a9a740f77e9c21fe [id=efecb95d0f474234959663b789c28b1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.787680 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a686ea49a29a4dd7b6e5dce5e1f9935a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.787933 27494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d5a3fbe93e44c228222ab1b1e78c187. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.788172 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4dec89fc6bb436ba6542c267eaf9c15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.788373 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60c32370a5274347afafc6a7e1e94270. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.788761 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c77bc0e8bb34712bf8109cadbc0fc06. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.793861 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c77bc0e8bb34712bf8109cadbc0fc06. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.795161 27353 tablet_service.cc:1468] Processing CreateTablet for tablet 314856bf269f4df7aa25f5a30cec0d5b (DEFAULT_TABLE table=default.Table_fc325fc838f44caa818ddcddb486d85f [id=46b4934a16734ec3a7c1570b2d0cfb53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.795384 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f8eef52bee431a86f0eef45fd11f9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.795990 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 351208baafc94f5eab531101d52e3b3f (DEFAULT_TABLE table=default.Table_f05d6db7f87643ebbb52602f99530905 [id=c393a6de2dbe4ccba47c6d0510a88e3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.796461 27489 tablet_service.cc:1468] Processing CreateTablet for tablet 6d13884fff8c43b690836c6a341a2d92 (DEFAULT_TABLE table=default.Table_2015884055c845e5a7bed9306978c4ec [id=cfe13c5e975047c08696b908602f91bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.796840 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 733f8024e1fa401ca9614a97db91a19b (DEFAULT_TABLE table=default.Table_4d90c70f5e544f839c056ae573d08f58 [id=46298581b9524c87bf3a0586f4c86c28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.799046 27488 tablet_service.cc:1468] Processing CreateTablet for tablet 5fb2426aa01e47598f268d0a90475271 (DEFAULT_TABLE table=default.Table_e4f1c7374cd94b529c3e923355498939 [id=a07de6053e474f988f800d6037e0f651]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.800248 27490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed4902c002ef4198802375ced39ef2ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.804901 27499 tablet_service.cc:1468] Processing CreateTablet for tablet 0ca6e2165a9d41299c1fe0bfa9475915 (DEFAULT_TABLE table=default.Table_dbc7009c4de44b8e88f3caa4b62ec665 [id=80977b1c4b154e158ce5a6e7e16e695d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.806270 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d5a3fbe93e44c228222ab1b1e78c187. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.806614 27354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60c32370a5274347afafc6a7e1e94270. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.795028 27347 tablet_service.cc:1468] Processing CreateTablet for tablet 5c42d84a6a5649208b0539a10da21aec (DEFAULT_TABLE table=default.Table_5187f549b3d44cce99c527429ce39a4b [id=bc5ade88feba4061a0f5ad0ffa347bad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.808382 27356 tablet_service.cc:1468] Processing CreateTablet for tablet 69f8eef52bee431a86f0eef45fd11f9f (DEFAULT_TABLE table=default.Table_5215f44c06fc49d6a9a740f77e9c21fe [id=efecb95d0f474234959663b789c28b1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.810765 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c77bc0e8bb34712bf8109cadbc0fc06 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:16.811494 27348 tablet_service.cc:1468] Processing CreateTablet for tablet 659a127405d341c7b074d3445854256f (DEFAULT_TABLE table=default.Table_30721ff3b68e4c08bdb5845184d8ada7 [id=c807184f006b4eac8c34c84733b6a5fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.812845 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c42d84a6a5649208b0539a10da21aec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.812415 27352 tablet_service.cc:1515] Processing DeleteTablet for tablet 4268316c28e8471a9f28807d20dbb395 with delete_type TABLET_DATA_DELETED (replaced by tablet a9f43781299e47a8aecc37a0b73d7a64) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:16.815232 28100 raft_consensus.cc:491] T 4268316c28e8471a9f28807d20dbb395 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:16.815635 27364 tablet_service.cc:1468] Processing CreateTablet for tablet 615f42e95bb24c8480f73d371fd63b9c (DEFAULT_TABLE table=default.Table_0d7367affcaf4dee8b0a8a1f524a7035 [id=27dadb1e246b45268355e924682cb3a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.817549 28100 raft_consensus.cc:513] T 4268316c28e8471a9f28807d20dbb395 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:16.818395 28100 leader_election.cc:290] T 4268316c28e8471a9f28807d20dbb395 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:16.819168 27349 tablet_service.cc:1468] Processing CreateTablet for tablet 6d13884fff8c43b690836c6a341a2d92 (DEFAULT_TABLE table=default.Table_2015884055c845e5a7bed9306978c4ec [id=cfe13c5e975047c08696b908602f91bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.820348 28136 raft_consensus.cc:2802] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250903 20:05:16.820487 27183 leader_election.cc:343] T fdc77810f27a40758e063b8b1150ab71 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:16.820808 27223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb2f50683e904dd39ee6eb128717d38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.822299 28136 raft_consensus.cc:491] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:16.822438 28136 raft_consensus.cc:3058] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:16.824301 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1d22e76ea72441db51f7e9db26eb520. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.824296 27695 tablet_bootstrap.cc:492] T deb84fc3465443d99c11097e53f5f575 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:16.826365 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2a5c74a281b44019872a6442d41bc373" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:16.826588 27248 raft_consensus.cc:2466] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
W20250903 20:05:16.827435 27321 leader_election.cc:343] T 4268316c28e8471a9f28807d20dbb395 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 4268316c28e8471a9f28807d20dbb395
W20250903 20:05:16.827306 27225 consensus_meta.cc:287] T 238f2f30377b42c2852b016018aa3245 P 89f8f0ee68b940db9ce250c0604f934a: Time spent flushing consensus metadata: real 0.554s user 0.007s sys 0.000s
I20250903 20:05:16.827711 27321 leader_election.cc:304] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:16.828047 27355 tablet_service.cc:1468] Processing CreateTablet for tablet a686ea49a29a4dd7b6e5dce5e1f9935a (DEFAULT_TABLE table=default.Table_00f2d88f1c284af3b6fbcaf8590938d5 [id=374fcca58f844f91a4561f61341feb3c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:16.828438 27183 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 245140us
I20250903 20:05:16.828755 27353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 314856bf269f4df7aa25f5a30cec0d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.829296 28135 raft_consensus.cc:491] T c9129a3e773a4fecb28ceee27778a294 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:16.829520 28135 raft_consensus.cc:513] T c9129a3e773a4fecb28ceee27778a294 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:16.830516 28135 leader_election.cc:290] T c9129a3e773a4fecb28ceee27778a294 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:16.830709 28135 raft_consensus.cc:491] T 5bc3b9771a0042fd87e706204dcbb712 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:16.830905 28135 raft_consensus.cc:513] T 5bc3b9771a0042fd87e706204dcbb712 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:16.831554 28135 leader_election.cc:290] T 5bc3b9771a0042fd87e706204dcbb712 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:16.834806 28100 raft_consensus.cc:2802] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:16.834960 28100 raft_consensus.cc:491] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:16.835580 28100 raft_consensus.cc:3058] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:16.835815 27352 tablet_service.cc:1468] Processing CreateTablet for tablet 733f8024e1fa401ca9614a97db91a19b (DEFAULT_TABLE table=default.Table_4d90c70f5e544f839c056ae573d08f58 [id=46298581b9524c87bf3a0586f4c86c28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.836737 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09fa32312c9648fbbd6fd327e9738066. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.837447 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ba2636641654d8596fd2a5b0644d073. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.850737 27693 ts_tablet_manager.cc:1428] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.450s user 0.000s sys 0.005s
I20250903 20:05:16.851123 27693 tablet_bootstrap.cc:492] T 79a0553c912049cca5971b1ad6f7e11c P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:16.852269 28100 raft_consensus.cc:513] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:16.856007 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cd92e3dafca6419f8eba7a4ef51a911e consensus state has no leader
I20250903 20:05:16.858184 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed4902c002ef4198802375ced39ef2ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.862092 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e17243085e464908adf84082adbb3800"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:16.862852 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e17243085e464908adf84082adbb3800 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.865175 28100 leader_election.cc:290] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:16.874905 27695 tablet_bootstrap.cc:654] T deb84fc3465443d99c11097e53f5f575 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:16.876078 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2a5c74a281b44019872a6442d41bc373" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:16.876346 27248 raft_consensus.cc:3058] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:16.805477 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 733f8024e1fa401ca9614a97db91a19b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.774181 27496 tablet_service.cc:1468] Processing CreateTablet for tablet ea3f82f4e4244219ab0919f3e341aead (DEFAULT_TABLE table=default.Table_e1977f95b5174bb2b40623d225aff528 [id=e9bd7251cdb64e6bbbdc984babbbe88d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.880704 27685 ts_tablet_manager.cc:1918] T 31342583c026438cbbae14106e75bbe5 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:16.885468 27366 tablet_service.cc:1468] Processing CreateTablet for tablet 5fb2426aa01e47598f268d0a90475271 (DEFAULT_TABLE table=default.Table_e4f1c7374cd94b529c3e923355498939 [id=a07de6053e474f988f800d6037e0f651]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.888737 27351 tablet_service.cc:1468] Processing CreateTablet for tablet 0ca6e2165a9d41299c1fe0bfa9475915 (DEFAULT_TABLE table=default.Table_dbc7009c4de44b8e88f3caa4b62ec665 [id=80977b1c4b154e158ce5a6e7e16e695d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.889925 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 733f8024e1fa401ca9614a97db91a19b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.890744 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet e0347eab6c8642beabdc2d7e7bbfdc9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:16.891626 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet ee40566e497740789575275d0cdca0db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:16.891848 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 60c32370a5274347afafc6a7e1e94270 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:16.892625 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 43fb27c857d543b7b78ea6108c406085 (DEFAULT_TABLE table=default.Table_4eafb1a74bd54d2f89763f3c3cadd0f7 [id=95e5cb3083814ab996fce9dc666c405e]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:16.892994 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ef02f91c0964446daf62ac1dde0330ff consensus state has no leader
I20250903 20:05:16.893869 27354 tablet_service.cc:1515] Processing DeleteTablet for tablet 5fb2426aa01e47598f268d0a90475271 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:16.894426 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 615f42e95bb24c8480f73d371fd63b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.894520 27684 log.cc:1199] T 8023fa7537234ce783c2bf90b575b00c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/8023fa7537234ce783c2bf90b575b00c
I20250903 20:05:16.894534 28136 raft_consensus.cc:513] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:16.896499 27221 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56534 (request call id 366) took 1004 ms. Trace:
I20250903 20:05:16.898626 27685 log.cc:1199] T 31342583c026438cbbae14106e75bbe5 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/31342583c026438cbbae14106e75bbe5
W20250903 20:05:16.900144 27059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 43fb27c857d543b7b78ea6108c406085 consensus state has no leader
I20250903 20:05:16.901611 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_08b8a5b3a94c4232b361faecf839c494"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:16.902395 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_08b8a5b3a94c4232b361faecf839c494 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.904232 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_46dac9447e4447f69f428c6c525b0bf2" } new_table_name: "default.Table_060249fde8eb4bd88887054d8691e260" modify_external_catalogs: true
I20250903 20:05:16.904350 27221 rpcz_store.cc:276] 0903 20:05:15.891573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.891822 (+ 249us) service_pool.cc:225] Handling call
0903 20:05:15.906942 (+ 15120us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0903 20:05:15.906965 (+ 23us) ts_tablet_manager.cc:676] Creating new metadata...
0903 20:05:16.416754 (+509789us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:16.896254 (+479500us) tablet_replica.cc:206] Creating consensus instance
0903 20:05:16.896447 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250903 20:05:16.904789 27354 tablet_service.cc:1468] Processing CreateTablet for tablet ef02f91c0964446daf62ac1dde0330ff (DEFAULT_TABLE table=default.Table_ff2b25e34cab4943949d8b2d514bd98f [id=b3135f352fb24246be537661e413991e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.884431 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b3121fce1c44d16a0b083313dc67b7a" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:16.905869 28136 leader_election.cc:290] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:16.910957 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet 33fb6b357bda43e7abeaed5634183ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:16.911273 27687 ts_tablet_manager.cc:1918] T e20fcd28c3ae4355a033d2b8429f84e2 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:16.911445 27687 log.cc:1199] T e20fcd28c3ae4355a033d2b8429f84e2 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/e20fcd28c3ae4355a033d2b8429f84e2
I20250903 20:05:16.911878 27687 ts_tablet_manager.cc:1939] T e20fcd28c3ae4355a033d2b8429f84e2 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:16.912802 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 363) took 1267 ms. Trace:
I20250903 20:05:16.912941 27687 rpcz_store.cc:276] 0903 20:05:15.645121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.645176 (+ 55us) service_pool.cc:225] Handling call
0903 20:05:16.646693 (+1001517us) ts_tablet_manager.cc:1145] Deleting tablet e20fcd28c3ae4355a033d2b8429f84e2
0903 20:05:16.646704 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:16.827172 (+180468us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:16.896756 (+ 69584us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:16.912759 (+ 16003us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21242,"tablet-delete.queue_time_us":1001255}
I20250903 20:05:16.913370 27687 tablet_replica.cc:331] T 925c0ab4873041da82702cd9377ec7bc P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:16.913542 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet e20fcd28c3ae4355a033d2b8429f84e2 (table default.Table_53cc54536434460ba6637a083afaddb2 [id=0e611bffbb74451d9e24eed4a553c2ca]) successfully deleted
I20250903 20:05:16.913599 27687 raft_consensus.cc:2241] T 925c0ab4873041da82702cd9377ec7bc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:16.913758 27687 raft_consensus.cc:2270] T 925c0ab4873041da82702cd9377ec7bc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:16.914605 27687 ts_tablet_manager.cc:1905] T 925c0ab4873041da82702cd9377ec7bc P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:16.914769 27184 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.829s user 0.010s sys 0.024s
I20250903 20:05:16.915206 27248 raft_consensus.cc:2466] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:16.915762 27321 leader_election.cc:304] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:16.916074 28100 raft_consensus.cc:2802] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:16.916220 28100 raft_consensus.cc:695] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:16.916496 28100 consensus_queue.cc:237] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:16.923662 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9129a3e773a4fecb28ceee27778a294" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:16.926398 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79a0553c912049cca5971b1ad6f7e11c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:16.931546 27685 ts_tablet_manager.cc:1939] T 31342583c026438cbbae14106e75bbe5 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:16.931989 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 322) took 2124 ms. Trace:
I20250903 20:05:16.932714 27685 rpcz_store.cc:276] 0903 20:05:14.807323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:14.807382 (+ 59us) service_pool.cc:225] Handling call
0903 20:05:15.561986 (+754604us) ts_tablet_manager.cc:1145] Deleting tablet 31342583c026438cbbae14106e75bbe5
0903 20:05:15.561994 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:15.740399 (+178405us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:16.838979 (+1098580us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:16.931947 (+ 92968us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":754390}
I20250903 20:05:16.933321 27685 tablet_replica.cc:331] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:16.933974 27685 raft_consensus.cc:2241] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:16.934487 27685 raft_consensus.cc:2270] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:16.935626 27685 ts_tablet_manager.cc:1905] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:16.940809 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d13884fff8c43b690836c6a341a2d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.941672 27491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 659a127405d341c7b074d3445854256f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.944144 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43fb27c857d543b7b78ea6108c406085. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.946588 27215 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56534 (request call id 370) took 1046 ms. Trace:
I20250903 20:05:16.948798 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 659a127405d341c7b074d3445854256f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.949410 27354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef02f91c0964446daf62ac1dde0330ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.950260 27215 rpcz_store.cc:276] 0903 20:05:15.900269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.900727 (+ 458us) service_pool.cc:225] Handling call
0903 20:05:15.925428 (+ 24701us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0903 20:05:15.925452 (+ 24us) ts_tablet_manager.cc:676] Creating new metadata...
0903 20:05:16.919133 (+993681us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:16.946345 (+ 27212us) tablet_replica.cc:206] Creating consensus instance
0903 20:05:16.946538 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250903 20:05:16.950867 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 351208baafc94f5eab531101d52e3b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.955171 27218 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56534 (request call id 369) took 1054 ms. Trace:
I20250903 20:05:16.955370 27355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a686ea49a29a4dd7b6e5dce5e1f9935a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.955387 27218 rpcz_store.cc:276] 0903 20:05:15.900145 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.900701 (+ 556us) service_pool.cc:225] Handling call
0903 20:05:15.905390 (+ 4689us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0903 20:05:15.905418 (+ 28us) ts_tablet_manager.cc:676] Creating new metadata...
0903 20:05:16.931024 (+1025606us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:16.953940 (+ 22916us) tablet_replica.cc:206] Creating consensus instance
0903 20:05:16.954298 (+ 358us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250903 20:05:16.956118 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fb2426aa01e47598f268d0a90475271. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.957569 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f8eef52bee431a86f0eef45fd11f9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.957782 27213 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56534 (request call id 373) took 1053 ms. Trace:
I20250903 20:05:16.957919 27213 rpcz_store.cc:276] 0903 20:05:15.903134 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.903533 (+ 399us) service_pool.cc:225] Handling call
0903 20:05:15.923819 (+ 20286us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0903 20:05:15.923860 (+ 41us) ts_tablet_manager.cc:676] Creating new metadata...
0903 20:05:16.924022 (+1000162us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:16.956829 (+ 32807us) tablet_replica.cc:206] Creating consensus instance
0903 20:05:16.956955 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250903 20:05:16.962385 27351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ca6e2165a9d41299c1fe0bfa9475915. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.964480 27209 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:56514 (request call id 96) took 1028 ms. Trace:
I20250903 20:05:16.964654 27209 rpcz_store.cc:276] 0903 20:05:15.934375 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.934446 (+ 71us) service_pool.cc:225] Handling call
0903 20:05:16.963008 (+1028562us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250903 20:05:16.968317 27184 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.868s user 0.010s sys 0.025s
I20250903 20:05:16.968567 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_196cfa99ce59433b85ad81b45d853081"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:16.969239 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_196cfa99ce59433b85ad81b45d853081 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:16.971318 27220 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56534 (request call id 372) took 1070 ms. Trace:
I20250903 20:05:16.971241 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ec03f537112c436ba9cce72cf2ff59a8" } modify_external_catalogs: true
I20250903 20:05:16.971482 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ec03f537112c436ba9cce72cf2ff59a8" } modify_external_catalogs: true
I20250903 20:05:16.972126 27059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_04bd610b6ae24395b30dc33c4d118258" } modify_external_catalogs: true
I20250903 20:05:16.972334 27059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_04bd610b6ae24395b30dc33c4d118258" } modify_external_catalogs: true
I20250903 20:05:16.975199 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e5def563dd414101b9ee8fbaf7156e29" } modify_external_catalogs: true
I20250903 20:05:16.975419 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e5def563dd414101b9ee8fbaf7156e29" } modify_external_catalogs: true
W20250903 20:05:16.983160 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1bf570855b204b96acd070587c434463 consensus state has no leader
I20250903 20:05:16.983618 27064 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d9105c35c28247f1a71866f2096d93f2" } modify_external_catalogs: true
I20250903 20:05:16.983834 27064 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d9105c35c28247f1a71866f2096d93f2" } modify_external_catalogs: true
I20250903 20:05:16.990746 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 4e34d6d5d68e4395aed78821ea077c1b (DEFAULT_TABLE table=default.Table_3a0bcaebe62e46e79afba5b118eeb855 [id=08f0b93efef74aa59b1ec3c286b3b45b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.991303 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e34d6d5d68e4395aed78821ea077c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.971493 27226 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56534 (request call id 371) took 1064 ms. Trace:
I20250903 20:05:16.975453 27207 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:56514 (request call id 92) took 1063 ms. Trace:
I20250903 20:05:16.992998 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_fdf0ad621407452eb74b6a9a56be2e85" } modify_external_catalogs: true
I20250903 20:05:16.993233 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_fdf0ad621407452eb74b6a9a56be2e85" } modify_external_catalogs: true
I20250903 20:05:16.994148 27226 rpcz_store.cc:276] 0903 20:05:15.900418 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.900696 (+ 278us) service_pool.cc:225] Handling call
0903 20:05:15.925729 (+ 25033us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0903 20:05:15.925768 (+ 39us) ts_tablet_manager.cc:676] Creating new metadata...
0903 20:05:16.907602 (+981834us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:16.964888 (+ 57286us) tablet_replica.cc:206] Creating consensus instance
0903 20:05:16.965042 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250903 20:05:16.994848 27221 tablet_service.cc:1468] Processing CreateTablet for tablet ee40566e497740789575275d0cdca0db (DEFAULT_TABLE table=default.Table_ee1ad081d0c846ca9fdc08acfb8318e9 [id=1794b6e2ac9443718c798588325d4a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:16.995290 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee40566e497740789575275d0cdca0db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:16.991480 27220 rpcz_store.cc:276] 0903 20:05:15.900528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.900749 (+ 221us) service_pool.cc:225] Handling call
0903 20:05:15.924391 (+ 23642us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0903 20:05:15.924425 (+ 34us) ts_tablet_manager.cc:676] Creating new metadata...
0903 20:05:16.967091 (+1042666us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:16.971151 (+ 4060us) tablet_replica.cc:206] Creating consensus instance
0903 20:05:16.971287 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250903 20:05:16.997218 27207 rpcz_store.cc:276] 0903 20:05:15.899453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.899514 (+ 61us) service_pool.cc:225] Handling call
0903 20:05:16.285613 (+386099us) spinlock_profiling.cc:243] Waited 1.41 ms on lock 0x6160001d1454. stack: 00007f589abe0dc6 00007f589abe0b80 00007f5899bc83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f58a6639700 00007f58aa3d371f 00007f58a4b04c36 00007f58a4b04965 00007f58a19910be 00007f58a198ff9e 00007f58a19961d5 00007f58a1996aff 00007f58a1996928 0000000000a774dc
0903 20:05:16.963126 (+677513us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3246336}
I20250903 20:05:16.980988 27685 ts_tablet_manager.cc:1918] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 20:05:16.982640 27213 tablet_service.cc:1468] Processing CreateTablet for tablet a3103e1595ec4c38ba65dca26fe1167c (DEFAULT_TABLE table=default.Table_fdf0ad621407452eb74b6a9a56be2e85 [id=e4270da20ece47aaa8897f47a02d86d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.001397 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3103e1595ec4c38ba65dca26fe1167c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:16.982275 27183 leader_election.cc:343] T 8b3121fce1c44d16a0b083313dc67b7a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:16.987267 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 47c1287bff1145a5b2ab6d14dbd608c7 (DEFAULT_TABLE table=default.Table_0cd65e00bd5a40bd9dc9db12fd0bda4d [id=15369700921549be8faa450d2ca98949]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.007429 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47c1287bff1145a5b2ab6d14dbd608c7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:17.008476 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 13496efe34944b7280e7a684f180e6e0 consensus state has no leader
I20250903 20:05:17.010344 27685 log.cc:1199] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/fd499b81344c4b58bf77f3622422763c
I20250903 20:05:17.009102 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet 68cff0e416a54a5eb37071f5a12f07ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:17.012951 27692 tablet_bootstrap.cc:492] T 691cb69759654fdc9223844049a06d02 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:17.014878 27527 raft_consensus.cc:3058] T 79a0553c912049cca5971b1ad6f7e11c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:17.015355 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fb2426aa01e47598f268d0a90475271. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.018249 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet e0347eab6c8642beabdc2d7e7bbfdc9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:17.018601 27684 ts_tablet_manager.cc:1939] T 8023fa7537234ce783c2bf90b575b00c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:17.019629 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 364) took 1262 ms. Trace:
I20250903 20:05:17.021229 27684 rpcz_store.cc:276] 0903 20:05:15.756770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.758333 (+ 1563us) service_pool.cc:225] Handling call
0903 20:05:16.681421 (+923088us) ts_tablet_manager.cc:1145] Deleting tablet 8023fa7537234ce783c2bf90b575b00c
0903 20:05:16.681428 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:16.756331 (+ 74903us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:16.768483 (+ 12152us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:17.019566 (+251083us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6729,"tablet-delete.queue_time_us":918288}
I20250903 20:05:17.022732 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 367) took 1265 ms. Trace:
I20250903 20:05:17.023949 27685 ts_tablet_manager.cc:1939] T fd499b81344c4b58bf77f3622422763c P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
W20250903 20:05:17.034339 27183 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 96510.1us
I20250903 20:05:17.034988 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 323) took 1989 ms. Trace:
I20250903 20:05:17.035105 27685 rpcz_store.cc:276] 0903 20:05:15.034975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.045508 (+ 10533us) service_pool.cc:225] Handling call
0903 20:05:16.933120 (+1887612us) ts_tablet_manager.cc:1145] Deleting tablet fd499b81344c4b58bf77f3622422763c
0903 20:05:16.933265 (+ 145us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:16.977590 (+ 44325us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:16.980857 (+ 3267us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:17.024402 (+ 43545us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1887280}
I20250903 20:05:17.035605 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 324) took 1922 ms. Trace:
I20250903 20:05:17.035709 27685 rpcz_store.cc:276] 0903 20:05:15.113555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.113615 (+ 60us) service_pool.cc:225] Handling call
0903 20:05:17.035405 (+1921790us) ts_tablet_manager.cc:1145] Deleting tablet 22bc87d1a4804b6c80f77f9692500d63
0903 20:05:17.035415 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.035538 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1921560}
I20250903 20:05:17.036098 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 325) took 1919 ms. Trace:
I20250903 20:05:17.036201 27685 rpcz_store.cc:276] 0903 20:05:15.116303 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.116360 (+ 57us) service_pool.cc:225] Handling call
0903 20:05:17.035933 (+1919573us) ts_tablet_manager.cc:1145] Deleting tablet 4c18a98d5c0f4a43ae91c9130047b681
0903 20:05:17.035942 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.036036 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1919376}
I20250903 20:05:17.036554 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 326) took 1879 ms. Trace:
I20250903 20:05:17.036634 27685 rpcz_store.cc:276] 0903 20:05:15.156618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.156669 (+ 51us) service_pool.cc:225] Handling call
0903 20:05:17.036405 (+1879736us) ts_tablet_manager.cc:1145] Deleting tablet 57a1167457b84ea8a493be232b1d088a
0903 20:05:17.036413 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.036500 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1879454}
I20250903 20:05:17.036978 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 327) took 1842 ms. Trace:
I20250903 20:05:17.037072 27685 rpcz_store.cc:276] 0903 20:05:15.194857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.194912 (+ 55us) service_pool.cc:225] Handling call
0903 20:05:17.036828 (+1841916us) ts_tablet_manager.cc:1145] Deleting tablet 4c18a98d5c0f4a43ae91c9130047b681
0903 20:05:17.036837 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.036928 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1841681}
I20250903 20:05:17.037348 27685 tablet_replica.cc:331] T 262a3549867a473aaaa7dfcc0b5872cd P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:17.037554 27685 raft_consensus.cc:2241] T 262a3549867a473aaaa7dfcc0b5872cd P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:17.037770 27685 raft_consensus.cc:2270] T 262a3549867a473aaaa7dfcc0b5872cd P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:17.042433 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 8a223f75045345ae833f52bcc8f239f9
I20250903 20:05:17.043944 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 6f8cfe53b91b41d7ba7f9074e1005af3
I20250903 20:05:17.044404 27685 ts_tablet_manager.cc:1905] T 262a3549867a473aaaa7dfcc0b5872cd P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:17.045176 27358 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f8cfe53b91b41d7ba7f9074e1005af3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:17.045680 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet a3103e1595ec4c38ba65dca26fe1167c
I20250903 20:05:17.046890 27358 tablet_service.cc:1515] Processing DeleteTablet for tablet a3103e1595ec4c38ba65dca26fe1167c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:17.047308 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet ed4902c002ef4198802375ced39ef2ad
I20250903 20:05:17.048484 27358 tablet_service.cc:1515] Processing DeleteTablet for tablet ed4902c002ef4198802375ced39ef2ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:17.049424 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet 47c1287bff1145a5b2ab6d14dbd608c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:17.051146 27492 tablet_service.cc:1515] Processing DeleteTablet for tablet 5fb2426aa01e47598f268d0a90475271 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:17.051074 27183 leader_election.cc:343] T c9129a3e773a4fecb28ceee27778a294 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:17.051483 27492 tablet_service.cc:1515] Processing DeleteTablet for tablet 33fb6b357bda43e7abeaed5634183ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:17.051700 27492 tablet_service.cc:1515] Processing DeleteTablet for tablet 60c32370a5274347afafc6a7e1e94270 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:17.052193 27492 tablet_service.cc:1468] Processing CreateTablet for tablet 43fb27c857d543b7b78ea6108c406085 (DEFAULT_TABLE table=default.Table_4eafb1a74bd54d2f89763f3c3cadd0f7 [id=95e5cb3083814ab996fce9dc666c405e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.054352 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea3f82f4e4244219ab0919f3e341aead. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.054767 27498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 615f42e95bb24c8480f73d371fd63b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.055320 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet 238f2f30377b42c2852b016018aa3245 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:17.055933 27354 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a223f75045345ae833f52bcc8f239f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:17.055159 27684 rpcz_store.cc:276] 0903 20:05:15.757125 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.758700 (+ 1575us) service_pool.cc:225] Handling call
0903 20:05:17.022535 (+1263835us) ts_tablet_manager.cc:1145] Deleting tablet ed096b452ba747f89e91edc15e6e3ac1
0903 20:05:17.022553 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.022685 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1259350}
I20250903 20:05:17.060959 27695 tablet_bootstrap.cc:492] T deb84fc3465443d99c11097e53f5f575 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:17.061128 27695 ts_tablet_manager.cc:1397] T deb84fc3465443d99c11097e53f5f575 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.448s user 0.015s sys 0.013s
I20250903 20:05:17.062029 27695 raft_consensus.cc:357] T deb84fc3465443d99c11097e53f5f575 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:17.062273 27695 raft_consensus.cc:383] T deb84fc3465443d99c11097e53f5f575 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:17.062373 27695 raft_consensus.cc:738] T deb84fc3465443d99c11097e53f5f575 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:17.062595 27695 consensus_queue.cc:260] T deb84fc3465443d99c11097e53f5f575 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:17.063599 27695 ts_tablet_manager.cc:1428] T deb84fc3465443d99c11097e53f5f575 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20250903 20:05:17.064246 27695 tablet_bootstrap.cc:492] T ecd339ec12e84e88acec427630cc8668 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:17.067467 27695 tablet_bootstrap.cc:654] T ecd339ec12e84e88acec427630cc8668 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:17.067893 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 314856bf269f4df7aa25f5a30cec0d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.069658 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet e78e6d212daf4589a6b4b65711e356cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:17.069721 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet e877d56bbbfa4c5a92d025d0311ce5c6
I20250903 20:05:17.070446 27213 tablet_service.cc:1468] Processing CreateTablet for tablet e2441aa3c2c748bba5360259fb663c81 (DEFAULT_TABLE table=default.Table_64dabe496cb94cfb9061d0d7eefcbd3a [id=741df86b1206422b9467c06cab14ddce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.070835 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2441aa3c2c748bba5360259fb663c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.073868 27354 tablet_service.cc:1515] Processing DeleteTablet for tablet e877d56bbbfa4c5a92d025d0311ce5c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:17.077581 27221 tablet_service.cc:1515] Processing DeleteTablet for tablet 120ca61b3da742dc9a2efa483ead7e97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:17.078282 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bc3b9771a0042fd87e706204dcbb712" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:17.078613 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet ee40566e497740789575275d0cdca0db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:17.082794 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 368) took 1318 ms. Trace:
I20250903 20:05:17.087738 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 7824a0bccdc9485abde124572e502791 (DEFAULT_TABLE table=default.Table_9a599b6ff41449a6ba1b8239b7472262 [id=c1afbb6949b9423399ab4fdc2bd49239]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.088567 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7824a0bccdc9485abde124572e502791. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.094697 27221 tablet_service.cc:1468] Processing CreateTablet for tablet de20d18d7f264811a0631aa1144aef2c (DEFAULT_TABLE table=default.Table_ea59a50e05934c84a8cf4e8f198d7093 [id=1fdfd927e5654aad860585fc23ce5f8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.095222 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de20d18d7f264811a0631aa1144aef2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.095341 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet 31342583c026438cbbae14106e75bbe5 with delete_type TABLET_DATA_DELETED (replaced by tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:16.997061 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 16549a6d9aa9432eb5cee19f69ce6957 (DEFAULT_TABLE table=default.Table_1eee3233aac9463982bf7a5c2f3e098b [id=2dfc9b9647ff4939b96f6759f3bd162b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.096570 27693 tablet_bootstrap.cc:700] T 79a0553c912049cca5971b1ad6f7e11c P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent opening tablet: real 0.107s user 0.000s sys 0.000s
I20250903 20:05:17.096946 27693 tablet_bootstrap.cc:654] T 79a0553c912049cca5971b1ad6f7e11c P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:17.098258 27220 tablet_service.cc:1468] Processing CreateTablet for tablet 0fef9aa7153a4902abce522e3ff0df3f (DEFAULT_TABLE table=default.Table_4ba98389160f4a7c802792a41f6bd00e [id=f8544f28af82478aa3762f84b040a6d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.098762 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fef9aa7153a4902abce522e3ff0df3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.098990 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_2cedbabdccbb444789b77ff293edf019"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:17.099695 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2cedbabdccbb444789b77ff293edf019 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:17.100934 27226 tablet_service.cc:1468] Processing CreateTablet for tablet 64c4dd2645034a7eaf243f5474603f10 (DEFAULT_TABLE table=default.Table_d9878f04c3404885815ae53a9054de14 [id=a98d20608a054ad29a951bb47cf61711]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.103196 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64c4dd2645034a7eaf243f5474603f10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.106739 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 3f303233e6c347289650c6cf83f8127f (DEFAULT_TABLE table=default.Table_58569618f9d04fdfbd005b3d0e15c049 [id=1cbfdd865e51404688e08bc60eec2197]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.107336 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f303233e6c347289650c6cf83f8127f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.110601 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16549a6d9aa9432eb5cee19f69ce6957. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:17.111914 27051 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.26.65.125:44059 (error 108) [suppressed 9 similar messages]
W20250903 20:05:17.095149 27462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 497983us
I20250903 20:05:17.116389 27684 rpcz_store.cc:276] 0903 20:05:15.757230 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.758867 (+ 1637us) service_pool.cc:225] Handling call
0903 20:05:17.057747 (+1298880us) ts_tablet_manager.cc:1145] Deleting tablet e9eb24dd5a6243c49a3389320a3f78a0
0903 20:05:17.057759 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.076082 (+ 18323us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1294017}
I20250903 20:05:17.120679 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ca6e2165a9d41299c1fe0bfa9475915. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.121397 27501 tablet_service.cc:1468] Processing CreateTablet for tablet ef02f91c0964446daf62ac1dde0330ff (DEFAULT_TABLE table=default.Table_ff2b25e34cab4943949d8b2d514bd98f [id=b3135f352fb24246be537661e413991e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.124573 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 16549a6d9aa9432eb5cee19f69ce6957 (DEFAULT_TABLE table=default.Table_1eee3233aac9463982bf7a5c2f3e098b [id=2dfc9b9647ff4939b96f6759f3bd162b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.130237 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a223f75045345ae833f52bcc8f239f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:17.134196 27320 leader_election.cc:343] T f7ab722c37264973b204cd4f1f95052d P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: Tablet not RUNNING: STOPPED
I20250903 20:05:17.134413 27320 leader_election.cc:304] T f7ab722c37264973b204cd4f1f95052d P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:17.134474 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet a3103e1595ec4c38ba65dca26fe1167c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:17.139125 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f8cfe53b91b41d7ba7f9074e1005af3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:17.145021 27320 leader_election.cc:304] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:17.149639 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet ed4902c002ef4198802375ced39ef2ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:17.149937 28100 raft_consensus.cc:2747] T f7ab722c37264973b204cd4f1f95052d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:17.152673 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet e877d56bbbfa4c5a92d025d0311ce5c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:17.153110 28132 raft_consensus.cc:2802] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:17.155707 28132 raft_consensus.cc:491] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:17.155887 28132 raft_consensus.cc:3058] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:17.161721 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet ed096b452ba747f89e91edc15e6e3ac1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed096b452ba747f89e91edc15e6e3ac1 already in progress: creating tablet
W20250903 20:05:17.170998 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet e9eb24dd5a6243c49a3389320a3f78a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9eb24dd5a6243c49a3389320a3f78a0 already in progress: creating tablet
I20250903 20:05:17.177382 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2a5c74a281b44019872a6442d41bc373" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:17.177798 27064 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 2a5c74a281b44019872a6442d41bc373 (table default.Table_5c08431ef8e04a6aab9b96ab78615d2f [id=a9ddfed19dab46a7bb4e049483e3996f]). Expected version 2 got 0
I20250903 20:05:17.184791 27353 tablet_service.cc:1468] Processing CreateTablet for tablet 8400e23873fd4be18de2998afc64a689 (DEFAULT_TABLE table=default.Table_d43fc55de38641f6946c7dab57874b91 [id=6103a53de91b4ad19b8f7afd7b3fd881]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.188426 27356 tablet_service.cc:1468] Processing CreateTablet for tablet 8e12f4c420874ab382d433982bddbf05 (DEFAULT_TABLE table=default.Table_42f95b6c82f949a082559298f94e747c [id=430a979602e1430eaab642507035c4f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.188360 27353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8400e23873fd4be18de2998afc64a689. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.190872 27493 tablet_service.cc:1468] Processing CreateTablet for tablet 8400e23873fd4be18de2998afc64a689 (DEFAULT_TABLE table=default.Table_d43fc55de38641f6946c7dab57874b91 [id=6103a53de91b4ad19b8f7afd7b3fd881]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.193022 27497 tablet_service.cc:1468] Processing CreateTablet for tablet 8e12f4c420874ab382d433982bddbf05 (DEFAULT_TABLE table=default.Table_42f95b6c82f949a082559298f94e747c [id=430a979602e1430eaab642507035c4f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.197935 27687 ts_tablet_manager.cc:1918] T 925c0ab4873041da82702cd9377ec7bc P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:17.198251 27349 tablet_service.cc:1468] Processing CreateTablet for tablet 8218868ff40b405387d560f614427c08 (DEFAULT_TABLE table=default.Table_91ae8b84b5b94998a81cf961b29bf4a4 [id=782821cd80cf493381e55730ebb5fefe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.198607 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e12f4c420874ab382d433982bddbf05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.200307 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_36c6bab093f446f78fe92cf5173f841e" } new_table_name: "default.Table_e79b97db68cd4a7d827d56a72a6d15e9" modify_external_catalogs: true
I20250903 20:05:17.203123 27687 log.cc:1199] T 925c0ab4873041da82702cd9377ec7bc P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/925c0ab4873041da82702cd9377ec7bc
I20250903 20:05:17.204702 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 36e1061915574407b6e7695be8b1e55a (DEFAULT_TABLE table=default.Table_3716b922d00f49ff9b1b7cef13737daf [id=836e087e0536485492f7ee6784fabe76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.205634 27687 ts_tablet_manager.cc:1939] T 925c0ab4873041da82702cd9377ec7bc P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:17.207070 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8218868ff40b405387d560f614427c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.208230 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e1061915574407b6e7695be8b1e55a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:17.210327 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3919f7f95a954f59b68f56d0051c0109 consensus state has no leader
I20250903 20:05:17.212204 27490 tablet_service.cc:1468] Processing CreateTablet for tablet 8218868ff40b405387d560f614427c08 (DEFAULT_TABLE table=default.Table_91ae8b84b5b94998a81cf961b29bf4a4 [id=782821cd80cf493381e55730ebb5fefe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.213572 27693 tablet_bootstrap.cc:492] T 79a0553c912049cca5971b1ad6f7e11c P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:17.213764 27693 ts_tablet_manager.cc:1397] T 79a0553c912049cca5971b1ad6f7e11c P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.363s user 0.002s sys 0.008s
I20250903 20:05:17.214847 27693 raft_consensus.cc:357] T 79a0553c912049cca5971b1ad6f7e11c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:17.215078 27693 raft_consensus.cc:383] T 79a0553c912049cca5971b1ad6f7e11c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:17.215205 27693 raft_consensus.cc:738] T 79a0553c912049cca5971b1ad6f7e11c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:17.215432 27361 tablet_service.cc:1468] Processing CreateTablet for tablet bc98762790f441038972366b1cc2c55b (DEFAULT_TABLE table=default.Table_24fa04bd95714589b18eb17010dbcec2 [id=bb35d1946b5b44afa5329d15a3866162]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:17.214107 27051 consensus_peers.cc:489] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 -> Peer 35dd69f3fa7a4fd8843b97f8a5d68296 (127.26.65.125:44059): Couldn't send request to peer 35dd69f3fa7a4fd8843b97f8a5d68296. Status: Network error: Client connection negotiation failed: client connection to 127.26.65.125:44059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:17.215878 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc98762790f441038972366b1cc2c55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.216969 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 362) took 1561 ms. Trace:
I20250903 20:05:17.217111 27687 rpcz_store.cc:276] 0903 20:05:15.644932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.645723 (+ 791us) service_pool.cc:225] Handling call
0903 20:05:16.913285 (+1267562us) ts_tablet_manager.cc:1145] Deleting tablet 925c0ab4873041da82702cd9377ec7bc
0903 20:05:16.913298 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:16.961573 (+ 48275us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:17.197246 (+235673us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:17.206529 (+ 9283us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1267296}
I20250903 20:05:17.217654 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 364) took 1530 ms. Trace:
I20250903 20:05:17.217757 27687 rpcz_store.cc:276] 0903 20:05:15.686709 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.686771 (+ 62us) service_pool.cc:225] Handling call
0903 20:05:17.217439 (+1530668us) ts_tablet_manager.cc:1145] Deleting tablet 1bad3c509460422396a17a3c4ae3d1e8
0903 20:05:17.217451 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.217594 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1530425}
I20250903 20:05:17.218103 27687 tablet_replica.cc:331] T cfa8a1d9011e460681443a6b77b95958 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:17.218350 27687 raft_consensus.cc:2241] T cfa8a1d9011e460681443a6b77b95958 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:17.218555 27687 raft_consensus.cc:2270] T cfa8a1d9011e460681443a6b77b95958 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:17.219197 27687 ts_tablet_manager.cc:1905] T cfa8a1d9011e460681443a6b77b95958 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:17.219339 27495 tablet_service.cc:1468] Processing CreateTablet for tablet 36e1061915574407b6e7695be8b1e55a (DEFAULT_TABLE table=default.Table_3716b922d00f49ff9b1b7cef13737daf [id=836e087e0536485492f7ee6784fabe76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.221889 27352 tablet_service.cc:1468] Processing CreateTablet for tablet f23f90aac8bc49889b1b882df186ef50 (DEFAULT_TABLE table=default.Table_da1b078cb62f476c8f2c2454db2713d2 [id=26058718e4174971ae0d8d74a6963893]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.222762 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 925c0ab4873041da82702cd9377ec7bc (table default.Table_1a5c8c3256c24c7f8ef8597efa53dde5 [id=7caef592f80a4d6cb6df4e2c3971de49]) successfully deleted
W20250903 20:05:17.223126 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 1bad3c509460422396a17a3c4ae3d1e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1bad3c509460422396a17a3c4ae3d1e8 already in progress: creating tablet
I20250903 20:05:17.224536 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f23f90aac8bc49889b1b882df186ef50. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.248098 28132 raft_consensus.cc:513] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:17.249459 28132 leader_election.cc:290] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:17.250607 27064 catalog_manager.cc:5582] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66). New cstate: current_term: 1 leader_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:17.251204 27064 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 2a5c74a281b44019872a6442d41bc373 (table default.Table_5c08431ef8e04a6aab9b96ab78615d2f [id=a9ddfed19dab46a7bb4e049483e3996f]). Expected version 2 got 0
I20250903 20:05:17.283530 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_359258cdcd734ef4b3463c80fa1c582a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:17.284293 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_359258cdcd734ef4b3463c80fa1c582a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:17.286659 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f02e6857b868499087599a49fe189d58" } new_table_name: "default.Table_50f6b8731a2c4ccb8bf58597acb5b760" modify_external_catalogs: true
I20250903 20:05:17.289179 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ca85cb69be1e4e709c000e5b77bcd3cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:17.289356 27693 consensus_queue.cc:260] T 79a0553c912049cca5971b1ad6f7e11c P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:17.289877 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca85cb69be1e4e709c000e5b77bcd3cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:17.290102 27693 ts_tablet_manager.cc:1428] T 79a0553c912049cca5971b1ad6f7e11c P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.076s user 0.002s sys 0.000s
I20250903 20:05:17.296084 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c42d84a6a5649208b0539a10da21aec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.296778 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 369) took 1364 ms. Trace:
I20250903 20:05:17.306695 27692 tablet_bootstrap.cc:654] T 691cb69759654fdc9223844049a06d02 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:17.307686 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e1061915574407b6e7695be8b1e55a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.310900 27057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_fdaf2e3d1322481bb38fbb5298ba40fc" } new_table_name: "default.Table_0253a17daef94a8fb2be91d233de40cd" modify_external_catalogs: true
I20250903 20:05:17.315196 27490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8218868ff40b405387d560f614427c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.317029 27693 tablet_bootstrap.cc:492] T 31342583c026438cbbae14106e75bbe5 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:17.324759 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 351208baafc94f5eab531101d52e3b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.326293 27684 rpcz_store.cc:276] 0903 20:05:15.757341 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.758925 (+ 1584us) service_pool.cc:225] Handling call
0903 20:05:17.120356 (+1361431us) ts_tablet_manager.cc:1145] Deleting tablet 1fbe54dc40dd47daa109ceeb07546b0a
0903 20:05:17.120375 (+ 19us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.121868 (+ 1493us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1352105}
I20250903 20:05:17.331640 27489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d13884fff8c43b690836c6a341a2d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.332774 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16549a6d9aa9432eb5cee19f69ce6957. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.332798 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 372) took 1575 ms. Trace:
I20250903 20:05:17.341244 27501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef02f91c0964446daf62ac1dde0330ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.341745 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8400e23873fd4be18de2998afc64a689. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.342396 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e12f4c420874ab382d433982bddbf05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.339174 27693 tablet_bootstrap.cc:654] T 31342583c026438cbbae14106e75bbe5 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:17.348212 27687 ts_tablet_manager.cc:1918] T cfa8a1d9011e460681443a6b77b95958 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:17.348385 27687 log.cc:1199] T cfa8a1d9011e460681443a6b77b95958 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/cfa8a1d9011e460681443a6b77b95958
I20250903 20:05:17.348764 27687 ts_tablet_manager.cc:1939] T cfa8a1d9011e460681443a6b77b95958 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:17.349603 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 365) took 1653 ms. Trace:
I20250903 20:05:17.349738 27687 rpcz_store.cc:276] 0903 20:05:15.695827 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.695886 (+ 59us) service_pool.cc:225] Handling call
0903 20:05:17.218004 (+1522118us) ts_tablet_manager.cc:1145] Deleting tablet cfa8a1d9011e460681443a6b77b95958
0903 20:05:17.218014 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.332480 (+114466us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:17.348068 (+ 15588us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:17.349558 (+ 1490us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1521883}
I20250903 20:05:17.350459 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet cfa8a1d9011e460681443a6b77b95958 (table default.Table_6d201f66980a4a7ab430058673f8d513 [id=058a64daaf104dfdace7d05d52388b79]) successfully deleted
I20250903 20:05:17.352694 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 367) took 1655 ms. Trace:
I20250903 20:05:17.352836 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 30cc0d4ba8d444f1b98232f906ea0c7b (DEFAULT_TABLE table=default.Table_91e306fcee12497e98a9d2da6302ba2b [id=57668609851c4124b73a3797bb6d2eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.352932 27687 rpcz_store.cc:276] 0903 20:05:15.697660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.698634 (+ 974us) service_pool.cc:225] Handling call
0903 20:05:17.350021 (+1651387us) ts_tablet_manager.cc:1145] Deleting tablet e9eb24dd5a6243c49a3389320a3f78a0
0903 20:05:17.350031 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.352642 (+ 2611us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1651156}
I20250903 20:05:17.353406 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30cc0d4ba8d444f1b98232f906ea0c7b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:17.353416 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet e9eb24dd5a6243c49a3389320a3f78a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9eb24dd5a6243c49a3389320a3f78a0 already in progress: creating tablet
W20250903 20:05:17.356628 27321 leader_election.cc:343] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 8023fa7537234ce783c2bf90b575b00c
I20250903 20:05:17.357479 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_2c769d77af8d429fabe07bdd9ab6c61c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:17.358129 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c769d77af8d429fabe07bdd9ab6c61c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:17.353821 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 368) took 1656 ms. Trace:
I20250903 20:05:17.358796 27687 rpcz_store.cc:276] 0903 20:05:15.697766 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.698903 (+ 1137us) service_pool.cc:225] Handling call
0903 20:05:17.353671 (+1654768us) ts_tablet_manager.cc:1145] Deleting tablet 1fbe54dc40dd47daa109ceeb07546b0a
0903 20:05:17.353680 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.353776 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1654583}
I20250903 20:05:17.358083 27213 tablet_service.cc:1468] Processing CreateTablet for tablet 39c2ebe4ada5426590ec3769c8662644 (DEFAULT_TABLE table=default.Table_d84f067d5a924a23964078d360422878 [id=5cb314181f104d63b1aed98c7198d5c3]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:17.359347 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 1fbe54dc40dd47daa109ceeb07546b0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fbe54dc40dd47daa109ceeb07546b0a already in progress: creating tablet
I20250903 20:05:17.359843 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 366) took 1662 ms. Trace:
I20250903 20:05:17.360042 27687 rpcz_store.cc:276] 0903 20:05:15.697563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.699106 (+ 1543us) service_pool.cc:225] Handling call
0903 20:05:17.359622 (+1660516us) ts_tablet_manager.cc:1145] Deleting tablet ed096b452ba747f89e91edc15e6e3ac1
0903 20:05:17.359633 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.359807 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1660325}
I20250903 20:05:17.359764 27491 tablet_service.cc:1468] Processing CreateTablet for tablet f23f90aac8bc49889b1b882df186ef50 (DEFAULT_TABLE table=default.Table_da1b078cb62f476c8f2c2454db2713d2 [id=26058718e4174971ae0d8d74a6963893]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.360347 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 259bc73ca9df47e9874692725a60bb1c (DEFAULT_TABLE table=default.Table_5e9a58e5be084c1fa563357bdab3d530 [id=60cc385fc1044966b4a12e13fdc7b5bb]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:17.360620 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet ed096b452ba747f89e91edc15e6e3ac1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed096b452ba747f89e91edc15e6e3ac1 already in progress: creating tablet
I20250903 20:05:17.361361 27687 tablet_replica.cc:331] T f7ab722c37264973b204cd4f1f95052d P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:17.361658 27687 raft_consensus.cc:2241] T f7ab722c37264973b204cd4f1f95052d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:17.363747 27687 raft_consensus.cc:2270] T f7ab722c37264973b204cd4f1f95052d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:17.364238 28030 raft_consensus.cc:491] T 1fef9400227449ed892daff588999caa P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:17.364425 27687 ts_tablet_manager.cc:1905] T f7ab722c37264973b204cd4f1f95052d P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:17.364394 28030 raft_consensus.cc:513] T 1fef9400227449ed892daff588999caa P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:17.364665 27488 tablet_service.cc:1468] Processing CreateTablet for tablet bc98762790f441038972366b1cc2c55b (DEFAULT_TABLE table=default.Table_24fa04bd95714589b18eb17010dbcec2 [id=bb35d1946b5b44afa5329d15a3866162]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.366478 28030 leader_election.cc:290] T 1fef9400227449ed892daff588999caa P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:17.367062 27684 rpcz_store.cc:276] 0903 20:05:15.757692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.759144 (+ 1452us) service_pool.cc:225] Handling call
0903 20:05:17.326671 (+1567527us) ts_tablet_manager.cc:1145] Deleting tablet cb4d99139516405bb65c7480ddf259f1
0903 20:05:17.330694 (+ 4023us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.332745 (+ 2051us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1559295}
I20250903 20:05:17.367579 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39c2ebe4ada5426590ec3769c8662644. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.369220 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 370) took 1611 ms. Trace:
I20250903 20:05:17.369807 27527 raft_consensus.cc:2466] T 79a0553c912049cca5971b1ad6f7e11c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:17.370540 27684 rpcz_store.cc:276] 0903 20:05:15.757453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.758980 (+ 1527us) service_pool.cc:225] Handling call
0903 20:05:17.367366 (+1608386us) ts_tablet_manager.cc:1145] Deleting tablet fbfabb6ff1714d2e921f45256b543759
0903 20:05:17.369059 (+ 1693us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.369179 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1599483}
I20250903 20:05:17.371979 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 371) took 1614 ms. Trace:
I20250903 20:05:17.372519 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 259bc73ca9df47e9874692725a60bb1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.373219 27492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43fb27c857d543b7b78ea6108c406085. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.374282 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 3919f7f95a954f59b68f56d0051c0109 (DEFAULT_TABLE table=default.Table_36c6bab093f446f78fe92cf5173f841e [id=147416d6b2ea4002953902e413c5a57e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.374699 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3919f7f95a954f59b68f56d0051c0109. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.379771 27687 ts_tablet_manager.cc:1918] T f7ab722c37264973b204cd4f1f95052d P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:17.379992 27687 log.cc:1199] T f7ab722c37264973b204cd4f1f95052d P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/f7ab722c37264973b204cd4f1f95052d
I20250903 20:05:17.380443 27687 ts_tablet_manager.cc:1939] T f7ab722c37264973b204cd4f1f95052d P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:17.379359 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 4e9a6a6e05464b7b97ae624f51402cf0 (DEFAULT_TABLE table=default.Table_f02e6857b868499087599a49fe189d58 [id=4eecee99ccf149d49f2f95c27b5b05ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.381978 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 369) took 1674 ms. Trace:
I20250903 20:05:17.383158 27687 rpcz_store.cc:276] 0903 20:05:15.707589 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.707835 (+ 246us) service_pool.cc:225] Handling call
0903 20:05:17.361248 (+1653413us) ts_tablet_manager.cc:1145] Deleting tablet f7ab722c37264973b204cd4f1f95052d
0903 20:05:17.361257 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.370204 (+ 8947us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:17.379618 (+ 9414us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:17.381917 (+ 2299us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1653157}
I20250903 20:05:17.384747 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 370) took 1676 ms. Trace:
I20250903 20:05:17.384925 27687 rpcz_store.cc:276] 0903 20:05:15.707781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.708056 (+ 275us) service_pool.cc:225] Handling call
0903 20:05:17.384180 (+1676124us) ts_tablet_manager.cc:1145] Deleting tablet fbfabb6ff1714d2e921f45256b543759
0903 20:05:17.384190 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.384317 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1675903}
W20250903 20:05:17.385483 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet fbfabb6ff1714d2e921f45256b543759 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbfabb6ff1714d2e921f45256b543759 already in progress: creating tablet
I20250903 20:05:17.388532 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e9a6a6e05464b7b97ae624f51402cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.389516 27693 tablet_bootstrap.cc:492] T 31342583c026438cbbae14106e75bbe5 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:17.389556 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2a5c74a281b44019872a6442d41bc373" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:17.389856 27693 ts_tablet_manager.cc:1397] T 31342583c026438cbbae14106e75bbe5 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.073s user 0.005s sys 0.000s
I20250903 20:05:17.389971 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 371) took 1668 ms. Trace:
I20250903 20:05:17.391031 27693 raft_consensus.cc:357] T 31342583c026438cbbae14106e75bbe5 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:17.391297 27693 raft_consensus.cc:383] T 31342583c026438cbbae14106e75bbe5 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:17.391399 27693 raft_consensus.cc:738] T 31342583c026438cbbae14106e75bbe5 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:17.391712 27693 consensus_queue.cc:260] T 31342583c026438cbbae14106e75bbe5 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:17.391901 27224 tablet_service.cc:1468] Processing CreateTablet for tablet 2385d3006f90486890ebe43e16a53d7a (DEFAULT_TABLE table=default.Table_478904b0f29b45aaaa87750ddf9ec041 [id=05afe1b4c02b403ab3402946b9b13c40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.390825 27687 rpcz_store.cc:276] 0903 20:05:15.720651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.720732 (+ 81us) service_pool.cc:225] Handling call
0903 20:05:17.386117 (+1665385us) ts_tablet_manager.cc:1145] Deleting tablet 5002b7e8118c49b6a247cb8dc4ca1aea
0903 20:05:17.386127 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.389487 (+ 3360us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1665132}
I20250903 20:05:17.392259 27693 ts_tablet_manager.cc:1428] T 31342583c026438cbbae14106e75bbe5 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:17.392213 27229 tablet_service.cc:1468] Processing CreateTablet for tablet d4c703dafbca4e669dbb18946ffac699 (DEFAULT_TABLE table=default.Table_bc654adac8a24afab120af90f0a73f36 [id=e6c1c8665bd841b5a932b2d54f9be4e1]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:17.392843 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 5002b7e8118c49b6a247cb8dc4ca1aea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5002b7e8118c49b6a247cb8dc4ca1aea already in progress: creating tablet
I20250903 20:05:17.393177 27216 tablet_service.cc:1468] Processing CreateTablet for tablet 5b46bb34ac214b1ba23fba1a33f746ac (DEFAULT_TABLE table=default.Table_03ce0e151fcc4259ba52504c9c5ddfcd [id=f261e84232234244824d0232efbb4399]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.393628 27219 tablet_service.cc:1468] Processing CreateTablet for tablet f4d3e1cfdfab4be1a9143b78bf3bc46d (DEFAULT_TABLE table=default.Table_016e6db78b02447b8777dba09ae06057 [id=386b8d4e7c4042d1947c19ac464c75a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.393915 27693 tablet_bootstrap.cc:492] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:17.395313 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2385d3006f90486890ebe43e16a53d7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.396240 27693 tablet_bootstrap.cc:654] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:17.398324 27219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4d3e1cfdfab4be1a9143b78bf3bc46d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.399760 27684 rpcz_store.cc:276] 0903 20:05:15.757571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.759061 (+ 1490us) service_pool.cc:225] Handling call
0903 20:05:17.370815 (+1611754us) ts_tablet_manager.cc:1145] Deleting tablet 5002b7e8118c49b6a247cb8dc4ca1aea
0903 20:05:17.370825 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.371938 (+ 1113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1601892}
I20250903 20:05:17.404480 27491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f23f90aac8bc49889b1b882df186ef50. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.405009 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc98762790f441038972366b1cc2c55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.406188 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1fef9400227449ed892daff588999caa" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:17.407238 27684 tablet_replica.cc:331] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:17.408016 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68cff0e416a54a5eb37071f5a12f07ab" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:17.407385 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 372) took 1685 ms. Trace:
I20250903 20:05:17.411728 27684 raft_consensus.cc:2241] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:17.411952 27684 pending_rounds.cc:70] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f: Trying to abort 1 pending ops.
I20250903 20:05:17.412465 27684 pending_rounds.cc:77] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384934223458304 op_type: NO_OP noop_request { }
I20250903 20:05:17.412606 27684 raft_consensus.cc:2887] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:17.413237 27684 raft_consensus.cc:2270] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:17.416440 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2a5c74a281b44019872a6442d41bc373" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:17.417083 27522 raft_consensus.cc:2466] T 2a5c74a281b44019872a6442d41bc373 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:17.419902 27684 ts_tablet_manager.cc:1905] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:17.409022 27687 rpcz_store.cc:276] 0903 20:05:15.721409 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.721470 (+ 61us) service_pool.cc:225] Handling call
0903 20:05:17.392974 (+1671504us) ts_tablet_manager.cc:1145] Deleting tablet cb4d99139516405bb65c7480ddf259f1
0903 20:05:17.406119 (+ 13145us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.406982 (+ 863us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1671173}
W20250903 20:05:17.423799 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 1fbe54dc40dd47daa109ceeb07546b0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fbe54dc40dd47daa109ceeb07546b0a already in progress: creating tablet
I20250903 20:05:17.426192 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2a5c74a281b44019872a6442d41bc373" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
W20250903 20:05:17.426807 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet cb4d99139516405bb65c7480ddf259f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb4d99139516405bb65c7480ddf259f1 already in progress: creating tablet
I20250903 20:05:17.433768 27527 raft_consensus.cc:3058] T 2a5c74a281b44019872a6442d41bc373 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:17.433702 27952 consensus_queue.cc:1035] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20250903 20:05:17.424263 27693 tablet_bootstrap.cc:492] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:17.436072 27693 ts_tablet_manager.cc:1397] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.042s user 0.006s sys 0.000s
I20250903 20:05:17.440697 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 79a0553c912049cca5971b1ad6f7e11c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:17.441949 27320 leader_election.cc:304] T 68cff0e416a54a5eb37071f5a12f07ab P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:17.442438 28132 raft_consensus.cc:2747] T 68cff0e416a54a5eb37071f5a12f07ab P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:17.444062 27320 leader_election.cc:343] T 4268316c28e8471a9f28807d20dbb395 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 4268316c28e8471a9f28807d20dbb395
I20250903 20:05:17.444201 27320 leader_election.cc:304] T 4268316c28e8471a9f28807d20dbb395 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:17.444602 28132 raft_consensus.cc:2747] T 4268316c28e8471a9f28807d20dbb395 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:17.445497 27320 leader_election.cc:343] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 8023fa7537234ce783c2bf90b575b00c
I20250903 20:05:17.445642 27320 leader_election.cc:304] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:17.446069 28132 raft_consensus.cc:2747] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20250903 20:05:17.448355 27462 consensus_peers.cc:489] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:17.452507 27693 raft_consensus.cc:357] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:17.463907 27057 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b4b15d77d07c4fa7856d9c198cf15450 consensus state has no leader
W20250903 20:05:17.464474 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4e9a6a6e05464b7b97ae624f51402cf0 consensus state has no leader
I20250903 20:05:17.452726 27693 raft_consensus.cc:383] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:17.465883 27693 raft_consensus.cc:738] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:17.469502 27693 consensus_queue.cc:260] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:17.470203 27693 ts_tablet_manager.cc:1428] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.034s user 0.001s sys 0.000s
I20250903 20:05:17.470655 27693 tablet_bootstrap.cc:492] T 262a3549867a473aaaa7dfcc0b5872cd P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:17.472173 28126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "35bdb0090d8747059b5ee103e786e979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:17.472393 28126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [sys.catalog]: This master's current role is: LEADER
I20250903 20:05:17.472530 27693 tablet_bootstrap.cc:654] T 262a3549867a473aaaa7dfcc0b5872cd P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:17.473510 27692 tablet_bootstrap.cc:492] T 691cb69759654fdc9223844049a06d02 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:17.473703 27692 ts_tablet_manager.cc:1397] T 691cb69759654fdc9223844049a06d02 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.564s user 0.009s sys 0.007s
I20250903 20:05:17.474345 27687 tablet_replica.cc:331] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:17.474730 27687 raft_consensus.cc:2241] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:17.474961 27687 raft_consensus.cc:2270] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:17.475555 27687 ts_tablet_manager.cc:1905] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:17.463066 27183 leader_election.cc:343] T 5bc3b9771a0042fd87e706204dcbb712 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:17.478046 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet cb4d99139516405bb65c7480ddf259f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb4d99139516405bb65c7480ddf259f1 already in progress: creating tablet
W20250903 20:05:17.478413 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet fbfabb6ff1714d2e921f45256b543759 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbfabb6ff1714d2e921f45256b543759 already in progress: creating tablet
I20250903 20:05:17.479254 27692 raft_consensus.cc:357] T 691cb69759654fdc9223844049a06d02 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:17.479460 27692 raft_consensus.cc:383] T 691cb69759654fdc9223844049a06d02 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:17.479545 27692 raft_consensus.cc:738] T 691cb69759654fdc9223844049a06d02 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:17.479758 27692 consensus_queue.cc:260] T 691cb69759654fdc9223844049a06d02 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:17.480580 27692 ts_tablet_manager.cc:1428] T 691cb69759654fdc9223844049a06d02 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250903 20:05:17.478705 27527 raft_consensus.cc:2466] T 2a5c74a281b44019872a6442d41bc373 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:17.480347 27695 tablet_bootstrap.cc:492] T ecd339ec12e84e88acec427630cc8668 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:17.482295 27249 raft_consensus.cc:2391] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
W20250903 20:05:17.484408 27183 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 92120us
I20250903 20:05:17.484457 27228 tablet_service.cc:1468] Processing CreateTablet for tablet 1bf570855b204b96acd070587c434463 (DEFAULT_TABLE table=default.Table_49ef7a892d00492d9e3b45f6dc7f2670 [id=5ff9845a7d804adeaaee6199ac7faee6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.486711 27222 tablet_service.cc:1468] Processing CreateTablet for tablet 17f16e9c8f794e03a2f8b3b6b35f8e84 (DEFAULT_TABLE table=default.Table_ed795c89b83a43c68ace2f6652900ca1 [id=d7a41e621fde497d8fcfb842c1e6f9e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.489254 27229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4c703dafbca4e669dbb18946ffac699. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.489678 27227 tablet_service.cc:1468] Processing CreateTablet for tablet b4b15d77d07c4fa7856d9c198cf15450 (DEFAULT_TABLE table=default.Table_fdaf2e3d1322481bb38fbb5298ba40fc [id=45cd523acd3c45b994d1b9cc6eb569ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.490824 27216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b46bb34ac214b1ba23fba1a33f746ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.490765 27212 tablet_service.cc:1468] Processing CreateTablet for tablet 6d4b8d4bf29e408cbdf9557c8b0d8a4b (DEFAULT_TABLE table=default.Table_a2fef2e8171c466e86ce74826ba6e281 [id=cbcfc7067aca4796b39e3eb195f38903]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.491575 27692 tablet_bootstrap.cc:492] T 73f65e7515994b898eec2ea23903aff4 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:17.494913 27692 tablet_bootstrap.cc:654] T 73f65e7515994b898eec2ea23903aff4 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:17.495075 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4b15d77d07c4fa7856d9c198cf15450. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:17.496754 27462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 45665.8us
I20250903 20:05:17.503449 27684 ts_tablet_manager.cc:1918] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:17.503698 27684 log.cc:1199] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/262a3549867a473aaaa7dfcc0b5872cd
I20250903 20:05:17.503134 27462 leader_election.cc:304] T 2a5c74a281b44019872a6442d41bc373 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:17.493242 27210 tablet_service.cc:1468] Processing CreateTablet for tablet 13496efe34944b7280e7a684f180e6e0 (DEFAULT_TABLE table=default.Table_a98f50198fb2489fad7fa669a4250ee7 [id=41f9673d6e604f45a8def10948ce21ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.504627 27684 ts_tablet_manager.cc:1939] T 262a3549867a473aaaa7dfcc0b5872cd P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:17.491765 27211 tablet_service.cc:1468] Processing CreateTablet for tablet 615f42e95bb24c8480f73d371fd63b9c (DEFAULT_TABLE table=default.Table_0d7367affcaf4dee8b0a8a1f524a7035 [id=27dadb1e246b45268355e924682cb3a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.507786 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 361) took 1751 ms. Trace:
I20250903 20:05:17.508122 27228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bf570855b204b96acd070587c434463. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.508244 27952 raft_consensus.cc:2747] T 2a5c74a281b44019872a6442d41bc373 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:17.508688 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 5002b7e8118c49b6a247cb8dc4ca1aea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5002b7e8118c49b6a247cb8dc4ca1aea already in progress: creating tablet
I20250903 20:05:17.507928 27684 rpcz_store.cc:276] 0903 20:05:15.756382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.761629 (+ 5247us) service_pool.cc:225] Handling call
0903 20:05:17.401679 (+1640050us) ts_tablet_manager.cc:1145] Deleting tablet 262a3549867a473aaaa7dfcc0b5872cd
0903 20:05:17.401691 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.496546 (+ 94855us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:17.503295 (+ 6749us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:17.507724 (+ 4429us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1632538}
I20250903 20:05:17.511862 27684 tablet_replica.cc:331] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:17.512612 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17f16e9c8f794e03a2f8b3b6b35f8e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.518568 27212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d4b8d4bf29e408cbdf9557c8b0d8a4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.519619 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_34c69a470a8a4e9eab936b4c856e4eb4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:17.520370 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34c69a470a8a4e9eab936b4c856e4eb4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:17.521307 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b613060864734552b4f204a8844f7ee8" } new_table_name: "default.Table_7b75f7edb8c84d6c85a4cbc3faac617d" modify_external_catalogs: true
I20250903 20:05:17.522136 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_93c4817ac1c64d589e71a4590d32adb2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:17.522820 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93c4817ac1c64d589e71a4590d32adb2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:17.524578 27183 leader_election.cc:304] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
W20250903 20:05:17.524947 27183 leader_election.cc:343] T 1fef9400227449ed892daff588999caa P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:17.526391 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_91ed95244dda4d37985c6af207e9e033"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:17.527052 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91ed95244dda4d37985c6af207e9e033 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:17.528076 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_d4c00bc20db74e4c84da0f99bd594b19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:17.528738 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4c00bc20db74e4c84da0f99bd594b19 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:17.529528 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_4a511c041bed4d8b88479f001d13c791"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:17.530110 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a511c041bed4d8b88479f001d13c791 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:17.530920 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_d37cdb880e224494969e67d03305b7fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:17.531443 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d37cdb880e224494969e67d03305b7fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:17.532507 27226 tablet_service.cc:1468] Processing CreateTablet for tablet 54f1ce5192f547f595dea23a9eabf8e6 (DEFAULT_TABLE table=default.Table_ee23ae4ae2834af98c0a3bef517f0e60 [id=4195535fe553487084ba71bc439c1930]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.533701 27695 ts_tablet_manager.cc:1397] T ecd339ec12e84e88acec427630cc8668 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.418s user 0.021s sys 0.063s
I20250903 20:05:17.535363 27210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13496efe34944b7280e7a684f180e6e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.540253 27687 ts_tablet_manager.cc:1918] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:17.540788 27687 log.cc:1199] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/8023fa7537234ce783c2bf90b575b00c
I20250903 20:05:17.540988 27693 tablet_bootstrap.cc:492] T 262a3549867a473aaaa7dfcc0b5872cd P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:17.541496 27693 ts_tablet_manager.cc:1397] T 262a3549867a473aaaa7dfcc0b5872cd P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.071s user 0.008s sys 0.000s
I20250903 20:05:17.542658 27687 ts_tablet_manager.cc:1939] T 8023fa7537234ce783c2bf90b575b00c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:17.545187 27693 raft_consensus.cc:357] T 262a3549867a473aaaa7dfcc0b5872cd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:17.546343 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 373) took 1806 ms. Trace:
I20250903 20:05:17.547009 27693 raft_consensus.cc:383] T 262a3549867a473aaaa7dfcc0b5872cd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:17.549405 27685 ts_tablet_manager.cc:1918] T 262a3549867a473aaaa7dfcc0b5872cd P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:17.549866 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3a0bcaebe62e46e79afba5b118eeb855" } new_table_name: "default.Table_3aecc9b035a24287af94e9c31d5d0714" modify_external_catalogs: true
I20250903 20:05:17.551649 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5c08431ef8e04a6aab9b96ab78615d2f" } modify_external_catalogs: true
I20250903 20:05:17.551904 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5c08431ef8e04a6aab9b96ab78615d2f" } modify_external_catalogs: true
I20250903 20:05:17.547595 27693 raft_consensus.cc:738] T 262a3549867a473aaaa7dfcc0b5872cd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:17.547219 27687 rpcz_store.cc:276] 0903 20:05:15.739809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.739881 (+ 72us) service_pool.cc:225] Handling call
0903 20:05:17.470392 (+1730511us) ts_tablet_manager.cc:1145] Deleting tablet 8023fa7537234ce783c2bf90b575b00c
0903 20:05:17.474250 (+ 3858us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.497776 (+ 23526us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:17.500982 (+ 3206us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:17.545947 (+ 44965us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1730185}
I20250903 20:05:17.553622 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2015884055c845e5a7bed9306978c4ec" } modify_external_catalogs: true
I20250903 20:05:17.553884 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2015884055c845e5a7bed9306978c4ec" } modify_external_catalogs: true
I20250903 20:05:17.559175 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 377) took 1732 ms. Trace:
I20250903 20:05:17.558604 27693 consensus_queue.cc:260] T 262a3549867a473aaaa7dfcc0b5872cd P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:17.561012 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24eadc70e10d4d02bcb643d45c9e0e36 consensus state has no leader
I20250903 20:05:17.561309 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 262a3549867a473aaaa7dfcc0b5872cd (table default.Table_14af7742726b4562a02fd2c9b2895277 [id=6d72901e5102476890f466fd406a7127]) successfully deleted
I20250903 20:05:17.561569 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_876161e56d1d4c4c9d6726471530d965" } modify_external_catalogs: true
I20250903 20:05:17.561791 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_876161e56d1d4c4c9d6726471530d965" } modify_external_catalogs: true
I20250903 20:05:17.560447 27687 rpcz_store.cc:276] 0903 20:05:15.826081 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.826163 (+ 82us) service_pool.cc:225] Handling call
0903 20:05:17.557471 (+1731308us) ts_tablet_manager.cc:1145] Deleting tablet a4f3230f50ec4c95b528fead8c191224
0903 20:05:17.557669 (+ 198us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.558759 (+ 1090us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1727778}
I20250903 20:05:17.566834 27685 log.cc:1199] T 262a3549867a473aaaa7dfcc0b5872cd P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/262a3549867a473aaaa7dfcc0b5872cd
I20250903 20:05:17.567101 27223 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56534 (request call id 375) took 1602 ms. Trace:
I20250903 20:05:17.567346 27217 tablet_service.cc:1468] Processing CreateTablet for tablet ed4902c002ef4198802375ced39ef2ad (DEFAULT_TABLE table=default.Table_ec03f537112c436ba9cce72cf2ff59a8 [id=a0ae37462e204c88876fa64fccbf77fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.568400 27695 raft_consensus.cc:357] T ecd339ec12e84e88acec427630cc8668 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:17.568455 27687 tablet_replica.cc:331] T 68cff0e416a54a5eb37071f5a12f07ab P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:17.569010 27693 ts_tablet_manager.cc:1428] T 262a3549867a473aaaa7dfcc0b5872cd P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.027s user 0.004s sys 0.000s
W20250903 20:05:17.569569 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet a4f3230f50ec4c95b528fead8c191224 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4f3230f50ec4c95b528fead8c191224 already in progress: creating tablet
I20250903 20:05:17.569953 27684 raft_consensus.cc:2241] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:17.570688 27684 pending_rounds.cc:70] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f: Trying to abort 1 pending ops.
I20250903 20:05:17.570784 27684 pending_rounds.cc:77] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384933810667520 op_type: NO_OP noop_request { }
I20250903 20:05:17.570888 27684 raft_consensus.cc:2887] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:17.570974 27211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 615f42e95bb24c8480f73d371fd63b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.571007 27684 raft_consensus.cc:2270] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:17.571609 27684 ts_tablet_manager.cc:1905] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:17.566428 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 6800e0b6269045f29682704fd847d5d1 (DEFAULT_TABLE table=default.Table_6656b728e12a482fa9e7893ef4842ac7 [id=6788b524a8654e34a4d9ee8cc8fc84d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.567236 27223 rpcz_store.cc:276] 0903 20:05:15.903411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.903840 (+ 429us) service_pool.cc:225] Handling call
0903 20:05:15.925121 (+ 21281us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0903 20:05:15.925149 (+ 28us) ts_tablet_manager.cc:676] Creating new metadata...
0903 20:05:17.359937 (+1434788us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:17.505617 (+145680us) tablet_replica.cc:206] Creating consensus instance
0903 20:05:17.505795 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250903 20:05:17.575575 28135 raft_consensus.cc:2802] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:17.588415 27952 raft_consensus.cc:491] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:17.588673 27952 raft_consensus.cc:513] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:17.589359 27952 leader_election.cc:290] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:17.590695 27695 raft_consensus.cc:383] T ecd339ec12e84e88acec427630cc8668 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250903 20:05:17.594069 27462 leader_election.cc:343] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 31342583c026438cbbae14106e75bbe5
I20250903 20:05:17.594274 27695 raft_consensus.cc:738] T ecd339ec12e84e88acec427630cc8668 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:17.594332 27223 tablet_service.cc:1468] Processing CreateTablet for tablet f9375fff5f5945078c4412aa21c06f0d (DEFAULT_TABLE table=default.Table_98cd4cb4df914678b7bb584d97bd0ca9 [id=51d8d2e93e264e74948b83bbef771ef1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.594880 28135 raft_consensus.cc:695] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:17.595793 27695 consensus_queue.cc:260] T ecd339ec12e84e88acec427630cc8668 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:17.596385 27695 ts_tablet_manager.cc:1428] T ecd339ec12e84e88acec427630cc8668 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.062s user 0.001s sys 0.001s
I20250903 20:05:17.597627 27684 ts_tablet_manager.cc:1918] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:17.597746 27684 log.cc:1199] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/04ccb3e4a8bd4789b9f19edd4a5b6145
I20250903 20:05:17.598160 27684 ts_tablet_manager.cc:1939] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:17.600112 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54f1ce5192f547f595dea23a9eabf8e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.601204 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 373) took 1816 ms. Trace:
I20250903 20:05:17.601330 27684 rpcz_store.cc:276] 0903 20:05:15.789414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.790147 (+ 733us) service_pool.cc:225] Handling call
0903 20:05:17.511495 (+1721348us) ts_tablet_manager.cc:1145] Deleting tablet 04ccb3e4a8bd4789b9f19edd4a5b6145
0903 20:05:17.511506 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.594529 (+ 83023us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:17.597495 (+ 2966us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:17.601158 (+ 3663us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1716504}
I20250903 20:05:17.601996 27687 raft_consensus.cc:2241] T 68cff0e416a54a5eb37071f5a12f07ab P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:17.602241 27687 raft_consensus.cc:2270] T 68cff0e416a54a5eb37071f5a12f07ab P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:17.603011 27685 ts_tablet_manager.cc:1939] T 262a3549867a473aaaa7dfcc0b5872cd P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:17.603055 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 376) took 1770 ms. Trace:
I20250903 20:05:17.603168 27684 rpcz_store.cc:276] 0903 20:05:15.835600 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.835661 (+ 61us) service_pool.cc:225] Handling call
0903 20:05:17.602892 (+1767231us) ts_tablet_manager.cc:1145] Deleting tablet a4f3230f50ec4c95b528fead8c191224
0903 20:05:17.602902 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.603013 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1766958}
I20250903 20:05:17.603653 27684 tablet_replica.cc:331] T 68cff0e416a54a5eb37071f5a12f07ab P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:17.603852 27684 raft_consensus.cc:2241] T 68cff0e416a54a5eb37071f5a12f07ab P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:17.604017 27684 raft_consensus.cc:2270] T 68cff0e416a54a5eb37071f5a12f07ab P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:17.605160 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31342583c026438cbbae14106e75bbe5" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:17.605398 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 04ccb3e4a8bd4789b9f19edd4a5b6145 (table default.Table_edfc268f076142a2b7c2ef149edf7f5c [id=b287b55c1542426fb4fd0372ef6bcd05]) successfully deleted
I20250903 20:05:17.605417 27386 raft_consensus.cc:2466] T 31342583c026438cbbae14106e75bbe5 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:17.605916 27462 leader_election.cc:304] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:17.606199 27952 raft_consensus.cc:2802] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:17.606305 27952 raft_consensus.cc:491] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:17.606418 27952 raft_consensus.cc:3058] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:17.606825 27684 ts_tablet_manager.cc:1905] T 68cff0e416a54a5eb37071f5a12f07ab P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:17.607661 27052 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 24eadc70e10d4d02bcb643d45c9e0e36 (table default.Table_7b75f7edb8c84d6c85a4cbc3faac617d [id=e9a291cff3f4497cad9393ab4c571bf7], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 24eadc70e10d4d02bcb643d45c9e0e36 consensus state has no leader
I20250903 20:05:17.609341 27952 raft_consensus.cc:513] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:17.609962 27952 leader_election.cc:290] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:17.605607 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet a4f3230f50ec4c95b528fead8c191224 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4f3230f50ec4c95b528fead8c191224 already in progress: creating tablet
I20250903 20:05:17.612407 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6800e0b6269045f29682704fd847d5d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.613297 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed4902c002ef4198802375ced39ef2ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.614933 27684 ts_tablet_manager.cc:1918] T 68cff0e416a54a5eb37071f5a12f07ab P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:17.615103 27684 log.cc:1199] T 68cff0e416a54a5eb37071f5a12f07ab P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/68cff0e416a54a5eb37071f5a12f07ab
I20250903 20:05:17.615625 27684 ts_tablet_manager.cc:1939] T 68cff0e416a54a5eb37071f5a12f07ab P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:17.617027 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 389) took 1528 ms. Trace:
I20250903 20:05:17.617221 27684 rpcz_store.cc:276] 0903 20:05:16.088376 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:16.088437 (+ 61us) service_pool.cc:225] Handling call
0903 20:05:17.603587 (+1515150us) ts_tablet_manager.cc:1145] Deleting tablet 68cff0e416a54a5eb37071f5a12f07ab
0903 20:05:17.603595 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.611430 (+ 7835us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:17.614811 (+ 3381us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:17.616954 (+ 2143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1511917}
W20250903 20:05:17.617796 27462 leader_election.cc:343] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 31342583c026438cbbae14106e75bbe5
I20250903 20:05:17.618386 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 68cff0e416a54a5eb37071f5a12f07ab (table default.Table_1e3f794ae5ec4ad6a9b4351b4bda7b15 [id=71d850a0cea14b0ba337a822ad99d513]) successfully deleted
I20250903 20:05:17.618569 27695 tablet_bootstrap.cc:492] T 4c543ac4df33429b9787c6d32bf1ebeb P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:17.618902 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 390) took 1530 ms. Trace:
I20250903 20:05:17.619048 27684 rpcz_store.cc:276] 0903 20:05:16.088705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:16.088780 (+ 75us) service_pool.cc:225] Handling call
0903 20:05:17.618738 (+1529958us) ts_tablet_manager.cc:1145] Deleting tablet 47c1287bff1145a5b2ab6d14dbd608c7
0903 20:05:17.618749 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.618839 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1526903}
I20250903 20:05:17.619482 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 330) took 2335 ms. Trace:
W20250903 20:05:17.619509 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 47c1287bff1145a5b2ab6d14dbd608c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47c1287bff1145a5b2ab6d14dbd608c7 already in progress: creating tablet
I20250903 20:05:17.619853 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 31342583c026438cbbae14106e75bbe5 (table default.Table_943bea896c744410a2a4d80ffb9ca439 [id=a950fbaecd7341989961ff3ea9a713e0]) successfully deleted
I20250903 20:05:17.619927 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 391) took 1530 ms. Trace:
I20250903 20:05:17.620098 27684 rpcz_store.cc:276] 0903 20:05:16.089185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:16.089253 (+ 68us) service_pool.cc:225] Handling call
0903 20:05:17.619777 (+1530524us) ts_tablet_manager.cc:1145] Deleting tablet 238f2f30377b42c2852b016018aa3245
0903 20:05:17.619787 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.619875 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1527743}
W20250903 20:05:17.620524 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 238f2f30377b42c2852b016018aa3245 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 238f2f30377b42c2852b016018aa3245 already in progress: creating tablet
I20250903 20:05:17.620745 27693 tablet_bootstrap.cc:492] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:17.622272 27687 ts_tablet_manager.cc:1905] T 68cff0e416a54a5eb37071f5a12f07ab P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:17.622658 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 4739f10128d64e2c986257b1d9deb90f (DEFAULT_TABLE table=default.Table_cc45498ec439442797707c4fdd3692bc [id=45d7231b92f54017bd1f5eeb837fa166]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.622833 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 392) took 1532 ms. Trace:
I20250903 20:05:17.622917 27684 rpcz_store.cc:276] 0903 20:05:16.090947 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:16.135242 (+ 44295us) service_pool.cc:225] Handling call
0903 20:05:17.622092 (+1486850us) ts_tablet_manager.cc:1145] Deleting tablet 120ca61b3da742dc9a2efa483ead7e97
0903 20:05:17.622102 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.622179 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1442155}
W20250903 20:05:17.623400 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 120ca61b3da742dc9a2efa483ead7e97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 120ca61b3da742dc9a2efa483ead7e97 already in progress: creating tablet
I20250903 20:05:17.623842 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 393) took 1488 ms. Trace:
I20250903 20:05:17.623973 27684 rpcz_store.cc:276] 0903 20:05:16.135413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:16.153986 (+ 18573us) service_pool.cc:225] Handling call
0903 20:05:17.623689 (+1469703us) ts_tablet_manager.cc:1145] Deleting tablet e78e6d212daf4589a6b4b65711e356cb
0903 20:05:17.623700 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.623792 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1443680}
W20250903 20:05:17.624468 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet e78e6d212daf4589a6b4b65711e356cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e78e6d212daf4589a6b4b65711e356cb already in progress: creating tablet
I20250903 20:05:17.624574 27223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9375fff5f5945078c4412aa21c06f0d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:17.625165 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 886b8783aa9d4618aad498d4a1115843 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 886b8783aa9d4618aad498d4a1115843 already in progress: creating tablet
I20250903 20:05:17.625222 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ea37668d64d4feeb92a162711609c13" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:17.625777 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet f326ec776ba446be8d5dda5c1dc5b6ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f326ec776ba446be8d5dda5c1dc5b6ec already in progress: creating tablet
I20250903 20:05:17.625751 27229 tablet_service.cc:1468] Processing CreateTablet for tablet 249362a0b8304719863bcb82c47273b3 (DEFAULT_TABLE table=default.Table_c750811a8d6c4069a3192ca531cf5712 [id=565dc96c38de4efeb4328c01e030eb8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.626034 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet 886b8783aa9d4618aad498d4a1115843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:17.626374 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 9c77bc0e8bb34712bf8109cadbc0fc06 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c77bc0e8bb34712bf8109cadbc0fc06 already in progress: creating tablet
I20250903 20:05:17.626509 27218 tablet_service.cc:1468] Processing CreateTablet for tablet f4dec89fc6bb436ba6542c267eaf9c15 (DEFAULT_TABLE table=default.Table_8c565fdbc0e641bfaee34e0b42f855ee [id=b0e22c62f5b3455e8b180eb1ae0de718]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:17.626906 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 950a6fcd35f64f2d889831fe9cdf503a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 950a6fcd35f64f2d889831fe9cdf503a already in progress: creating tablet
W20250903 20:05:17.627384 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet f71d3dd9af2543299296f21381d1d563 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f71d3dd9af2543299296f21381d1d563 already in progress: creating tablet
I20250903 20:05:17.627712 27684 tablet_replica.cc:331] T 79a0553c912049cca5971b1ad6f7e11c P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:17.627921 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fdc77810f27a40758e063b8b1150ab71" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:17.628284 27684 raft_consensus.cc:2241] T 79a0553c912049cca5971b1ad6f7e11c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:17.628741 27684 raft_consensus.cc:2270] T 79a0553c912049cca5971b1ad6f7e11c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:17.628970 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31342583c026438cbbae14106e75bbe5" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:17.628484 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9129a3e773a4fecb28ceee27778a294" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:17.629714 27386 raft_consensus.cc:3058] T 31342583c026438cbbae14106e75bbe5 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:17.630589 27384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79a0553c912049cca5971b1ad6f7e11c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:17.630752 27384 raft_consensus.cc:3058] T 79a0553c912049cca5971b1ad6f7e11c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:17.632848 27684 ts_tablet_manager.cc:1905] T 79a0553c912049cca5971b1ad6f7e11c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:17.633397 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet fd499b81344c4b58bf77f3622422763c (table default.Table_e914faf335d44d019f91bda71758c473 [id=374b9abdf6a54b38876a96d8ec4a0368]) successfully deleted
W20250903 20:05:17.633662 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 22bc87d1a4804b6c80f77f9692500d63 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22bc87d1a4804b6c80f77f9692500d63 already in progress: creating tablet
W20250903 20:05:17.633852 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 4c18a98d5c0f4a43ae91c9130047b681 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c18a98d5c0f4a43ae91c9130047b681 already in progress: creating tablet
W20250903 20:05:17.634126 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 4c18a98d5c0f4a43ae91c9130047b681 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c18a98d5c0f4a43ae91c9130047b681 already in progress: creating tablet
I20250903 20:05:17.634696 27225 tablet_service.cc:1468] Processing CreateTablet for tablet ea3f82f4e4244219ab0919f3e341aead (DEFAULT_TABLE table=default.Table_e1977f95b5174bb2b40623d225aff528 [id=e9bd7251cdb64e6bbbdc984babbbe88d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.634960 27685 rpcz_store.cc:276] 0903 20:05:15.276949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.277361 (+ 412us) service_pool.cc:225] Handling call
0903 20:05:17.037268 (+1759907us) ts_tablet_manager.cc:1145] Deleting tablet 262a3549867a473aaaa7dfcc0b5872cd
0903 20:05:17.037276 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.059278 (+ 22002us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:17.525220 (+465942us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:17.612002 (+ 86782us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1759632}
I20250903 20:05:17.638474 27229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 249362a0b8304719863bcb82c47273b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.629480 27382 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bc3b9771a0042fd87e706204dcbb712" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:17.640046 27692 tablet_bootstrap.cc:492] T 73f65e7515994b898eec2ea23903aff4 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:17.642345 28030 raft_consensus.cc:491] T eced730bdbc14a8c9f6b7b7e53435fd0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:17.641861 27692 ts_tablet_manager.cc:1397] T 73f65e7515994b898eec2ea23903aff4 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.157s user 0.004s sys 0.007s
I20250903 20:05:17.643612 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 262a3549867a473aaaa7dfcc0b5872cd (table default.Table_14af7742726b4562a02fd2c9b2895277 [id=6d72901e5102476890f466fd406a7127]) successfully deleted
I20250903 20:05:17.644079 27693 tablet_bootstrap.cc:654] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:17.645121 27221 tablet_service.cc:1468] Processing CreateTablet for tablet 7d5a3fbe93e44c228222ab1b1e78c187 (DEFAULT_TABLE table=default.Table_af52d46a98f249a7982d675a93661af0 [id=04c003dc02d74dce9ad96aa42ff9d6dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.645843 27685 tablet_replica.cc:331] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:17.646410 27224 tablet_service.cc:1468] Processing CreateTablet for tablet 60c32370a5274347afafc6a7e1e94270 (DEFAULT_TABLE table=default.Table_e9cb6a34300a4887b169da952eee37a5 [id=ed3fb47033d145de97ae00dfdf7f8243]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.652727 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1fef9400227449ed892daff588999caa" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:17.630111 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b3121fce1c44d16a0b083313dc67b7a" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:17.642375 27684 ts_tablet_manager.cc:1918] T 79a0553c912049cca5971b1ad6f7e11c P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:17.653390 27692 raft_consensus.cc:357] T 73f65e7515994b898eec2ea23903aff4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:17.655076 27684 log.cc:1199] T 79a0553c912049cca5971b1ad6f7e11c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/79a0553c912049cca5971b1ad6f7e11c
I20250903 20:05:17.655503 27692 raft_consensus.cc:383] T 73f65e7515994b898eec2ea23903aff4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:17.656615 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4739f10128d64e2c986257b1d9deb90f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.656574 28030 raft_consensus.cc:513] T eced730bdbc14a8c9f6b7b7e53435fd0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:17.657318 28030 leader_election.cc:290] T eced730bdbc14a8c9f6b7b7e53435fd0 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:17.627079 28135 consensus_queue.cc:237] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:17.657905 27684 ts_tablet_manager.cc:1939] T 79a0553c912049cca5971b1ad6f7e11c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:17.656559 27692 raft_consensus.cc:738] T 73f65e7515994b898eec2ea23903aff4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:17.658953 27692 consensus_queue.cc:260] T 73f65e7515994b898eec2ea23903aff4 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:17.659143 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eced730bdbc14a8c9f6b7b7e53435fd0" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:17.662320 27692 ts_tablet_manager.cc:1428] T 73f65e7515994b898eec2ea23903aff4 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.020s user 0.003s sys 0.000s
I20250903 20:05:17.663707 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 79a0553c912049cca5971b1ad6f7e11c (table default.Table_afc0f683c5d8460484160a3a9c3e6df5 [id=b89ea241456b4393994ad07d839053f1]) successfully deleted
I20250903 20:05:17.664302 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4dec89fc6bb436ba6542c267eaf9c15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.664687 27692 tablet_bootstrap.cc:492] T 44db427ee53a4d0a907dfd63282d6d4f P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
W20250903 20:05:17.666399 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet e0347eab6c8642beabdc2d7e7bbfdc9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0347eab6c8642beabdc2d7e7bbfdc9b already in progress: creating tablet
I20250903 20:05:17.668259 27386 raft_consensus.cc:2466] T 31342583c026438cbbae14106e75bbe5 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:17.668767 27462 leader_election.cc:304] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:17.669166 28169 raft_consensus.cc:2802] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:17.669279 28169 raft_consensus.cc:695] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:17.669564 28169 consensus_queue.cc:237] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:17.670980 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4e34d6d5d68e4395aed78821ea077c1b consensus state has no leader
W20250903 20:05:17.672019 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 5fb2426aa01e47598f268d0a90475271 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5fb2426aa01e47598f268d0a90475271 already in progress: creating tablet
W20250903 20:05:17.672290 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 33fb6b357bda43e7abeaed5634183ef6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33fb6b357bda43e7abeaed5634183ef6 already in progress: creating tablet
I20250903 20:05:17.672891 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 2a5c74a281b44019872a6442d41bc373
I20250903 20:05:17.673416 27063 master_service.cc:948] ReplaceTablet: received request to replace tablet 541278ca9ffb4e41b341139c2faa337f from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:17.674381 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet 2a5c74a281b44019872a6442d41bc373 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:17 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:17.674496 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2cedbabdccbb444789b77ff293edf019" } modify_external_catalogs: true
I20250903 20:05:17.674672 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2cedbabdccbb444789b77ff293edf019" } modify_external_catalogs: true
I20250903 20:05:17.675199 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ca35470d1cb0403089893a84293d1ea2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:17.675866 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca35470d1cb0403089893a84293d1ea2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:17.676309 27384 raft_consensus.cc:2466] T 79a0553c912049cca5971b1ad6f7e11c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
W20250903 20:05:17.675429 27184 consensus_peers.cc:489] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:17.677317 27059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_91ae8b84b5b94998a81cf961b29bf4a4" } modify_external_catalogs: true
I20250903 20:05:17.677511 27059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_91ae8b84b5b94998a81cf961b29bf4a4" } modify_external_catalogs: true
I20250903 20:05:17.677585 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 2a5c74a281b44019872a6442d41bc373 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:17 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:17.680058 27058 master_service.cc:948] ReplaceTablet: received request to replace tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:17.680967 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60c32370a5274347afafc6a7e1e94270. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.681295 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5eacd4f8a0974bdbabb297ffd1cd4a98" } new_table_name: "default.Table_f1f4f57d96f04f0caff06d9a642f5c46" modify_external_catalogs: true
I20250903 20:05:17.683372 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea3f82f4e4244219ab0919f3e341aead. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.683516 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet 68cff0e416a54a5eb37071f5a12f07ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:17.683782 27491 tablet_service.cc:1515] Processing DeleteTablet for tablet 691cb69759654fdc9223844049a06d02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:17.684334 27693 tablet_bootstrap.cc:492] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
W20250903 20:05:17.684306 27183 leader_election.cc:343] T eced730bdbc14a8c9f6b7b7e53435fd0 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:17.684499 27693 ts_tablet_manager.cc:1397] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.101s user 0.001s sys 0.004s
I20250903 20:05:17.685441 27693 raft_consensus.cc:357] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:17.685757 27693 raft_consensus.cc:383] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:17.685861 27693 raft_consensus.cc:738] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:17.686102 27693 consensus_queue.cc:260] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:17.686650 27693 ts_tablet_manager.cc:1428] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250903 20:05:17.687959 27693 tablet_bootstrap.cc:492] T c9c870fb14e3469fb2979d8462b49121 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:17.688352 28132 raft_consensus.cc:491] T 262a3549867a473aaaa7dfcc0b5872cd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:17.688550 28132 raft_consensus.cc:513] T 262a3549867a473aaaa7dfcc0b5872cd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:17.689417 28132 leader_election.cc:290] T 262a3549867a473aaaa7dfcc0b5872cd P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:17.690140 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 31342583c026438cbbae14106e75bbe5 with delete_type TABLET_DATA_DELETED (replaced by tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:17.692662 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 691cb69759654fdc9223844049a06d02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:17.695215 27320 leader_election.cc:343] T 262a3549867a473aaaa7dfcc0b5872cd P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 262a3549867a473aaaa7dfcc0b5872cd
I20250903 20:05:17.697520 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 04ccb3e4a8bd4789b9f19edd4a5b6145 with delete_type TABLET_DATA_DELETED (replaced by tablet a4c582b441f6467c99ba0b090abcf4cc) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:17.697965 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 262a3549867a473aaaa7dfcc0b5872cd with delete_type TABLET_DATA_DELETED (replaced by tablet 7c398634d6304a12b35c0579fba4c813) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:17.698326 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d5a3fbe93e44c228222ab1b1e78c187. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.699124 27687 ts_tablet_manager.cc:1918] T 68cff0e416a54a5eb37071f5a12f07ab P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:17.700865 27687 log.cc:1199] T 68cff0e416a54a5eb37071f5a12f07ab P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/68cff0e416a54a5eb37071f5a12f07ab
I20250903 20:05:17.699612 27693 tablet_bootstrap.cc:654] T c9c870fb14e3469fb2979d8462b49121 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:17.702433 27321 leader_election.cc:343] T 262a3549867a473aaaa7dfcc0b5872cd P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 262a3549867a473aaaa7dfcc0b5872cd
I20250903 20:05:17.702593 27321 leader_election.cc:304] T 262a3549867a473aaaa7dfcc0b5872cd P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:17.703069 28132 raft_consensus.cc:2747] T 262a3549867a473aaaa7dfcc0b5872cd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:17.704442 27687 ts_tablet_manager.cc:1939] T 68cff0e416a54a5eb37071f5a12f07ab P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:17.704772 27220 tablet_service.cc:1468] Processing CreateTablet for tablet 9c77bc0e8bb34712bf8109cadbc0fc06 (DEFAULT_TABLE table=default.Table_119f28a77b72449f9374b78ff4b7e016 [id=07b954506fb7480f816756d30f35b4d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.705214 27488 tablet_service.cc:1468] Processing CreateTablet for tablet b84f487773c24735a61d0fd0898b2bcb (DEFAULT_TABLE table=default.Table_2cedbabdccbb444789b77ff293edf019 [id=cdf58c7117de4317b34a57666ad17d85]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.705637 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b84f487773c24735a61d0fd0898b2bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.710670 27499 tablet_service.cc:1468] Processing CreateTablet for tablet 6d21f38759d743af972635f8b5b8d8d2 (DEFAULT_TABLE table=default.Table_e17243085e464908adf84082adbb3800 [id=2f0e4907f017416bb5790e27fc79aeb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.711226 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d21f38759d743af972635f8b5b8d8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.711406 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c77bc0e8bb34712bf8109cadbc0fc06. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.710855 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 389) took 1367 ms. Trace:
I20250903 20:05:17.712134 27687 rpcz_store.cc:276] 0903 20:05:16.342093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:16.342294 (+ 201us) service_pool.cc:225] Handling call
0903 20:05:17.567812 (+1225518us) ts_tablet_manager.cc:1145] Deleting tablet 68cff0e416a54a5eb37071f5a12f07ab
0903 20:05:17.567824 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.672975 (+105151us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:17.694903 (+ 21928us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:17.709316 (+ 14413us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1223929}
I20250903 20:05:17.715126 27491 tablet_service.cc:1468] Processing CreateTablet for tablet 847d0a0a3d2b45938cec9058c77f6231 (DEFAULT_TABLE table=default.Table_196cfa99ce59433b85ad81b45d853081 [id=88d56563f5dc4fecb59c106ff78499e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.715584 27491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 847d0a0a3d2b45938cec9058c77f6231. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.717133 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 390) took 1373 ms. Trace:
I20250903 20:05:17.717236 27687 rpcz_store.cc:276] 0903 20:05:16.343301 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:16.343382 (+ 81us) service_pool.cc:225] Handling call
0903 20:05:17.714181 (+1370799us) ts_tablet_manager.cc:1145] Deleting tablet 47c1287bff1145a5b2ab6d14dbd608c7
0903 20:05:17.714195 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.717068 (+ 2873us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1367585}
I20250903 20:05:17.717715 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 392) took 1370 ms. Trace:
I20250903 20:05:17.717818 27687 rpcz_store.cc:276] 0903 20:05:16.347031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:16.347162 (+ 131us) service_pool.cc:225] Handling call
0903 20:05:17.717516 (+1370354us) ts_tablet_manager.cc:1145] Deleting tablet 120ca61b3da742dc9a2efa483ead7e97
0903 20:05:17.717527 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.717642 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1369513}
I20250903 20:05:17.718268 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 393) took 1370 ms. Trace:
I20250903 20:05:17.718355 27687 rpcz_store.cc:276] 0903 20:05:16.347725 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:16.347799 (+ 74us) service_pool.cc:225] Handling call
0903 20:05:17.718072 (+1370273us) ts_tablet_manager.cc:1145] Deleting tablet e78e6d212daf4589a6b4b65711e356cb
0903 20:05:17.718083 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.718189 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1369919}
I20250903 20:05:17.718591 27497 tablet_service.cc:1468] Processing CreateTablet for tablet 05e949476a834762973cb98923420f69 (DEFAULT_TABLE table=default.Table_08b8a5b3a94c4232b361faecf839c494 [id=7ba2fae44f1340ab9dd9f4deb586ca76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.718699 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 391) took 1373 ms. Trace:
I20250903 20:05:17.718802 27687 rpcz_store.cc:276] 0903 20:05:16.345571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:16.345658 (+ 87us) service_pool.cc:225] Handling call
0903 20:05:17.718551 (+1372893us) ts_tablet_manager.cc:1145] Deleting tablet 238f2f30377b42c2852b016018aa3245
0903 20:05:17.718560 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.718645 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1366459}
I20250903 20:05:17.719004 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05e949476a834762973cb98923420f69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.719280 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 421) took 1338 ms. Trace:
I20250903 20:05:17.719370 27687 rpcz_store.cc:276] 0903 20:05:16.380865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:16.669104 (+288239us) service_pool.cc:225] Handling call
0903 20:05:17.719084 (+1049980us) ts_tablet_manager.cc:1145] Deleting tablet 886b8783aa9d4618aad498d4a1115843
0903 20:05:17.719095 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.719230 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":957968}
I20250903 20:05:17.719795 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 430) took 1338 ms. Trace:
I20250903 20:05:17.719861 27687 rpcz_store.cc:276] 0903 20:05:16.381768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:16.743395 (+361627us) service_pool.cc:225] Handling call
0903 20:05:17.719616 (+976221us) ts_tablet_manager.cc:1145] Deleting tablet 950a6fcd35f64f2d889831fe9cdf503a
0903 20:05:17.719627 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.719739 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":957864}
I20250903 20:05:17.720245 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 431) took 1338 ms. Trace:
I20250903 20:05:17.720335 27687 rpcz_store.cc:276] 0903 20:05:16.381905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:16.757082 (+375177us) service_pool.cc:225] Handling call
0903 20:05:17.720094 (+963012us) ts_tablet_manager.cc:1145] Deleting tablet f71d3dd9af2543299296f21381d1d563
0903 20:05:17.720105 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.720189 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":956629}
I20250903 20:05:17.723423 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 68cff0e416a54a5eb37071f5a12f07ab (table default.Table_1e3f794ae5ec4ad6a9b4351b4bda7b15 [id=71d850a0cea14b0ba337a822ad99d513]) successfully deleted
W20250903 20:05:17.723776 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 47c1287bff1145a5b2ab6d14dbd608c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47c1287bff1145a5b2ab6d14dbd608c7 already in progress: creating tablet
W20250903 20:05:17.724013 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 120ca61b3da742dc9a2efa483ead7e97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 120ca61b3da742dc9a2efa483ead7e97 already in progress: creating tablet
W20250903 20:05:17.724248 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet e78e6d212daf4589a6b4b65711e356cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e78e6d212daf4589a6b4b65711e356cb already in progress: creating tablet
W20250903 20:05:17.724471 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 238f2f30377b42c2852b016018aa3245 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 238f2f30377b42c2852b016018aa3245 already in progress: creating tablet
W20250903 20:05:17.724686 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 886b8783aa9d4618aad498d4a1115843 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 886b8783aa9d4618aad498d4a1115843 already in progress: creating tablet
W20250903 20:05:17.724892 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 950a6fcd35f64f2d889831fe9cdf503a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 950a6fcd35f64f2d889831fe9cdf503a already in progress: creating tablet
W20250903 20:05:17.725114 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet f71d3dd9af2543299296f21381d1d563 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f71d3dd9af2543299296f21381d1d563 already in progress: creating tablet
I20250903 20:05:17.726199 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 05e949476a834762973cb98923420f69 (DEFAULT_TABLE table=default.Table_08b8a5b3a94c4232b361faecf839c494 [id=7ba2fae44f1340ab9dd9f4deb586ca76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.726800 27349 tablet_service.cc:1468] Processing CreateTablet for tablet 847d0a0a3d2b45938cec9058c77f6231 (DEFAULT_TABLE table=default.Table_196cfa99ce59433b85ad81b45d853081 [id=88d56563f5dc4fecb59c106ff78499e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.736086 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 6d21f38759d743af972635f8b5b8d8d2 (DEFAULT_TABLE table=default.Table_e17243085e464908adf84082adbb3800 [id=2f0e4907f017416bb5790e27fc79aeb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.736434 27365 tablet_service.cc:1468] Processing CreateTablet for tablet b84f487773c24735a61d0fd0898b2bcb (DEFAULT_TABLE table=default.Table_2cedbabdccbb444789b77ff293edf019 [id=cdf58c7117de4317b34a57666ad17d85]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.744766 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 446) took 1057 ms. Trace:
I20250903 20:05:17.744947 27684 rpcz_store.cc:276] 0903 20:05:16.687402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.051616 (+364214us) service_pool.cc:225] Handling call
0903 20:05:17.744606 (+692990us) ts_tablet_manager.cc:1145] Deleting tablet 60c32370a5274347afafc6a7e1e94270
0903 20:05:17.744615 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.744727 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":692789}
W20250903 20:05:17.745429 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 60c32370a5274347afafc6a7e1e94270 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60c32370a5274347afafc6a7e1e94270 already in progress: creating tablet
I20250903 20:05:17.747427 27692 tablet_bootstrap.cc:654] T 44db427ee53a4d0a907dfd63282d6d4f P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:17.747723 27685 raft_consensus.cc:2241] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:17.750164 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 444) took 1062 ms. Trace:
I20250903 20:05:17.751529 27684 rpcz_store.cc:276] 0903 20:05:16.687200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.018421 (+331221us) service_pool.cc:225] Handling call
0903 20:05:17.749972 (+731551us) ts_tablet_manager.cc:1145] Deleting tablet ee40566e497740789575275d0cdca0db
0903 20:05:17.749983 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.750120 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":671245}
I20250903 20:05:17.751966 26885 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskH7eH49/build/asan/bin/kudu
/tmp/dist-test-taskH7eH49/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.65.125:44059
--webserver_interface=127.26.65.125
--webserver_port=41613
--builtin_ntp_servers=127.26.65.84:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.65.126:39015,127.26.65.125:44059,127.26.65.124:40911
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250903 20:05:17.752390 27684 tablet_replica.cc:331] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:17.753248 27228 tablet_service.cc:1515] Processing DeleteTablet for tablet f326ec776ba446be8d5dda5c1dc5b6ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:17.753453 27228 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c77bc0e8bb34712bf8109cadbc0fc06 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:17.753643 27228 tablet_service.cc:1515] Processing DeleteTablet for tablet 950a6fcd35f64f2d889831fe9cdf503a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:17.753834 27228 tablet_service.cc:1515] Processing DeleteTablet for tablet f71d3dd9af2543299296f21381d1d563 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:17.758476 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet ee40566e497740789575275d0cdca0db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee40566e497740789575275d0cdca0db already in progress: creating tablet
I20250903 20:05:17.761417 27206 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:56514 (request call id 94) took 1861 ms. Trace:
I20250903 20:05:17.765709 27205 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:56514 (request call id 93) took 1866 ms. Trace:
I20250903 20:05:17.769013 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05e949476a834762973cb98923420f69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.772552 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_03ce0e151fcc4259ba52504c9c5ddfcd" } new_table_name: "default.Table_975e53d37e054938a5b63957252ff50d" modify_external_catalogs: true
I20250903 20:05:17.773864 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_367e5dbc110d47b38f43d9dfdcfce307"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:17.774499 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_367e5dbc110d47b38f43d9dfdcfce307 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:17.774613 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_272defb026dc43978827f668aeedc48c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:17.774930 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 847d0a0a3d2b45938cec9058c77f6231. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:17.775147 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_272defb026dc43978827f668aeedc48c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:17.778586 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d21f38759d743af972635f8b5b8d8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.781713 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 6d13884fff8c43b690836c6a341a2d92
I20250903 20:05:17.782310 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 8218868ff40b405387d560f614427c08
I20250903 20:05:17.782618 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_bc654adac8a24afab120af90f0a73f36" } new_table_name: "default.Table_ccda245b73064dc2b5634f9593b1888c" modify_external_catalogs: true
I20250903 20:05:17.783004 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e1991ea3a3d14b85a4ce9160fa00bc35" } new_table_name: "default.Table_4258f924ddb3449bb5f49fb7a88a5073" modify_external_catalogs: true
I20250903 20:05:17.784185 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b84f487773c24735a61d0fd0898b2bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.784345 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 541278ca9ffb4e41b341139c2faa337f
I20250903 20:05:17.784749 27063 catalog_manager.cc:6652] ReplaceTablet: tablet 541278ca9ffb4e41b341139c2faa337f deleted and replaced by tablet e9126e4a18c546c28927f22c9f16426e
I20250903 20:05:17.785029 28132 raft_consensus.cc:491] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:17.785228 28132 raft_consensus.cc:513] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:17.785851 28132 leader_election.cc:290] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:17.786377 27352 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d13884fff8c43b690836c6a341a2d92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:17 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:17.786634 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 8218868ff40b405387d560f614427c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:17 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:17.786826 27356 tablet_service.cc:1515] Processing DeleteTablet for tablet 541278ca9ffb4e41b341139c2faa337f with delete_type TABLET_DATA_DELETED (replaced by tablet e9126e4a18c546c28927f22c9f16426e) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:17.774621 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_cd761ea7907f44a6bcf538648450b701"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:17.787817 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd761ea7907f44a6bcf538648450b701 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:17.767266 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 351208baafc94f5eab531101d52e3b3f (DEFAULT_TABLE table=default.Table_f05d6db7f87643ebbb52602f99530905 [id=c393a6de2dbe4ccba47c6d0510a88e3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.793182 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 428) took 1411 ms. Trace:
I20250903 20:05:17.793315 27687 rpcz_store.cc:276] 0903 20:05:16.381582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:16.736118 (+354536us) service_pool.cc:225] Handling call
0903 20:05:17.720568 (+984450us) ts_tablet_manager.cc:1145] Deleting tablet f326ec776ba446be8d5dda5c1dc5b6ec
0903 20:05:17.768608 (+ 48040us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.793110 (+ 24502us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":931804}
I20250903 20:05:17.793736 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 429) took 1412 ms. Trace:
W20250903 20:05:17.793807 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet f326ec776ba446be8d5dda5c1dc5b6ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f326ec776ba446be8d5dda5c1dc5b6ec already in progress: creating tablet
I20250903 20:05:17.793821 27687 rpcz_store.cc:276] 0903 20:05:16.381655 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:16.742923 (+361268us) service_pool.cc:225] Handling call
0903 20:05:17.793586 (+1050663us) ts_tablet_manager.cc:1145] Deleting tablet 9c77bc0e8bb34712bf8109cadbc0fc06
0903 20:05:17.793596 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.793683 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":982596}
I20250903 20:05:17.794143 27687 tablet_replica.cc:331] T 4268316c28e8471a9f28807d20dbb395 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
W20250903 20:05:17.794272 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 9c77bc0e8bb34712bf8109cadbc0fc06 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c77bc0e8bb34712bf8109cadbc0fc06 already in progress: creating tablet
I20250903 20:05:17.794368 27687 raft_consensus.cc:2241] T 4268316c28e8471a9f28807d20dbb395 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:17.794658 27695 tablet_bootstrap.cc:654] T 4c543ac4df33429b9787c6d32bf1ebeb P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:17.754201 27228 tablet_service.cc:1468] Processing CreateTablet for tablet 5c42d84a6a5649208b0539a10da21aec (DEFAULT_TABLE table=default.Table_5187f549b3d44cce99c527429ce39a4b [id=bc5ade88feba4061a0f5ad0ffa347bad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.796676 27206 rpcz_store.cc:276] 0903 20:05:15.899712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.899776 (+ 64us) service_pool.cc:225] Handling call
0903 20:05:17.403022 (+1503246us) spinlock_profiling.cc:243] Waited 491 ms on lock 0x6160000d3354. stack: 00007f589abe0dc6 00007f589abe0b80 00007f5899bc83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f58a6639700 00007f58aa3d371f 00007f58a4b04c36 00007f58a4b04965 00007f58a19910be 00007f58a198ff9e 00007f58a19961d5 00007f58a1996aff 00007f58a1996928 0000000000a774dc
0903 20:05:17.511335 (+108313us) spinlock_profiling.cc:243] Waited 2.46 ms on lock 0x61800009408c. stack: 00007f589abe0dc6 00007f589abe0b80 00007f5899bc83ad 0000000000972d94 0000000000972d5d 000000000097272a 00000000009f0c70 00007f58aa3d3841 00007f58a4b04c36 00007f58a4b04965 00007f58a19910be 00007f58a198ff9e 00007f58a19961d5 00007f58a1996aff 00007f58a1996928 0000000000a774dc
0903 20:05:17.753807 (+242472us) spinlock_profiling.cc:243] Waited 41.6 ms on lock 0x6160000c4054. stack: 00007f589abe0dc6 00007f589abe0b80 00007f5899bc83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f58a6639700 00007f58aa3d371f 00007f58a4b04c36 00007f58a4b04965 00007f58a19910be 00007f58a198ff9e 00007f58a19961d5 00007f58a1996aff 00007f58a1996928 0000000000a774dc
0903 20:05:17.761333 (+ 7526us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1229811712}
I20250903 20:05:17.798331 27685 pending_rounds.cc:70] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a: Trying to abort 1 pending ops.
I20250903 20:05:17.799916 27687 raft_consensus.cc:2270] T 4268316c28e8471a9f28807d20dbb395 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:17.800871 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet b84f487773c24735a61d0fd0898b2bcb
I20250903 20:05:17.801735 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet f9709dfd18654c5bb071e39da7c8889f
I20250903 20:05:17.796604 27205 rpcz_store.cc:276] 0903 20:05:15.899622 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.900672 (+ 1050us) service_pool.cc:225] Handling call
0903 20:05:17.402541 (+1501869us) spinlock_profiling.cc:243] Waited 457 ms on lock 0x6160000d3354. stack: 00007f589abe0dc6 00007f589abe0b80 00007f5899bc83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f58a6639700 00007f58aa3d371f 00007f58a4b04c36 00007f58a4b04965 00007f58a19910be 00007f58a198ff9e 00007f58a19961d5 00007f58a1996aff 00007f58a1996928 0000000000a774dc
0903 20:05:17.761578 (+359037us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x6160000c4054. stack: 00007f589abe0dc6 00007f589abe0b80 00007f5899bc83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f58a6639700 00007f58aa3d371f 00007f58a4b04c36 00007f58a4b04965 00007f58a19910be 00007f58a198ff9e 00007f58a19961d5 00007f58a1996aff 00007f58a1996928 0000000000a774dc
0903 20:05:17.765631 (+ 4053us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1291988480}
I20250903 20:05:17.802878 27356 tablet_service.cc:1515] Processing DeleteTablet for tablet f9709dfd18654c5bb071e39da7c8889f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:17 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:17.804430 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet b84f487773c24735a61d0fd0898b2bcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:17 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:17.805380 27693 tablet_bootstrap.cc:492] T c9c870fb14e3469fb2979d8462b49121 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:17.805549 27693 ts_tablet_manager.cc:1397] T c9c870fb14e3469fb2979d8462b49121 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.119s user 0.001s sys 0.006s
I20250903 20:05:17.806577 27693 raft_consensus.cc:357] T c9c870fb14e3469fb2979d8462b49121 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:17.806777 27693 raft_consensus.cc:383] T c9c870fb14e3469fb2979d8462b49121 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:17.806849 27693 raft_consensus.cc:738] T c9c870fb14e3469fb2979d8462b49121 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:17.807194 27693 consensus_queue.cc:260] T c9c870fb14e3469fb2979d8462b49121 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:17.807483 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 912eee5b48ef467bbd6bbe34f8aa390c consensus state has no leader
I20250903 20:05:17.807703 27693 ts_tablet_manager.cc:1428] T c9c870fb14e3469fb2979d8462b49121 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:17.808043 27693 tablet_bootstrap.cc:492] T 44db427ee53a4d0a907dfd63282d6d4f P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:17.810058 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "691cb69759654fdc9223844049a06d02" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:17.810322 27527 raft_consensus.cc:2466] T 691cb69759654fdc9223844049a06d02 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:17.810904 27693 tablet_bootstrap.cc:654] T 44db427ee53a4d0a907dfd63282d6d4f P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:17.814327 27320 leader_election.cc:304] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:17.814687 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet 73f65e7515994b898eec2ea23903aff4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:17.815495 28132 raft_consensus.cc:2802] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:17.815605 28132 raft_consensus.cc:491] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:17.815721 28132 raft_consensus.cc:3058] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:17.816133 27684 raft_consensus.cc:2241] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:17.816499 27684 pending_rounds.cc:70] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f: Trying to abort 1 pending ops.
I20250903 20:05:17.816876 27684 pending_rounds.cc:77] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384942777040896 op_type: NO_OP noop_request { }
I20250903 20:05:17.817016 27684 raft_consensus.cc:2887] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:17.817651 27684 raft_consensus.cc:2270] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:17.819651 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d13884fff8c43b690836c6a341a2d92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:17 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:17.820098 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 314856bf269f4df7aa25f5a30cec0d5b (DEFAULT_TABLE table=default.Table_fc325fc838f44caa818ddcddb486d85f [id=46b4934a16734ec3a7c1570b2d0cfb53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.820501 27685 pending_rounds.cc:77] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384942726635520 op_type: NO_OP noop_request { }
I20250903 20:05:17.820713 27491 tablet_service.cc:1515] Processing DeleteTablet for tablet 8218868ff40b405387d560f614427c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:17 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:17.821473 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet f9709dfd18654c5bb071e39da7c8889f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:17 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:17.821696 27491 tablet_service.cc:1515] Processing DeleteTablet for tablet 541278ca9ffb4e41b341139c2faa337f with delete_type TABLET_DATA_DELETED (replaced by tablet e9126e4a18c546c28927f22c9f16426e) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:17.821895 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet b84f487773c24735a61d0fd0898b2bcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:17 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:17.822108 27684 ts_tablet_manager.cc:1905] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:17.834239 27219 tablet_service.cc:1468] Processing CreateTablet for tablet 6d13884fff8c43b690836c6a341a2d92 (DEFAULT_TABLE table=default.Table_2015884055c845e5a7bed9306978c4ec [id=cfe13c5e975047c08696b908602f91bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.836871 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 351208baafc94f5eab531101d52e3b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.839148 27685 raft_consensus.cc:2887] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:17.841038 27685 raft_consensus.cc:2270] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:17.841415 27228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c42d84a6a5649208b0539a10da21aec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.840744 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "691cb69759654fdc9223844049a06d02" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:17.842573 27223 tablet_service.cc:1468] Processing CreateTablet for tablet 69f8eef52bee431a86f0eef45fd11f9f (DEFAULT_TABLE table=default.Table_5215f44c06fc49d6a9a740f77e9c21fe [id=efecb95d0f474234959663b789c28b1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.850862 27684 ts_tablet_manager.cc:1918] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:17.852041 27684 log.cc:1199] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/31342583c026438cbbae14106e75bbe5
I20250903 20:05:17.854195 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 314856bf269f4df7aa25f5a30cec0d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.855576 27248 raft_consensus.cc:2466] T 691cb69759654fdc9223844049a06d02 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
W20250903 20:05:17.855540 28180 consensus_queue.cc:1162] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "89f8f0ee68b940db9ce250c0604f934a" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20250903 20:05:17.855931 27213 tablet_service.cc:1468] Processing CreateTablet for tablet 659a127405d341c7b074d3445854256f (DEFAULT_TABLE table=default.Table_30721ff3b68e4c08bdb5845184d8ada7 [id=c807184f006b4eac8c34c84733b6a5fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.856375 27211 tablet_service.cc:1468] Processing CreateTablet for tablet 5fb2426aa01e47598f268d0a90475271 (DEFAULT_TABLE table=default.Table_e4f1c7374cd94b529c3e923355498939 [id=a07de6053e474f988f800d6037e0f651]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.856812 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 733f8024e1fa401ca9614a97db91a19b (DEFAULT_TABLE table=default.Table_4d90c70f5e544f839c056ae573d08f58 [id=46298581b9524c87bf3a0586f4c86c28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.855966 27222 tablet_service.cc:1468] Processing CreateTablet for tablet a686ea49a29a4dd7b6e5dce5e1f9935a (DEFAULT_TABLE table=default.Table_00f2d88f1c284af3b6fbcaf8590938d5 [id=374fcca58f844f91a4561f61341feb3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.856889 27685 ts_tablet_manager.cc:1905] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:17.857843 27184 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 166243us
I20250903 20:05:17.858325 27692 tablet_bootstrap.cc:492] T 44db427ee53a4d0a907dfd63282d6d4f P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:17.858505 27692 ts_tablet_manager.cc:1397] T 44db427ee53a4d0a907dfd63282d6d4f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.194s user 0.004s sys 0.001s
I20250903 20:05:17.859403 27692 raft_consensus.cc:357] T 44db427ee53a4d0a907dfd63282d6d4f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:17.859596 27692 raft_consensus.cc:383] T 44db427ee53a4d0a907dfd63282d6d4f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:17.859683 27692 raft_consensus.cc:738] T 44db427ee53a4d0a907dfd63282d6d4f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:17.860133 27692 consensus_queue.cc:260] T 44db427ee53a4d0a907dfd63282d6d4f P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:17.862257 27692 ts_tablet_manager.cc:1428] T 44db427ee53a4d0a907dfd63282d6d4f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250903 20:05:17.863999 27692 tablet_bootstrap.cc:492] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:17.867014 27684 ts_tablet_manager.cc:1939] T 31342583c026438cbbae14106e75bbe5 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:17.870807 27223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f8eef52bee431a86f0eef45fd11f9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.874542 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 31342583c026438cbbae14106e75bbe5 (table default.Table_943bea896c744410a2a4d80ffb9ca439 [id=a950fbaecd7341989961ff3ea9a713e0]) successfully deleted
W20250903 20:05:17.875499 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 8a223f75045345ae833f52bcc8f239f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a223f75045345ae833f52bcc8f239f9 already in progress: creating tablet
W20250903 20:05:17.876374 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet a3103e1595ec4c38ba65dca26fe1167c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3103e1595ec4c38ba65dca26fe1167c already in progress: creating tablet
I20250903 20:05:17.885629 27687 ts_tablet_manager.cc:1905] T 4268316c28e8471a9f28807d20dbb395 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:17.886474 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 659a127405d341c7b074d3445854256f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.886852 27211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fb2426aa01e47598f268d0a90475271. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.888195 27219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d13884fff8c43b690836c6a341a2d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.886791 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet 5fb2426aa01e47598f268d0a90475271 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:17.880016 28132 raft_consensus.cc:513] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:17.889990 27212 tablet_service.cc:1468] Processing CreateTablet for tablet 0ca6e2165a9d41299c1fe0bfa9475915 (DEFAULT_TABLE table=default.Table_dbc7009c4de44b8e88f3caa4b62ec665 [id=80977b1c4b154e158ce5a6e7e16e695d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.895668 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_480e84d976d54812a2a0ee3d43a4b08b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:17.896350 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_480e84d976d54812a2a0ee3d43a4b08b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:17.898550 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_832c9e0fbe804a61bcf618c06392b38c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:17.899185 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_832c9e0fbe804a61bcf618c06392b38c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:17.900179 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_86616565ede946a5b1043149012526f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:17.900475 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_4f815c3438474e3ca7b8e4b9d0be87d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:17.900848 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86616565ede946a5b1043149012526f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:17.900971 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f815c3438474e3ca7b8e4b9d0be87d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:17.903347 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5b46bb34ac214b1ba23fba1a33f746ac consensus state has no leader
I20250903 20:05:17.903873 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_fed56630d54540548ffb7f00228a1ee9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:17.904491 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fed56630d54540548ffb7f00228a1ee9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:17.905416 27057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ab7e78ea930c4d88bf2e62caa36ca317" } new_table_name: "default.Table_9d4921e7acb249219b9c88ec3615a646" modify_external_catalogs: true
W20250903 20:05:17.908126 27184 leader_election.cc:343] T 6ea37668d64d4feeb92a162711609c13 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:17.909709 27692 tablet_bootstrap.cc:654] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:17.909117 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet ee40566e497740789575275d0cdca0db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:17.909231 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 733f8024e1fa401ca9614a97db91a19b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.913560 27684 tablet_replica.cc:331] T 2a5c74a281b44019872a6442d41bc373 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:17.913862 27684 raft_consensus.cc:2241] T 2a5c74a281b44019872a6442d41bc373 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:17.914070 27684 raft_consensus.cc:2270] T 2a5c74a281b44019872a6442d41bc373 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:17.914448 28132 leader_election.cc:290] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:17.914619 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 6f8cfe53b91b41d7ba7f9074e1005af3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f8cfe53b91b41d7ba7f9074e1005af3 already in progress: creating tablet
W20250903 20:05:17.915277 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet ed4902c002ef4198802375ced39ef2ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed4902c002ef4198802375ced39ef2ad already in progress: creating tablet
W20250903 20:05:17.918005 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet e877d56bbbfa4c5a92d025d0311ce5c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e877d56bbbfa4c5a92d025d0311ce5c6 already in progress: creating tablet
I20250903 20:05:17.918782 27684 ts_tablet_manager.cc:1905] T 2a5c74a281b44019872a6442d41bc373 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:17.925639 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d4c703dafbca4e669dbb18946ffac699 consensus state has no leader
I20250903 20:05:17.935691 27361 tablet_service.cc:1468] Processing CreateTablet for tablet b3d9491f777b422da4b9fca803beabff (DEFAULT_TABLE table=default.Table_93c4817ac1c64d589e71a4590d32adb2 [id=31352bcca5164b158205533584d1b352]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.936149 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3d9491f777b422da4b9fca803beabff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.943044 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 5f2eefeebbfc47249e2f72b7c52e905d (DEFAULT_TABLE table=default.Table_359258cdcd734ef4b3463c80fa1c582a [id=7587e35155ff40be9d3f8810e208b591]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.943632 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f2eefeebbfc47249e2f72b7c52e905d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.943809 27356 tablet_service.cc:1468] Processing CreateTablet for tablet 112c0dccb60c4c7c9efc164d9c8443ea (DEFAULT_TABLE table=default.Table_4a511c041bed4d8b88479f001d13c791 [id=c91f93024bca4ad4a06d55cf35266df5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.944240 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 112c0dccb60c4c7c9efc164d9c8443ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.946609 27349 tablet_service.cc:1468] Processing CreateTablet for tablet 5132e6d4288c447e80f0608ba366b69c (DEFAULT_TABLE table=default.Table_2c769d77af8d429fabe07bdd9ab6c61c [id=3a3fd5d2b11246cb92787d0471494285]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.946903 27352 tablet_service.cc:1468] Processing CreateTablet for tablet 7cbbeb8bec5e464b85b8568e916c765f (DEFAULT_TABLE table=default.Table_d37cdb880e224494969e67d03305b7fa [id=aaaf37ae7fe84ec9a4ebdd0ab1827589]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.947034 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5132e6d4288c447e80f0608ba366b69c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.947278 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cbbeb8bec5e464b85b8568e916c765f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.947566 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 54602d7f723d4c10af12408a43ed1f02 (DEFAULT_TABLE table=default.Table_d4c00bc20db74e4c84da0f99bd594b19 [id=2eed1473230e4beab5701483fd3e4315]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.947932 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54602d7f723d4c10af12408a43ed1f02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.949930 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet e0347eab6c8642beabdc2d7e7bbfdc9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:17.946619 27353 tablet_service.cc:1468] Processing CreateTablet for tablet 41056c5c5003415993bee74d7621bd14 (DEFAULT_TABLE table=default.Table_ca85cb69be1e4e709c000e5b77bcd3cf [id=a5ff0cf9c6ee49f39c35a5f7518bade4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.950572 27348 tablet_service.cc:1468] Processing CreateTablet for tablet 878f6ad087614c15bc7e29c784f2c3c9 (DEFAULT_TABLE table=default.Table_91ed95244dda4d37985c6af207e9e033 [id=5470fea324a94f1fba9da16adb5bef60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.953075 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet 33fb6b357bda43e7abeaed5634183ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:17.954275 27183 consensus_peers.cc:489] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 68cff0e416a54a5eb37071f5a12f07ab. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:17.954525 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet 60c32370a5274347afafc6a7e1e94270 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:17.956161 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet 44db427ee53a4d0a907dfd63282d6d4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:17.956516 27212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ca6e2165a9d41299c1fe0bfa9475915. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.957829 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 43fb27c857d543b7b78ea6108c406085 (DEFAULT_TABLE table=default.Table_4eafb1a74bd54d2f89763f3c3cadd0f7 [id=95e5cb3083814ab996fce9dc666c405e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.955255 27184 leader_election.cc:304] T 6ea37668d64d4feeb92a162711609c13 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:17.963609 27216 tablet_service.cc:1468] Processing CreateTablet for tablet 16549a6d9aa9432eb5cee19f69ce6957 (DEFAULT_TABLE table=default.Table_1eee3233aac9463982bf7a5c2f3e098b [id=2dfc9b9647ff4939b96f6759f3bd162b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.964023 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a686ea49a29a4dd7b6e5dce5e1f9935a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.964413 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43fb27c857d543b7b78ea6108c406085. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.964444 28030 raft_consensus.cc:2747] T 6ea37668d64d4feeb92a162711609c13 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:17.964583 27066 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 46e3a65118b744298b5536f88c22f1db (table default.Table_4258f924ddb3449bb5f49fb7a88a5073 [id=decb9fcf2ee64971ba8bcd8326736852]). Expected version 1 got 0
I20250903 20:05:17.965248 27692 tablet_bootstrap.cc:492] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:17.965260 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "691cb69759654fdc9223844049a06d02" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:17.965446 27692 ts_tablet_manager.cc:1397] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.102s user 0.001s sys 0.004s
I20250903 20:05:17.966624 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_dbc7009c4de44b8e88f3caa4b62ec665" } modify_external_catalogs: true
I20250903 20:05:17.966828 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_dbc7009c4de44b8e88f3caa4b62ec665" } modify_external_catalogs: true
I20250903 20:05:17.967161 27692 raft_consensus.cc:357] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:17.967432 27692 raft_consensus.cc:383] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:17.967522 27692 raft_consensus.cc:738] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:17.967727 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_aef24ecf95294a9188a4338dfedc525d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:17.968186 27692 consensus_queue.cc:260] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:17.968264 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_d1b7ba8a5d3440f7be2eb198bbf5e7fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:17.968483 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aef24ecf95294a9188a4338dfedc525d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:17.968739 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1b7ba8a5d3440f7be2eb198bbf5e7fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:17.968909 27692 ts_tablet_manager.cc:1428] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250903 20:05:17.971990 27692 tablet_bootstrap.cc:492] T eced730bdbc14a8c9f6b7b7e53435fd0 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:17.979853 27500 tablet_service.cc:1468] Processing CreateTablet for tablet 54602d7f723d4c10af12408a43ed1f02 (DEFAULT_TABLE table=default.Table_d4c00bc20db74e4c84da0f99bd594b19 [id=2eed1473230e4beab5701483fd3e4315]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.980032 27491 tablet_service.cc:1468] Processing CreateTablet for tablet b3d9491f777b422da4b9fca803beabff (DEFAULT_TABLE table=default.Table_93c4817ac1c64d589e71a4590d32adb2 [id=31352bcca5164b158205533584d1b352]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.980345 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54602d7f723d4c10af12408a43ed1f02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.980562 27497 tablet_service.cc:1468] Processing CreateTablet for tablet 5f2eefeebbfc47249e2f72b7c52e905d (DEFAULT_TABLE table=default.Table_359258cdcd734ef4b3463c80fa1c582a [id=7587e35155ff40be9d3f8810e208b591]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.980931 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f2eefeebbfc47249e2f72b7c52e905d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.982537 27492 tablet_service.cc:1468] Processing CreateTablet for tablet 7cbbeb8bec5e464b85b8568e916c765f (DEFAULT_TABLE table=default.Table_d37cdb880e224494969e67d03305b7fa [id=aaaf37ae7fe84ec9a4ebdd0ab1827589]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.982899 27492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cbbeb8bec5e464b85b8568e916c765f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.986591 27488 tablet_service.cc:1468] Processing CreateTablet for tablet 5132e6d4288c447e80f0608ba366b69c (DEFAULT_TABLE table=default.Table_2c769d77af8d429fabe07bdd9ab6c61c [id=3a3fd5d2b11246cb92787d0471494285]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.986994 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5132e6d4288c447e80f0608ba366b69c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.987052 27496 tablet_service.cc:1468] Processing CreateTablet for tablet 878f6ad087614c15bc7e29c784f2c3c9 (DEFAULT_TABLE table=default.Table_91ed95244dda4d37985c6af207e9e033 [id=5470fea324a94f1fba9da16adb5bef60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.987464 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 878f6ad087614c15bc7e29c784f2c3c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.987900 27490 tablet_service.cc:1468] Processing CreateTablet for tablet 112c0dccb60c4c7c9efc164d9c8443ea (DEFAULT_TABLE table=default.Table_4a511c041bed4d8b88479f001d13c791 [id=c91f93024bca4ad4a06d55cf35266df5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.988283 27490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 112c0dccb60c4c7c9efc164d9c8443ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.990504 27493 tablet_service.cc:1468] Processing CreateTablet for tablet 41056c5c5003415993bee74d7621bd14 (DEFAULT_TABLE table=default.Table_ca85cb69be1e4e709c000e5b77bcd3cf [id=a5ff0cf9c6ee49f39c35a5f7518bade4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:17.990406 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "691cb69759654fdc9223844049a06d02" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:17.990900 27527 raft_consensus.cc:3058] T 691cb69759654fdc9223844049a06d02 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:17.990940 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41056c5c5003415993bee74d7621bd14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.993968 27527 raft_consensus.cc:2466] T 691cb69759654fdc9223844049a06d02 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
W20250903 20:05:17.994757 27320 consensus_peers.cc:489] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:17.995714 27491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3d9491f777b422da4b9fca803beabff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:17.998435 27320 leader_election.cc:304] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:17.999053 28132 raft_consensus.cc:2802] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:17.999217 28132 raft_consensus.cc:695] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:17.999466 28132 consensus_queue.cc:237] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.004861 28030 raft_consensus.cc:491] T deb84fc3465443d99c11097e53f5f575 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:18.005643 27248 raft_consensus.cc:3058] T 691cb69759654fdc9223844049a06d02 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:18.007593 27226 tablet_service.cc:1468] Processing CreateTablet for tablet ef02f91c0964446daf62ac1dde0330ff (DEFAULT_TABLE table=default.Table_ff2b25e34cab4943949d8b2d514bd98f [id=b3135f352fb24246be537661e413991e]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:18.008183 27184 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 103788us
W20250903 20:05:18.008543 27184 leader_election.cc:343] T fdc77810f27a40758e063b8b1150ab71 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:18.008719 27184 leader_election.cc:304] T fdc77810f27a40758e063b8b1150ab71 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:18.009289 28181 raft_consensus.cc:2747] T fdc77810f27a40758e063b8b1150ab71 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:18.009843 28030 raft_consensus.cc:513] T deb84fc3465443d99c11097e53f5f575 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.010802 28030 leader_election.cc:290] T deb84fc3465443d99c11097e53f5f575 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:18.014863 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "deb84fc3465443d99c11097e53f5f575" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:18.019469 27216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16549a6d9aa9432eb5cee19f69ce6957. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.031111 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_64dabe496cb94cfb9061d0d7eefcbd3a" } modify_external_catalogs: true
I20250903 20:05:18.031354 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_64dabe496cb94cfb9061d0d7eefcbd3a" } modify_external_catalogs: true
I20250903 20:05:18.032789 27065 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_367e5dbc110d47b38f43d9dfdcfce307" } modify_external_catalogs: true
I20250903 20:05:18.032970 27065 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_367e5dbc110d47b38f43d9dfdcfce307" } modify_external_catalogs: true
I20250903 20:05:18.035216 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet c9c870fb14e3469fb2979d8462b49121 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:18.037357 27064 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_34c69a470a8a4e9eab936b4c856e4eb4" } modify_external_catalogs: true
I20250903 20:05:18.037528 27064 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_34c69a470a8a4e9eab936b4c856e4eb4" } modify_external_catalogs: true
I20250903 20:05:18.039428 28132 consensus_queue.cc:1035] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:18.039788 27210 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a223f75045345ae833f52bcc8f239f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.042400 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_dacd8238c8334c1c9f66e745f5036a06" } modify_external_catalogs: true
I20250903 20:05:18.042596 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_dacd8238c8334c1c9f66e745f5036a06" } modify_external_catalogs: true
I20250903 20:05:18.047536 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef02f91c0964446daf62ac1dde0330ff. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:18.048066 27184 leader_election.cc:343] T c9129a3e773a4fecb28ceee27778a294 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:18.048611 27210 tablet_service.cc:1515] Processing DeleteTablet for tablet a3103e1595ec4c38ba65dca26fe1167c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.051971 27693 tablet_bootstrap.cc:492] T 44db427ee53a4d0a907dfd63282d6d4f P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:18.052142 27693 ts_tablet_manager.cc:1397] T 44db427ee53a4d0a907dfd63282d6d4f P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.244s user 0.024s sys 0.046s
I20250903 20:05:18.052371 27229 tablet_service.cc:1468] Processing CreateTablet for tablet 8400e23873fd4be18de2998afc64a689 (DEFAULT_TABLE table=default.Table_d43fc55de38641f6946c7dab57874b91 [id=6103a53de91b4ad19b8f7afd7b3fd881]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.052989 27693 raft_consensus.cc:357] T 44db427ee53a4d0a907dfd63282d6d4f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.053179 27693 raft_consensus.cc:383] T 44db427ee53a4d0a907dfd63282d6d4f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:18.053256 27693 raft_consensus.cc:738] T 44db427ee53a4d0a907dfd63282d6d4f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:18.053483 27693 consensus_queue.cc:260] T 44db427ee53a4d0a907dfd63282d6d4f P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.053982 27693 ts_tablet_manager.cc:1428] T 44db427ee53a4d0a907dfd63282d6d4f P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:18.054332 27693 tablet_bootstrap.cc:492] T 73f65e7515994b898eec2ea23903aff4 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:18.054555 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_00f2d88f1c284af3b6fbcaf8590938d5" } new_table_name: "default.Table_a7a9140280274ad2ac9a71b678fc8ca7" modify_external_catalogs: true
I20250903 20:05:18.056098 27693 tablet_bootstrap.cc:654] T 73f65e7515994b898eec2ea23903aff4 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:18.058138 27491 tablet_service.cc:1515] Processing DeleteTablet for tablet c9c870fb14e3469fb2979d8462b49121 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:18.058837 27353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41056c5c5003415993bee74d7621bd14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.059530 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 878f6ad087614c15bc7e29c784f2c3c9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:18.060843 27183 leader_election.cc:343] T deb84fc3465443d99c11097e53f5f575 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:18.061020 27183 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 45259.2us
I20250903 20:05:18.072846 28169 raft_consensus.cc:491] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:18.073064 28169 raft_consensus.cc:513] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.073876 28169 leader_election.cc:290] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:18.078697 27184 leader_election.cc:304] T c9129a3e773a4fecb28ceee27778a294 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:18.079727 27462 leader_election.cc:343] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: c9c870fb14e3469fb2979d8462b49121
I20250903 20:05:18.080155 27384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9c870fb14e3469fb2979d8462b49121" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:18.080428 27384 raft_consensus.cc:2466] T c9c870fb14e3469fb2979d8462b49121 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:18.081400 27227 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f8cfe53b91b41d7ba7f9074e1005af3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.082711 27248 raft_consensus.cc:2466] T 691cb69759654fdc9223844049a06d02 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:18.083084 27462 leader_election.cc:304] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:18.083422 27693 tablet_bootstrap.cc:492] T 73f65e7515994b898eec2ea23903aff4 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:18.083602 27693 ts_tablet_manager.cc:1397] T 73f65e7515994b898eec2ea23903aff4 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.012s
I20250903 20:05:18.084501 27693 raft_consensus.cc:357] T 73f65e7515994b898eec2ea23903aff4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:18.084749 27693 raft_consensus.cc:383] T 73f65e7515994b898eec2ea23903aff4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:18.084823 27693 raft_consensus.cc:738] T 73f65e7515994b898eec2ea23903aff4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:18.085024 28169 raft_consensus.cc:2802] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:18.085146 28169 raft_consensus.cc:491] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:18.085268 28169 raft_consensus.cc:3058] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:18.085198 27693 consensus_queue.cc:260] T 73f65e7515994b898eec2ea23903aff4 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:18.085732 27693 ts_tablet_manager.cc:1428] T 73f65e7515994b898eec2ea23903aff4 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:18.086071 27693 tablet_bootstrap.cc:492] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:18.086192 27687 ts_tablet_manager.cc:1918] T 4268316c28e8471a9f28807d20dbb395 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:18.086333 27687 log.cc:1199] T 4268316c28e8471a9f28807d20dbb395 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/4268316c28e8471a9f28807d20dbb395
I20250903 20:05:18.086735 27687 ts_tablet_manager.cc:1939] T 4268316c28e8471a9f28807d20dbb395 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:18.087852 27693 tablet_bootstrap.cc:654] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:18.088140 28169 raft_consensus.cc:513] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.091552 27692 tablet_bootstrap.cc:654] T eced730bdbc14a8c9f6b7b7e53435fd0 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:18.091583 27384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9c870fb14e3469fb2979d8462b49121" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:18.091809 27384 raft_consensus.cc:3058] T c9c870fb14e3469fb2979d8462b49121 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:18.092056 28169 leader_election.cc:290] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:18.094409 28181 raft_consensus.cc:2747] T c9129a3e773a4fecb28ceee27778a294 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:18.094540 27384 raft_consensus.cc:2466] T c9c870fb14e3469fb2979d8462b49121 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:18.096493 27462 leader_election.cc:304] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:18.097261 27208 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:56514 (request call id 95) took 2176 ms. Trace:
I20250903 20:05:18.097712 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet ed4902c002ef4198802375ced39ef2ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.097460 27684 ts_tablet_manager.cc:1918] T 2a5c74a281b44019872a6442d41bc373 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:18.097911 27684 log.cc:1199] T 2a5c74a281b44019872a6442d41bc373 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/2a5c74a281b44019872a6442d41bc373
I20250903 20:05:18.098423 27684 ts_tablet_manager.cc:1939] T 2a5c74a281b44019872a6442d41bc373 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:18.098609 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 8e12f4c420874ab382d433982bddbf05 (DEFAULT_TABLE table=default.Table_42f95b6c82f949a082559298f94e747c [id=430a979602e1430eaab642507035c4f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.102568 27219 tablet_service.cc:1468] Processing CreateTablet for tablet f23f90aac8bc49889b1b882df186ef50 (DEFAULT_TABLE table=default.Table_da1b078cb62f476c8f2c2454db2713d2 [id=26058718e4174971ae0d8d74a6963893]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:18.104430 27057 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 548b50819f694fa3ad508da9f442a99c consensus state has no leader
W20250903 20:05:18.104620 27184 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 73458.8us
I20250903 20:05:18.104842 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 36e1061915574407b6e7695be8b1e55a (DEFAULT_TABLE table=default.Table_3716b922d00f49ff9b1b7cef13737daf [id=836e087e0536485492f7ee6784fabe76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.105279 27210 tablet_service.cc:1515] Processing DeleteTablet for tablet e877d56bbbfa4c5a92d025d0311ce5c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.111356 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 44db427ee53a4d0a907dfd63282d6d4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:18.115257 27208 rpcz_store.cc:276] 0903 20:05:15.905181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.905676 (+ 495us) service_pool.cc:225] Handling call
0903 20:05:16.898155 (+992479us) spinlock_profiling.cc:243] Waited 57 ms on lock 0x6160000d3954. stack: 00007f589abe0dc6 00007f589abe0b80 00007f5899bc83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f58a6639700 00007f58aa3d371f 00007f58a4b04c36 00007f58a4b04965 00007f58a19910be 00007f58a198ff9e 00007f58a19961d5 00007f58a1996aff 00007f58a1996928 0000000000a774dc
0903 20:05:17.402489 (+504334us) spinlock_profiling.cc:243] Waited 502 ms on lock 0x6160000d3354. stack: 00007f589abe0dc6 00007f589abe0b80 00007f5899bc83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f58a6639700 00007f58aa3d371f 00007f58a4b04c36 00007f58a4b04965 00007f58a19910be 00007f58a198ff9e 00007f58a19961d5 00007f58a1996aff 00007f58a1996928 0000000000a774dc
0903 20:05:17.635918 (+233429us) spinlock_profiling.cc:243] Waited 3.11 ms on lock 0x6160000c8630. stack: 00007f589abe0dc6 00007f589abe0b80 00007f5899bc83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f58a7987859 00007f58aa3d3528 00007f58a4b04c36 00007f58a4b04965 00007f58a19910be 00007f58a198ff9e 00007f58a19961d5 00007f58a1996aff 00007f58a1996928 0000000000a774dc
0903 20:05:17.795080 (+159162us) spinlock_profiling.cc:243] Waited 27 ms on lock 0x6160000c4054. stack: 00007f589abe0dc6 00007f589abe0b80 00007f5899bc83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f58a6639700 00007f58aa3d371f 00007f58a4b04c36 00007f58a4b04965 00007f58a19910be 00007f58a198ff9e 00007f58a19961d5 00007f58a1996aff 00007f58a1996928 0000000000a774dc
0903 20:05:18.081935 (+286855us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1355527168}
I20250903 20:05:18.111704 27227 tablet_service.cc:1468] Processing CreateTablet for tablet 8218868ff40b405387d560f614427c08 (DEFAULT_TABLE table=default.Table_91ae8b84b5b94998a81cf961b29bf4a4 [id=782821cd80cf493381e55730ebb5fefe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.116430 28132 raft_consensus.cc:491] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:18.116679 28132 raft_consensus.cc:513] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.117578 28132 leader_election.cc:290] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:18.117321 27220 tablet_service.cc:1468] Processing CreateTablet for tablet bc98762790f441038972366b1cc2c55b (DEFAULT_TABLE table=default.Table_24fa04bd95714589b18eb17010dbcec2 [id=bb35d1946b5b44afa5329d15a3866162]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.119228 27695 tablet_bootstrap.cc:492] T 4c543ac4df33429b9787c6d32bf1ebeb P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:18.119385 27695 ts_tablet_manager.cc:1397] T 4c543ac4df33429b9787c6d32bf1ebeb P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.509s user 0.018s sys 0.065s
I20250903 20:05:18.120468 27229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8400e23873fd4be18de2998afc64a689. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:18.123103 27184 leader_election.cc:343] T 5bc3b9771a0042fd87e706204dcbb712 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:18.123261 27184 leader_election.cc:304] T 5bc3b9771a0042fd87e706204dcbb712 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:18.125454 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e1061915574407b6e7695be8b1e55a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.125373 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 433) took 1666 ms. Trace:
I20250903 20:05:18.125576 27687 rpcz_store.cc:276] 0903 20:05:16.458663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:16.761124 (+302461us) service_pool.cc:225] Handling call
0903 20:05:17.794055 (+1032931us) ts_tablet_manager.cc:1145] Deleting tablet 4268316c28e8471a9f28807d20dbb395
0903 20:05:17.794065 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.932299 (+138234us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:18.086044 (+153745us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:18.125306 (+ 39262us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":980286}
I20250903 20:05:18.126179 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 4268316c28e8471a9f28807d20dbb395 (table default.Table_6ab02d2c339e463fbed5c3429c7c4ec9 [id=f70d7ea93e154e6199f6766316353862]) successfully deleted
I20250903 20:05:18.126782 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 443) took 1529 ms. Trace:
I20250903 20:05:18.126943 27687 rpcz_store.cc:276] 0903 20:05:16.596989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:16.882687 (+285698us) service_pool.cc:225] Handling call
0903 20:05:18.126558 (+1243871us) ts_tablet_manager.cc:1145] Deleting tablet e0347eab6c8642beabdc2d7e7bbfdc9b
0903 20:05:18.126576 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:18.126731 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1235610}
W20250903 20:05:18.127579 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet e0347eab6c8642beabdc2d7e7bbfdc9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0347eab6c8642beabdc2d7e7bbfdc9b already in progress: creating tablet
I20250903 20:05:18.128111 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 444) took 1531 ms. Trace:
I20250903 20:05:18.128383 27687 rpcz_store.cc:276] 0903 20:05:16.597067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:16.889455 (+292388us) service_pool.cc:225] Handling call
0903 20:05:18.127904 (+1238449us) ts_tablet_manager.cc:1145] Deleting tablet ee40566e497740789575275d0cdca0db
0903 20:05:18.127918 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:18.128034 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1236125}
W20250903 20:05:18.128924 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet ee40566e497740789575275d0cdca0db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee40566e497740789575275d0cdca0db already in progress: creating tablet
I20250903 20:05:18.129360 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 446) took 1532 ms. Trace:
I20250903 20:05:18.129359 27695 raft_consensus.cc:357] T 4c543ac4df33429b9787c6d32bf1ebeb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:18.129593 27695 raft_consensus.cc:383] T 4c543ac4df33429b9787c6d32bf1ebeb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:18.129511 27687 rpcz_store.cc:276] 0903 20:05:16.597261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:16.891791 (+294530us) service_pool.cc:225] Handling call
0903 20:05:18.129189 (+1237398us) ts_tablet_manager.cc:1145] Deleting tablet 60c32370a5274347afafc6a7e1e94270
0903 20:05:18.129204 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:18.129312 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1237243}
W20250903 20:05:18.130028 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 60c32370a5274347afafc6a7e1e94270 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60c32370a5274347afafc6a7e1e94270 already in progress: creating tablet
I20250903 20:05:18.131907 27695 raft_consensus.cc:738] T 4c543ac4df33429b9787c6d32bf1ebeb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:18.132297 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc98762790f441038972366b1cc2c55b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:18.135602 27184 leader_election.cc:343] T 1fef9400227449ed892daff588999caa P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:18.137152 28181 raft_consensus.cc:2747] T 5bc3b9771a0042fd87e706204dcbb712 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:18.137396 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04ccb3e4a8bd4789b9f19edd4a5b6145" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:18.137991 28181 raft_consensus.cc:491] T ecd339ec12e84e88acec427630cc8668 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:18.138311 27695 consensus_queue.cc:260] T 4c543ac4df33429b9787c6d32bf1ebeb P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:18.141747 27184 leader_election.cc:304] T 1fef9400227449ed892daff588999caa P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:18.143353 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 442) took 1546 ms. Trace:
I20250903 20:05:18.143507 27687 rpcz_store.cc:276] 0903 20:05:16.596891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:16.874559 (+277668us) service_pool.cc:225] Handling call
0903 20:05:18.143134 (+1268575us) ts_tablet_manager.cc:1145] Deleting tablet 5fb2426aa01e47598f268d0a90475271
0903 20:05:18.143147 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:18.143296 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1249081}
I20250903 20:05:18.143934 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 445) took 1546 ms. Trace:
I20250903 20:05:18.144047 27687 rpcz_store.cc:276] 0903 20:05:16.597155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:16.890971 (+293816us) service_pool.cc:225] Handling call
0903 20:05:18.143785 (+1252814us) ts_tablet_manager.cc:1145] Deleting tablet 33fb6b357bda43e7abeaed5634183ef6
0903 20:05:18.143796 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:18.143899 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1232646}
W20250903 20:05:18.144371 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 5fb2426aa01e47598f268d0a90475271 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5fb2426aa01e47598f268d0a90475271 already in progress: creating tablet
I20250903 20:05:18.144443 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 451) took 1099 ms. Trace:
I20250903 20:05:18.144546 27687 rpcz_store.cc:276] 0903 20:05:17.045005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.045074 (+ 69us) service_pool.cc:225] Handling call
0903 20:05:18.144316 (+1099242us) ts_tablet_manager.cc:1145] Deleting tablet 6f8cfe53b91b41d7ba7f9074e1005af3
0903 20:05:18.144324 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:18.144404 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1098974}
I20250903 20:05:18.144945 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 452) took 1098 ms. Trace:
I20250903 20:05:18.145057 27687 rpcz_store.cc:276] 0903 20:05:17.046704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.046764 (+ 60us) service_pool.cc:225] Handling call
0903 20:05:18.144755 (+1097991us) ts_tablet_manager.cc:1145] Deleting tablet a3103e1595ec4c38ba65dca26fe1167c
0903 20:05:18.144764 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:18.144896 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1097725}
I20250903 20:05:18.145438 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 453) took 1097 ms. Trace:
I20250903 20:05:18.145542 27687 rpcz_store.cc:276] 0903 20:05:17.048339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.048398 (+ 59us) service_pool.cc:225] Handling call
0903 20:05:18.145263 (+1096865us) ts_tablet_manager.cc:1145] Deleting tablet ed4902c002ef4198802375ced39ef2ad
0903 20:05:18.145270 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:18.145397 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1096645}
W20250903 20:05:18.145908 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 33fb6b357bda43e7abeaed5634183ef6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33fb6b357bda43e7abeaed5634183ef6 already in progress: creating tablet
W20250903 20:05:18.107810 27462 leader_election.cc:343] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: c9c870fb14e3469fb2979d8462b49121
I20250903 20:05:18.111338 28169 raft_consensus.cc:2802] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
W20250903 20:05:18.146183 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 6f8cfe53b91b41d7ba7f9074e1005af3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f8cfe53b91b41d7ba7f9074e1005af3 already in progress: creating tablet
I20250903 20:05:18.146359 28169 raft_consensus.cc:695] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
W20250903 20:05:18.146448 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet a3103e1595ec4c38ba65dca26fe1167c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3103e1595ec4c38ba65dca26fe1167c already in progress: creating tablet
I20250903 20:05:18.146651 28169 consensus_queue.cc:237] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:18.147066 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet ed4902c002ef4198802375ced39ef2ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed4902c002ef4198802375ced39ef2ad already in progress: creating tablet
W20250903 20:05:18.150175 27320 leader_election.cc:343] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 04ccb3e4a8bd4789b9f19edd4a5b6145
I20250903 20:05:18.150370 27320 leader_election.cc:304] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:18.150784 27693 tablet_bootstrap.cc:492] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:18.151129 28132 raft_consensus.cc:2747] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:18.151144 27693 ts_tablet_manager.cc:1397] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.065s user 0.016s sys 0.023s
I20250903 20:05:18.153278 27684 tablet_replica.cc:331] T 691cb69759654fdc9223844049a06d02 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:18.153494 27684 raft_consensus.cc:2241] T 691cb69759654fdc9223844049a06d02 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:18.153676 27684 raft_consensus.cc:2270] T 691cb69759654fdc9223844049a06d02 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:18.157101 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 450) took 1102 ms. Trace:
I20250903 20:05:18.157413 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet e2441aa3c2c748bba5360259fb663c81
I20250903 20:05:18.158389 27684 ts_tablet_manager.cc:1905] T 691cb69759654fdc9223844049a06d02 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:18.138155 28181 raft_consensus.cc:513] T ecd339ec12e84e88acec427630cc8668 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.159557 28181 leader_election.cc:290] T ecd339ec12e84e88acec427630cc8668 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:18.159763 27692 tablet_bootstrap.cc:492] T eced730bdbc14a8c9f6b7b7e53435fd0 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:18.159932 27692 ts_tablet_manager.cc:1397] T eced730bdbc14a8c9f6b7b7e53435fd0 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.188s user 0.000s sys 0.004s
I20250903 20:05:18.160290 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecd339ec12e84e88acec427630cc8668" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:18.161048 27184 leader_election.cc:343] T 8b3121fce1c44d16a0b083313dc67b7a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:18.161201 27184 leader_election.cc:304] T 8b3121fce1c44d16a0b083313dc67b7a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:18.163966 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 2a5c74a281b44019872a6442d41bc373 (table default.Table_5c08431ef8e04a6aab9b96ab78615d2f [id=a9ddfed19dab46a7bb4e049483e3996f]) successfully deleted
W20250903 20:05:18.164830 27051 catalog_manager.cc:4895] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 68cff0e416a54a5eb37071f5a12f07ab because the tablet was not found. No further retry: Not found: Tablet not found: 68cff0e416a54a5eb37071f5a12f07ab
I20250903 20:05:18.165160 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8218868ff40b405387d560f614427c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.168599 28181 raft_consensus.cc:2747] T 8b3121fce1c44d16a0b083313dc67b7a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:18.172426 27183 leader_election.cc:343] T ecd339ec12e84e88acec427630cc8668 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:18.172786 27219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f23f90aac8bc49889b1b882df186ef50. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.173256 27692 raft_consensus.cc:357] T eced730bdbc14a8c9f6b7b7e53435fd0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.173511 27692 raft_consensus.cc:383] T eced730bdbc14a8c9f6b7b7e53435fd0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:18.173621 27692 raft_consensus.cc:738] T eced730bdbc14a8c9f6b7b7e53435fd0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:18.174397 27692 consensus_queue.cc:260] T eced730bdbc14a8c9f6b7b7e53435fd0 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.176515 27692 ts_tablet_manager.cc:1428] T eced730bdbc14a8c9f6b7b7e53435fd0 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250903 20:05:18.176789 27692 tablet_bootstrap.cc:492] T e8bc99de4ffe43d58351d170b7c61d8a P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:18.179973 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet e2441aa3c2c748bba5360259fb663c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:18.183687 27695 ts_tablet_manager.cc:1428] T 4c543ac4df33429b9787c6d32bf1ebeb P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20250903 20:05:18.184042 28030 raft_consensus.cc:2747] T 1fef9400227449ed892daff588999caa P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:18.184899 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e12f4c420874ab382d433982bddbf05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.189814 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_3bcc60f1d7ed42d89cfaf71c0fc18fdd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:18.190096 27057 master_service.cc:948] ReplaceTablet: received request to replace tablet aa9f5e0964b348b7af184f60cd145666 from {username='slave'} at 127.0.0.1:45936
W20250903 20:05:18.190587 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3bcc60f1d7ed42d89cfaf71c0fc18fdd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:18.191864 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5ca9587a0da045e8a6baf89e4ee8b537" } new_table_name: "default.Table_4126233ea8a445a8837131a6052aca70" modify_external_catalogs: true
I20250903 20:05:18.191792 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_10c65228c77a4a9d8ccc535d2792f59f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:18.192384 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10c65228c77a4a9d8ccc535d2792f59f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:18.195827 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3716b922d00f49ff9b1b7cef13737daf" } new_table_name: "default.Table_116e90ba6d134656a9b891dd8b28d5e3" modify_external_catalogs: true
I20250903 20:05:18.199143 27065 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet babdb5ae26cc40b9bef95c2cfe8791a8
I20250903 20:05:18.201848 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e79d289ca0384e109873b8d5f2c78905"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:18.202378 27695 tablet_bootstrap.cc:492] T 3f78a951bb024cf290116b0fe438ddb4 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
W20250903 20:05:18.202570 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e79d289ca0384e109873b8d5f2c78905 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:18.209120 28201 tablet.cc:1722] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:18.212745 27064 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet a29012df93144097b5b93661ae664dba
I20250903 20:05:18.216154 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_edfc268f076142a2b7c2ef149edf7f5c" } new_table_name: "default.Table_39de38c00cdb44eb85fdb76134b7139c" modify_external_catalogs: true
I20250903 20:05:18.216621 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet eced730bdbc14a8c9f6b7b7e53435fd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:18.218565 27684 ts_tablet_manager.cc:1918] T 691cb69759654fdc9223844049a06d02 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:18.222107 27228 tablet_service.cc:1515] Processing DeleteTablet for tablet 2a5c74a281b44019872a6442d41bc373 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:17 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.224670 27684 log.cc:1199] T 691cb69759654fdc9223844049a06d02 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/691cb69759654fdc9223844049a06d02
I20250903 20:05:18.225036 28030 raft_consensus.cc:491] T 4c543ac4df33429b9787c6d32bf1ebeb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:18.225651 27684 ts_tablet_manager.cc:1939] T 691cb69759654fdc9223844049a06d02 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:18.229293 27693 raft_consensus.cc:357] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.229622 27693 raft_consensus.cc:383] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:18.184732 27692 tablet_bootstrap.cc:654] T e8bc99de4ffe43d58351d170b7c61d8a P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:18.229749 27693 raft_consensus.cc:738] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:18.230003 27693 consensus_queue.cc:260] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.230747 27693 ts_tablet_manager.cc:1428] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.079s user 0.003s sys 0.000s
I20250903 20:05:18.231130 27693 tablet_bootstrap.cc:492] T eced730bdbc14a8c9f6b7b7e53435fd0 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:18.231448 27228 tablet_service.cc:1515] Processing DeleteTablet for tablet deb84fc3465443d99c11097e53f5f575 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.233448 27693 tablet_bootstrap.cc:654] T eced730bdbc14a8c9f6b7b7e53435fd0 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:18.234077 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet e2441aa3c2c748bba5360259fb663c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:18.234890 27684 tablet_replica.cc:331] T 73f65e7515994b898eec2ea23903aff4 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:18.235121 27684 raft_consensus.cc:2241] T 73f65e7515994b898eec2ea23903aff4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:18.235289 27684 raft_consensus.cc:2270] T 73f65e7515994b898eec2ea23903aff4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:18.235651 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 691cb69759654fdc9223844049a06d02 (table default.Table_64c39ce0250747edacd69ba845d72455 [id=dbfd528fcaba4b519108e3c4a5a5c36f]) successfully deleted
I20250903 20:05:18.235877 27684 ts_tablet_manager.cc:1905] T 73f65e7515994b898eec2ea23903aff4 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:18.237633 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 73f65e7515994b898eec2ea23903aff4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:18.238351 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 847d0a0a3d2b45938cec9058c77f6231 (DEFAULT_TABLE table=default.Table_196cfa99ce59433b85ad81b45d853081 [id=88d56563f5dc4fecb59c106ff78499e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.238656 27223 tablet_service.cc:1515] Processing DeleteTablet for tablet 1fef9400227449ed892daff588999caa with delete_type TABLET_DATA_DELETED (replaced by tablet 67ca1dcf6fc7435db555392235f9900f) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.240806 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "deb84fc3465443d99c11097e53f5f575" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:18.240859 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eced730bdbc14a8c9f6b7b7e53435fd0" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:18.243106 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet fa05f54ebc324b40855fa3288760e9e9
I20250903 20:05:18.243212 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecd339ec12e84e88acec427630cc8668" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:18.245975 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a686ea49a29a4dd7b6e5dce5e1f9935a consensus state has no leader
I20250903 20:05:18.247447 27062 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet e8bc99de4ffe43d58351d170b7c61d8a (table default.Table_c5256f043b61433b8debe9e13625bd0d [id=416ff29f66d14564826a6734dde1c604]). Expected version 2 got 0
I20250903 20:05:18.248152 27062 master_service.cc:948] ReplaceTablet: received request to replace tablet a4f3230f50ec4c95b528fead8c191224 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:18.248081 27687 rpcz_store.cc:276] 0903 20:05:17.043631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.055839 (+ 12208us) service_pool.cc:225] Handling call
0903 20:05:18.145757 (+1089918us) ts_tablet_manager.cc:1145] Deleting tablet 8a223f75045345ae833f52bcc8f239f9
0903 20:05:18.145766 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:18.145841 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1089675}
I20250903 20:05:18.248708 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_4d67ac5e5c6645f69648aea15fa389ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:18.249290 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d67ac5e5c6645f69648aea15fa389ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:18.250764 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet aa9f5e0964b348b7af184f60cd145666
I20250903 20:05:18.251209 27057 catalog_manager.cc:6652] ReplaceTablet: tablet aa9f5e0964b348b7af184f60cd145666 deleted and replaced by tablet 26decef5841a4a3b85f2a5f578ae95dc
I20250903 20:05:18.251528 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e942cce7cfd5471dbbca333e31f4ba66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:18.252060 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 454) took 1181 ms. Trace:
W20250903 20:05:18.252080 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e942cce7cfd5471dbbca333e31f4ba66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:18.252257 27687 rpcz_store.cc:276] 0903 20:05:17.071027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.073758 (+ 2731us) service_pool.cc:225] Handling call
0903 20:05:18.248804 (+1175046us) ts_tablet_manager.cc:1145] Deleting tablet e877d56bbbfa4c5a92d025d0311ce5c6
0903 20:05:18.248818 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:18.252015 (+ 3197us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1174712}
I20250903 20:05:18.252848 27687 tablet_replica.cc:331] T 79a0553c912049cca5971b1ad6f7e11c P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:18.253464 27061 master_service.cc:948] ReplaceTablet: received request to replace tablet 63a1f26fac3b4b939ebb79c6a07d8f8e from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:18.253784 27684 ts_tablet_manager.cc:1918] T 73f65e7515994b898eec2ea23903aff4 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:18.254012 27684 log.cc:1199] T 73f65e7515994b898eec2ea23903aff4 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/73f65e7515994b898eec2ea23903aff4
I20250903 20:05:18.254583 27684 ts_tablet_manager.cc:1939] T 73f65e7515994b898eec2ea23903aff4 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
W20250903 20:05:18.255504 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 8a223f75045345ae833f52bcc8f239f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a223f75045345ae833f52bcc8f239f9 already in progress: creating tablet
W20250903 20:05:18.255767 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet e877d56bbbfa4c5a92d025d0311ce5c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e877d56bbbfa4c5a92d025d0311ce5c6 already in progress: creating tablet
I20250903 20:05:18.256297 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet fa05f54ebc324b40855fa3288760e9e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:18.256492 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet aa9f5e0964b348b7af184f60cd145666 with delete_type TABLET_DATA_DELETED (replaced by tablet 26decef5841a4a3b85f2a5f578ae95dc) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:18.256842 27687 raft_consensus.cc:2241] T 79a0553c912049cca5971b1ad6f7e11c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:18.257061 27687 raft_consensus.cc:2270] T 79a0553c912049cca5971b1ad6f7e11c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:18.257726 27687 ts_tablet_manager.cc:1905] T 79a0553c912049cca5971b1ad6f7e11c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:18.257897 28030 raft_consensus.cc:513] T 4c543ac4df33429b9787c6d32bf1ebeb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:18.259287 27684 tablet_replica.cc:331] T 44db427ee53a4d0a907dfd63282d6d4f P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:18.259220 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 6d21f38759d743af972635f8b5b8d8d2 (DEFAULT_TABLE table=default.Table_e17243085e464908adf84082adbb3800 [id=2f0e4907f017416bb5790e27fc79aeb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.259559 27684 raft_consensus.cc:2241] T 44db427ee53a4d0a907dfd63282d6d4f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:18.260412 27221 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d13884fff8c43b690836c6a341a2d92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:17 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.260699 27684 raft_consensus.cc:2270] T 44db427ee53a4d0a907dfd63282d6d4f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:18.262499 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet fa05f54ebc324b40855fa3288760e9e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:18.262923 27223 tablet_service.cc:1468] Processing CreateTablet for tablet 05e949476a834762973cb98923420f69 (DEFAULT_TABLE table=default.Table_08b8a5b3a94c4232b361faecf839c494 [id=7ba2fae44f1340ab9dd9f4deb586ca76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.263083 27684 ts_tablet_manager.cc:1905] T 44db427ee53a4d0a907dfd63282d6d4f P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:18.264014 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet aa9f5e0964b348b7af184f60cd145666 with delete_type TABLET_DATA_DELETED (replaced by tablet 26decef5841a4a3b85f2a5f578ae95dc) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:18.264549 27221 tablet_service.cc:1515] Processing DeleteTablet for tablet f9709dfd18654c5bb071e39da7c8889f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:17 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.268088 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c543ac4df33429b9787c6d32bf1ebeb" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:18.269073 27183 leader_election.cc:343] T 4c543ac4df33429b9787c6d32bf1ebeb P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:18.269948 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 5f2eefeebbfc47249e2f72b7c52e905d (DEFAULT_TABLE table=default.Table_359258cdcd734ef4b3463c80fa1c582a [id=7587e35155ff40be9d3f8810e208b591]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.270447 27224 tablet_service.cc:1515] Processing DeleteTablet for tablet 541278ca9ffb4e41b341139c2faa337f with delete_type TABLET_DATA_DELETED (replaced by tablet e9126e4a18c546c28927f22c9f16426e) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.271185 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 73f65e7515994b898eec2ea23903aff4 (table default.Table_d5626bea3c88471b846f37ae4ed79715 [id=75e5cb0bab9b46a6b5caec7c99efc7b8]) successfully deleted
W20250903 20:05:18.271449 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 6d13884fff8c43b690836c6a341a2d92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d13884fff8c43b690836c6a341a2d92 already in progress: creating tablet
I20250903 20:05:18.271540 27228 tablet_service.cc:1468] Processing CreateTablet for tablet b84f487773c24735a61d0fd0898b2bcb (DEFAULT_TABLE table=default.Table_2cedbabdccbb444789b77ff293edf019 [id=cdf58c7117de4317b34a57666ad17d85]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:18.271656 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 8218868ff40b405387d560f614427c08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8218868ff40b405387d560f614427c08 already in progress: creating tablet
W20250903 20:05:18.271857 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet f9709dfd18654c5bb071e39da7c8889f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9709dfd18654c5bb071e39da7c8889f already in progress: creating tablet
I20250903 20:05:18.271857 27210 tablet_service.cc:1515] Processing DeleteTablet for tablet b84f487773c24735a61d0fd0898b2bcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:17 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:18.272081 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 541278ca9ffb4e41b341139c2faa337f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 541278ca9ffb4e41b341139c2faa337f already in progress: creating tablet
W20250903 20:05:18.272302 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet b84f487773c24735a61d0fd0898b2bcb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b84f487773c24735a61d0fd0898b2bcb already in progress: creating tablet
I20250903 20:05:18.273394 28030 leader_election.cc:290] T 4c543ac4df33429b9787c6d32bf1ebeb P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:18.278383 27221 tablet_service.cc:1515] Processing DeleteTablet for tablet ecd339ec12e84e88acec427630cc8668 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.278813 27221 tablet_service.cc:1468] Processing CreateTablet for tablet 112c0dccb60c4c7c9efc164d9c8443ea (DEFAULT_TABLE table=default.Table_4a511c041bed4d8b88479f001d13c791 [id=c91f93024bca4ad4a06d55cf35266df5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.279781 27684 ts_tablet_manager.cc:1918] T 44db427ee53a4d0a907dfd63282d6d4f P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:18.279987 27684 log.cc:1199] T 44db427ee53a4d0a907dfd63282d6d4f P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/44db427ee53a4d0a907dfd63282d6d4f
I20250903 20:05:18.280454 27684 ts_tablet_manager.cc:1939] T 44db427ee53a4d0a907dfd63282d6d4f P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:18.280406 27217 tablet_service.cc:1468] Processing CreateTablet for tablet b3d9491f777b422da4b9fca803beabff (DEFAULT_TABLE table=default.Table_93c4817ac1c64d589e71a4590d32adb2 [id=31352bcca5164b158205533584d1b352]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.281242 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d21f38759d743af972635f8b5b8d8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.282111 27210 tablet_service.cc:1468] Processing CreateTablet for tablet 54602d7f723d4c10af12408a43ed1f02 (DEFAULT_TABLE table=default.Table_d4c00bc20db74e4c84da0f99bd594b19 [id=2eed1473230e4beab5701483fd3e4315]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:18.283766 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a4c582b441f6467c99ba0b090abcf4cc consensus state has no leader
I20250903 20:05:18.284127 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_c2bb0708a3bc47a3a1a3dbbae0964cd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:18.284175 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_9abad42b86c747b68c5d9ef37d8d2788"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:18.284724 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2bb0708a3bc47a3a1a3dbbae0964cd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:18.284793 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9abad42b86c747b68c5d9ef37d8d2788 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:18.285290 27066 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 516830fc42e84af693b3af52caee30ee consensus state has no leader
W20250903 20:05:18.285678 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 36e1061915574407b6e7695be8b1e55a consensus state has no leader
I20250903 20:05:18.287590 27684 tablet_replica.cc:331] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:18.288175 27684 raft_consensus.cc:2241] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:18.288523 27684 pending_rounds.cc:70] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f: Trying to abort 1 pending ops.
I20250903 20:05:18.289515 27684 pending_rounds.cc:77] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384944731103232 op_type: NO_OP noop_request { }
I20250903 20:05:18.289757 27684 raft_consensus.cc:2887] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:18.289964 27684 raft_consensus.cc:2270] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:18.293570 27228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b84f487773c24735a61d0fd0898b2bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.293138 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 44db427ee53a4d0a907dfd63282d6d4f (table default.Table_8e482428f5d8407795dcd3111aefc596 [id=5c647f2080d043f3851e5767b1efd161]) successfully deleted
I20250903 20:05:18.296208 27684 ts_tablet_manager.cc:1905] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:18.296141 27224 tablet_service.cc:1468] Processing CreateTablet for tablet 5132e6d4288c447e80f0608ba366b69c (DEFAULT_TABLE table=default.Table_2c769d77af8d429fabe07bdd9ab6c61c [id=3a3fd5d2b11246cb92787d0471494285]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.297621 27223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05e949476a834762973cb98923420f69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.297806 27211 tablet_service.cc:1515] Processing DeleteTablet for tablet 8218868ff40b405387d560f614427c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:17 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.299394 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3d9491f777b422da4b9fca803beabff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.303009 27211 tablet_service.cc:1468] Processing CreateTablet for tablet 7cbbeb8bec5e464b85b8568e916c765f (DEFAULT_TABLE table=default.Table_d37cdb880e224494969e67d03305b7fa [id=aaaf37ae7fe84ec9a4ebdd0ab1827589]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.304395 27695 tablet_bootstrap.cc:654] T 3f78a951bb024cf290116b0fe438ddb4 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:18.304666 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 847d0a0a3d2b45938cec9058c77f6231. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.306162 27220 tablet_service.cc:1468] Processing CreateTablet for tablet 41056c5c5003415993bee74d7621bd14 (DEFAULT_TABLE table=default.Table_ca85cb69be1e4e709c000e5b77bcd3cf [id=a5ff0cf9c6ee49f39c35a5f7518bade4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.307922 27684 ts_tablet_manager.cc:1918] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:18.308326 28208 raft_consensus.cc:491] T 73f65e7515994b898eec2ea23903aff4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:18.308514 28208 raft_consensus.cc:513] T 73f65e7515994b898eec2ea23903aff4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:18.309376 28208 leader_election.cc:290] T 73f65e7515994b898eec2ea23903aff4 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
W20250903 20:05:18.310891 27320 leader_election.cc:343] T 73f65e7515994b898eec2ea23903aff4 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 73f65e7515994b898eec2ea23903aff4
I20250903 20:05:18.311575 27060 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_4258f924ddb3449bb5f49fb7a88a5073 [id=decb9fcf2ee64971ba8bcd8326736852] alter complete (version 1)
W20250903 20:05:18.311764 27184 leader_election.cc:343] T deb84fc3465443d99c11097e53f5f575 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:18.311826 27684 log.cc:1199] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/c9c870fb14e3469fb2979d8462b49121
I20250903 20:05:18.312458 27684 ts_tablet_manager.cc:1939] T c9c870fb14e3469fb2979d8462b49121 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:18.312727 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5132e6d4288c447e80f0608ba366b69c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.313308 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_37e1d397cf99443d87211c3a8e1b9b82" } new_table_name: "default.Table_9502dacb275544f5abe3e80a9487371a" modify_external_catalogs: true
I20250903 20:05:18.316900 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet c9c870fb14e3469fb2979d8462b49121 (table default.Table_8b2ba396ac1b424493dbc17297d2d133 [id=3403a412096a4ef98878426a43899d7c]) successfully deleted
I20250903 20:05:18.317183 27684 tablet_replica.cc:331] T eced730bdbc14a8c9f6b7b7e53435fd0 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
W20250903 20:05:18.317211 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet e2441aa3c2c748bba5360259fb663c81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2441aa3c2c748bba5360259fb663c81 already in progress: creating tablet
I20250903 20:05:18.317555 27684 raft_consensus.cc:2241] T eced730bdbc14a8c9f6b7b7e53435fd0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:18.317443 27227 tablet_service.cc:1515] Processing DeleteTablet for tablet fa05f54ebc324b40855fa3288760e9e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.317821 27684 raft_consensus.cc:2270] T eced730bdbc14a8c9f6b7b7e53435fd0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:18.318125 27216 tablet_service.cc:1468] Processing CreateTablet for tablet 878f6ad087614c15bc7e29c784f2c3c9 (DEFAULT_TABLE table=default.Table_91ed95244dda4d37985c6af207e9e033 [id=5470fea324a94f1fba9da16adb5bef60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.318576 27685 ts_tablet_manager.cc:1918] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:18.319307 27349 tablet_service.cc:1468] Processing CreateTablet for tablet 4fa886f5be0f489290921e61dcaa4916 (DEFAULT_TABLE table=default.Table_ca35470d1cb0403089893a84293d1ea2 [id=b7afa73399c94853a86ca78575d42726]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.319726 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fa886f5be0f489290921e61dcaa4916. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.320044 27684 ts_tablet_manager.cc:1905] T eced730bdbc14a8c9f6b7b7e53435fd0 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:18.320816 27184 leader_election.cc:304] T deb84fc3465443d99c11097e53f5f575 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:18.325089 27685 log.cc:1199] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/79a0553c912049cca5971b1ad6f7e11c
I20250903 20:05:18.327312 27213 tablet_service.cc:1515] Processing DeleteTablet for tablet e2441aa3c2c748bba5360259fb663c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.328365 27499 tablet_service.cc:1468] Processing CreateTablet for tablet 4fa886f5be0f489290921e61dcaa4916 (DEFAULT_TABLE table=default.Table_ca35470d1cb0403089893a84293d1ea2 [id=b7afa73399c94853a86ca78575d42726]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:18.328418 27321 leader_election.cc:343] T 73f65e7515994b898eec2ea23903aff4 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 73f65e7515994b898eec2ea23903aff4
I20250903 20:05:18.328595 27321 leader_election.cc:304] T 73f65e7515994b898eec2ea23903aff4 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:18.328868 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fa886f5be0f489290921e61dcaa4916. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.327304 28202 tablet.cc:1722] T 46e3a65118b744298b5536f88c22f1db P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:18.331415 27349 tablet_service.cc:1468] Processing CreateTablet for tablet 55b2cdee20264ae6b64ebe3607fe1ca6 (DEFAULT_TABLE table=default.Table_cd761ea7907f44a6bcf538648450b701 [id=bf319dc4de7b48cd86e517fdc7b92d11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.331817 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55b2cdee20264ae6b64ebe3607fe1ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.334255 27211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cbbeb8bec5e464b85b8568e916c765f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.334347 28208 raft_consensus.cc:2747] T 73f65e7515994b898eec2ea23903aff4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:18.334333 27184 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 55273.1us
W20250903 20:05:18.334599 27184 leader_election.cc:343] T eced730bdbc14a8c9f6b7b7e53435fd0 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:18.334760 27184 leader_election.cc:304] T eced730bdbc14a8c9f6b7b7e53435fd0 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:18.335250 27184 leader_election.cc:343] T ecd339ec12e84e88acec427630cc8668 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:18.335391 27184 leader_election.cc:304] T ecd339ec12e84e88acec427630cc8668 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:18.335354 27491 tablet_service.cc:1468] Processing CreateTablet for tablet 55b2cdee20264ae6b64ebe3607fe1ca6 (DEFAULT_TABLE table=default.Table_cd761ea7907f44a6bcf538648450b701 [id=bf319dc4de7b48cd86e517fdc7b92d11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.335464 28181 raft_consensus.cc:2747] T eced730bdbc14a8c9f6b7b7e53435fd0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:18.335649 28181 raft_consensus.cc:2747] T ecd339ec12e84e88acec427630cc8668 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:18.335788 27491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55b2cdee20264ae6b64ebe3607fe1ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.338650 27213 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c543ac4df33429b9787c6d32bf1ebeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.338950 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c543ac4df33429b9787c6d32bf1ebeb" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:18.339401 28030 raft_consensus.cc:2747] T deb84fc3465443d99c11097e53f5f575 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:18.340073 27213 tablet_service.cc:1468] Processing CreateTablet for tablet 55b2cdee20264ae6b64ebe3607fe1ca6 (DEFAULT_TABLE table=default.Table_cd761ea7907f44a6bcf538648450b701 [id=bf319dc4de7b48cd86e517fdc7b92d11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.340807 27353 tablet_service.cc:1468] Processing CreateTablet for tablet e9126e4a18c546c28927f22c9f16426e (DEFAULT_TABLE table=default.Table_01215977307c453e9236e5bd72bc8427 [id=7ae71054c4d34d66a210e7c8f7460d4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.341252 27353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9126e4a18c546c28927f22c9f16426e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.340415 27212 tablet_service.cc:1468] Processing CreateTablet for tablet 4fa886f5be0f489290921e61dcaa4916 (DEFAULT_TABLE table=default.Table_ca35470d1cb0403089893a84293d1ea2 [id=b7afa73399c94853a86ca78575d42726]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:18.341858 27184 leader_election.cc:343] T 4c543ac4df33429b9787c6d32bf1ebeb P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:18.340131 27227 tablet_service.cc:1515] Processing DeleteTablet for tablet aa9f5e0964b348b7af184f60cd145666 with delete_type TABLET_DATA_DELETED (replaced by tablet 26decef5841a4a3b85f2a5f578ae95dc) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.344209 27216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 878f6ad087614c15bc7e29c784f2c3c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.345832 27184 leader_election.cc:304] T 4c543ac4df33429b9787c6d32bf1ebeb P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:18.346073 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41056c5c5003415993bee74d7621bd14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.346105 28030 raft_consensus.cc:2747] T 4c543ac4df33429b9787c6d32bf1ebeb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:18.347131 27493 tablet_service.cc:1468] Processing CreateTablet for tablet e9126e4a18c546c28927f22c9f16426e (DEFAULT_TABLE table=default.Table_01215977307c453e9236e5bd72bc8427 [id=7ae71054c4d34d66a210e7c8f7460d4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.347608 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9126e4a18c546c28927f22c9f16426e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.350373 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f2eefeebbfc47249e2f72b7c52e905d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.350548 27496 tablet_service.cc:1468] Processing CreateTablet for tablet 97d261080b8b479ca12267b6aa2e2ab0 (DEFAULT_TABLE table=default.Table_272defb026dc43978827f668aeedc48c [id=dfb54ed86e5148c6bbe8df6525819b9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.350939 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97d261080b8b479ca12267b6aa2e2ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.353374 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 112c0dccb60c4c7c9efc164d9c8443ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.353605 27210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54602d7f723d4c10af12408a43ed1f02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.356946 27685 ts_tablet_manager.cc:1939] T 79a0553c912049cca5971b1ad6f7e11c P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:18.357357 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 331) took 3080 ms. Trace:
I20250903 20:05:18.358155 27685 rpcz_store.cc:276] 0903 20:05:15.277087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.277679 (+ 592us) service_pool.cc:225] Handling call
0903 20:05:17.635266 (+2357587us) ts_tablet_manager.cc:1145] Deleting tablet 79a0553c912049cca5971b1ad6f7e11c
0903 20:05:17.640271 (+ 5005us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:17.788797 (+148526us) spinlock_profiling.cc:243] Waited 97.9 ms on lock 0x6160000d3954. stack: 00007f589abe0dc6 00007f589abe0b80 00007f5899bc83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f58a666a600 00007f58a797db13 00007f58aa4602b7 00007f58aa492344 00007f58aa473da0 00007f58aa473b2d 0000000000a774dc 00007f589ac2714f 00007f589ac2a41b 00007f589ac2a298
0903 20:05:18.202936 (+414139us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:18.295945 (+ 93009us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:18.357323 (+ 61378us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":225139456,"tablet-delete.queue_time_us":2357377}
I20250903 20:05:18.359391 27222 tablet_service.cc:1468] Processing CreateTablet for tablet 97d261080b8b479ca12267b6aa2e2ab0 (DEFAULT_TABLE table=default.Table_272defb026dc43978827f668aeedc48c [id=dfb54ed86e5148c6bbe8df6525819b9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.359297 27227 tablet_service.cc:1468] Processing CreateTablet for tablet e9126e4a18c546c28927f22c9f16426e (DEFAULT_TABLE table=default.Table_01215977307c453e9236e5bd72bc8427 [id=7ae71054c4d34d66a210e7c8f7460d4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.361442 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 79a0553c912049cca5971b1ad6f7e11c (table default.Table_afc0f683c5d8460484160a3a9c3e6df5 [id=b89ea241456b4393994ad07d839053f1]) successfully deleted
I20250903 20:05:18.362258 27685 tablet_replica.cc:331] T 264127511a234e9fbf409f3aa2860ccc P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:18.362665 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_2b5e97597b354141839a4ec0ca6bf046"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:18.363365 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b5e97597b354141839a4ec0ca6bf046 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:18.364496 27685 raft_consensus.cc:2241] T 264127511a234e9fbf409f3aa2860ccc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:18.364859 27212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fa886f5be0f489290921e61dcaa4916. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.368818 27685 raft_consensus.cc:2270] T 264127511a234e9fbf409f3aa2860ccc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:18.369266 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97d261080b8b479ca12267b6aa2e2ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.371456 27685 ts_tablet_manager.cc:1905] T 264127511a234e9fbf409f3aa2860ccc P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:18.373100 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_647a8fb3a7d74bd48d49b76c3266b37f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:18.373756 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_647a8fb3a7d74bd48d49b76c3266b37f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:18.373822 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55b2cdee20264ae6b64ebe3607fe1ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.374267 27684 ts_tablet_manager.cc:1918] T eced730bdbc14a8c9f6b7b7e53435fd0 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:18.374636 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9126e4a18c546c28927f22c9f16426e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.375212 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_781d39d6b06f49c28fa93c3692c7b5f7" } modify_external_catalogs: true
I20250903 20:05:18.375388 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_781d39d6b06f49c28fa93c3692c7b5f7" } modify_external_catalogs: true
I20250903 20:05:18.376794 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_d89675ff457643349b686394094cc04a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:18.377530 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d89675ff457643349b686394094cc04a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:18.382431 27684 log.cc:1199] T eced730bdbc14a8c9f6b7b7e53435fd0 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/eced730bdbc14a8c9f6b7b7e53435fd0
I20250903 20:05:18.383064 27684 ts_tablet_manager.cc:1939] T eced730bdbc14a8c9f6b7b7e53435fd0 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:18.383155 27693 tablet_bootstrap.cc:492] T eced730bdbc14a8c9f6b7b7e53435fd0 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:18.383347 27693 ts_tablet_manager.cc:1397] T eced730bdbc14a8c9f6b7b7e53435fd0 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.152s user 0.004s sys 0.007s
I20250903 20:05:18.386819 27693 raft_consensus.cc:357] T eced730bdbc14a8c9f6b7b7e53435fd0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.387051 27693 raft_consensus.cc:383] T eced730bdbc14a8c9f6b7b7e53435fd0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:18.388531 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet eced730bdbc14a8c9f6b7b7e53435fd0 (table default.Table_b520cd881f58424dadf3fa4a013812ee [id=2c179236954b41cfa9242e16946ef9bb]) successfully deleted
W20250903 20:05:18.388856 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet fa05f54ebc324b40855fa3288760e9e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fa05f54ebc324b40855fa3288760e9e9 already in progress: creating tablet
W20250903 20:05:18.389181 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet aa9f5e0964b348b7af184f60cd145666 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa9f5e0964b348b7af184f60cd145666 already in progress: creating tablet
I20250903 20:05:18.385463 28207 tablet.cc:1722] T 46e3a65118b744298b5536f88c22f1db P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:18.392421 27687 ts_tablet_manager.cc:1918] T 79a0553c912049cca5971b1ad6f7e11c P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:18.392597 27687 log.cc:1199] T 79a0553c912049cca5971b1ad6f7e11c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/79a0553c912049cca5971b1ad6f7e11c
I20250903 20:05:18.392954 27687 ts_tablet_manager.cc:1939] T 79a0553c912049cca5971b1ad6f7e11c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:18.394097 27687 tablet_replica.cc:331] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:18.394341 27687 raft_consensus.cc:2241] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:18.394286 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 79a0553c912049cca5971b1ad6f7e11c (table default.Table_afc0f683c5d8460484160a3a9c3e6df5 [id=b89ea241456b4393994ad07d839053f1]) successfully deleted
I20250903 20:05:18.394910 27687 raft_consensus.cc:2270] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:18.387144 27693 raft_consensus.cc:738] T eced730bdbc14a8c9f6b7b7e53435fd0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:18.395355 27693 consensus_queue.cc:260] T eced730bdbc14a8c9f6b7b7e53435fd0 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.399379 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 0ca6e2165a9d41299c1fe0bfa9475915
I20250903 20:05:18.400467 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 0ca6e2165a9d41299c1fe0bfa9475915 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:17 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:18.403707 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet 0ca6e2165a9d41299c1fe0bfa9475915 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:17 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:18.403853 27692 tablet_bootstrap.cc:492] T e8bc99de4ffe43d58351d170b7c61d8a P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:18.404047 27692 ts_tablet_manager.cc:1397] T e8bc99de4ffe43d58351d170b7c61d8a P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.227s user 0.003s sys 0.013s
I20250903 20:05:18.404542 27219 tablet_service.cc:1515] Processing DeleteTablet for tablet 0ca6e2165a9d41299c1fe0bfa9475915 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:17 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:18.405263 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 0ca6e2165a9d41299c1fe0bfa9475915 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ca6e2165a9d41299c1fe0bfa9475915 already in progress: creating tablet
I20250903 20:05:18.405277 27692 raft_consensus.cc:357] T e8bc99de4ffe43d58351d170b7c61d8a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.405500 27692 raft_consensus.cc:383] T e8bc99de4ffe43d58351d170b7c61d8a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:18.405575 27692 raft_consensus.cc:738] T e8bc99de4ffe43d58351d170b7c61d8a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:18.405843 27692 consensus_queue.cc:260] T e8bc99de4ffe43d58351d170b7c61d8a P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.406253 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_8c565fdbc0e641bfaee34e0b42f855ee" } modify_external_catalogs: true
I20250903 20:05:18.406481 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_8c565fdbc0e641bfaee34e0b42f855ee" } modify_external_catalogs: true
I20250903 20:05:18.408805 27693 ts_tablet_manager.cc:1428] T eced730bdbc14a8c9f6b7b7e53435fd0 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20250903 20:05:18.409663 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet eced730bdbc14a8c9f6b7b7e53435fd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:18.421742 27693 tablet_bootstrap.cc:492] T 264127511a234e9fbf409f3aa2860ccc P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:18.423563 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_bb8c68be79034a5c8469862c43953131" } modify_external_catalogs: true
I20250903 20:05:18.423794 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_bb8c68be79034a5c8469862c43953131" } modify_external_catalogs: true
I20250903 20:05:18.423833 27693 tablet_bootstrap.cc:654] T 264127511a234e9fbf409f3aa2860ccc P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:18.433511 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 63a1f26fac3b4b939ebb79c6a07d8f8e
I20250903 20:05:18.434079 27061 catalog_manager.cc:6652] ReplaceTablet: tablet 63a1f26fac3b4b939ebb79c6a07d8f8e deleted and replaced by tablet 4789bd72aa454b639df76bd5ad379850
I20250903 20:05:18.434418 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 63a1f26fac3b4b939ebb79c6a07d8f8e with delete_type TABLET_DATA_DELETED (replaced by tablet 4789bd72aa454b639df76bd5ad379850) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:18.436682 27695 tablet_bootstrap.cc:492] T 3f78a951bb024cf290116b0fe438ddb4 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:18.437069 27695 ts_tablet_manager.cc:1397] T 3f78a951bb024cf290116b0fe438ddb4 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.248s user 0.005s sys 0.009s
I20250903 20:05:18.437912 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet 63a1f26fac3b4b939ebb79c6a07d8f8e with delete_type TABLET_DATA_DELETED (replaced by tablet 4789bd72aa454b639df76bd5ad379850) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:18.438424 27695 raft_consensus.cc:357] T 3f78a951bb024cf290116b0fe438ddb4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:18.439714 27695 raft_consensus.cc:383] T 3f78a951bb024cf290116b0fe438ddb4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:18.439834 27695 raft_consensus.cc:738] T 3f78a951bb024cf290116b0fe438ddb4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:18.440673 27061 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet e8bc99de4ffe43d58351d170b7c61d8a (table default.Table_c5256f043b61433b8debe9e13625bd0d [id=416ff29f66d14564826a6734dde1c604]). Expected version 2 got 0
I20250903 20:05:18.441145 27692 ts_tablet_manager.cc:1428] T e8bc99de4ffe43d58351d170b7c61d8a P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.037s user 0.001s sys 0.001s
I20250903 20:05:18.441567 27692 tablet_bootstrap.cc:492] T 264127511a234e9fbf409f3aa2860ccc P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
W20250903 20:05:18.442080 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 63a1f26fac3b4b939ebb79c6a07d8f8e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63a1f26fac3b4b939ebb79c6a07d8f8e already in progress: creating tablet
I20250903 20:05:18.445881 27349 tablet_service.cc:1468] Processing CreateTablet for tablet 97d261080b8b479ca12267b6aa2e2ab0 (DEFAULT_TABLE table=default.Table_272defb026dc43978827f668aeedc48c [id=dfb54ed86e5148c6bbe8df6525819b9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.447345 27061 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet e8bc99de4ffe43d58351d170b7c61d8a (table default.Table_c5256f043b61433b8debe9e13625bd0d [id=416ff29f66d14564826a6734dde1c604]). Expected version 2 got 0
I20250903 20:05:18.448312 27223 tablet_service.cc:1515] Processing DeleteTablet for tablet 63a1f26fac3b4b939ebb79c6a07d8f8e with delete_type TABLET_DATA_DELETED (replaced by tablet 4789bd72aa454b639df76bd5ad379850) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.449256 27687 ts_tablet_manager.cc:1905] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:18.450147 27692 tablet_bootstrap.cc:654] T 264127511a234e9fbf409f3aa2860ccc P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:18.450552 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97d261080b8b479ca12267b6aa2e2ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.458683 27695 consensus_queue.cc:260] T 3f78a951bb024cf290116b0fe438ddb4 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:18.467761 27695 ts_tablet_manager.cc:1428] T 3f78a951bb024cf290116b0fe438ddb4 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20250903 20:05:18.468647 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6b435f3c61214c74a632c94e955abcdf" } modify_external_catalogs: true
I20250903 20:05:18.468890 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6b435f3c61214c74a632c94e955abcdf" } modify_external_catalogs: true
I20250903 20:05:18.469686 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_9d9f52b616d74e7396568e15ecaa1f34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:18.470768 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_116e90ba6d134656a9b891dd8b28d5e3" } new_table_name: "default.Table_07159d0e0aef45218e8698b043568648" modify_external_catalogs: true
W20250903 20:05:18.474308 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d9f52b616d74e7396568e15ecaa1f34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:18.478368 27693 tablet_bootstrap.cc:492] T 264127511a234e9fbf409f3aa2860ccc P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:18.478539 27693 ts_tablet_manager.cc:1397] T 264127511a234e9fbf409f3aa2860ccc P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.000s
I20250903 20:05:18.479355 27693 raft_consensus.cc:357] T 264127511a234e9fbf409f3aa2860ccc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:18.479537 27693 raft_consensus.cc:383] T 264127511a234e9fbf409f3aa2860ccc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:18.479607 27693 raft_consensus.cc:738] T 264127511a234e9fbf409f3aa2860ccc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:18.479799 27693 consensus_queue.cc:260] T 264127511a234e9fbf409f3aa2860ccc P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:18.480276 27693 ts_tablet_manager.cc:1428] T 264127511a234e9fbf409f3aa2860ccc P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:18.480578 27693 tablet_bootstrap.cc:492] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:18.482426 27693 tablet_bootstrap.cc:654] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:18.487318 27695 tablet_bootstrap.cc:492] T fbfabb6ff1714d2e921f45256b543759 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:18.488322 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet 3f78a951bb024cf290116b0fe438ddb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.501294 27693 tablet_bootstrap.cc:492] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:18.501513 27693 ts_tablet_manager.cc:1397] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.004s
I20250903 20:05:18.502434 27693 raft_consensus.cc:357] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.502648 27693 raft_consensus.cc:383] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:18.502727 27693 raft_consensus.cc:738] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:18.503015 27693 consensus_queue.cc:260] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.503472 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet f0efbb7eefb84ab6bee5b9625b91e298
I20250903 20:05:18.503618 27693 ts_tablet_manager.cc:1428] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:18.504237 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet f0efbb7eefb84ab6bee5b9625b91e298 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:18.504833 27693 tablet_bootstrap.cc:492] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:18.507133 27693 tablet_bootstrap.cc:654] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:18.509864 28201 tablet.cc:1722] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250903 20:05:18.517439 27695 tablet_bootstrap.cc:654] T fbfabb6ff1714d2e921f45256b543759 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:18.511518 28207 tablet.cc:1722] T c8a293c6fa034e9fb29682439cb3bcc2 P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250903 20:05:18.519536 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet f0efbb7eefb84ab6bee5b9625b91e298 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:18.527135 27051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.125:44059, user_credentials={real_user=slave}} blocked reactor thread for 50694.6us
I20250903 20:05:18.528410 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet f0efbb7eefb84ab6bee5b9625b91e298 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:18.535950 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet f4dec89fc6bb436ba6542c267eaf9c15
I20250903 20:05:18.536990 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet a38f65e992884635aae9942681c0dc4f
W20250903 20:05:18.537901 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet f0efbb7eefb84ab6bee5b9625b91e298 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0efbb7eefb84ab6bee5b9625b91e298 already in progress: creating tablet
I20250903 20:05:18.539656 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet a38f65e992884635aae9942681c0dc4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.539769 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet f4dec89fc6bb436ba6542c267eaf9c15 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:18.540104 27348 tablet_service.cc:1515] Processing DeleteTablet for tablet a38f65e992884635aae9942681c0dc4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:18.537869 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 36e1061915574407b6e7695be8b1e55a consensus state has no leader
I20250903 20:05:18.541015 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_be46ef5a0c524f32b001a4fd218d497f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:18.541875 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be46ef5a0c524f32b001a4fd218d497f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:18.543473 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5e9a58e5be084c1fa563357bdab3d530" } new_table_name: "default.Table_8a37ddf23e9041d59958722b82a977f9" modify_external_catalogs: true
I20250903 20:05:18.547381 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_832c9e0fbe804a61bcf618c06392b38c" } new_table_name: "default.Table_fd71464db83e4f1d8595472f6e3eb0b7" modify_external_catalogs: true
I20250903 20:05:18.551203 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_9fa3fddcea2e482baf53322b4072b3b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:18.551415 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_c4a34f406b12427ea406d72fc1be9db9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:18.551838 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4a34f406b12427ea406d72fc1be9db9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:18.551842 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet f4dec89fc6bb436ba6542c267eaf9c15 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.553203 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_232f4e198e7746f8a49e0481f1166138" } new_table_name: "default.Table_6f10e1448c2c4efabebd7d91b7bf5d79" modify_external_catalogs: true
I20250903 20:05:18.554430 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet f4dec89fc6bb436ba6542c267eaf9c15 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:18.554721 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet a38f65e992884635aae9942681c0dc4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:18.551203 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_41a441a055bb47d283b45107f9ac0385"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:18.555823 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41a441a055bb47d283b45107f9ac0385 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:18.551839 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9fa3fddcea2e482baf53322b4072b3b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:18.558243 27685 ts_tablet_manager.cc:1918] T 264127511a234e9fbf409f3aa2860ccc P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:18.558507 27685 log.cc:1199] T 264127511a234e9fbf409f3aa2860ccc P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/264127511a234e9fbf409f3aa2860ccc
W20250903 20:05:18.558774 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet f4dec89fc6bb436ba6542c267eaf9c15 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4dec89fc6bb436ba6542c267eaf9c15 already in progress: creating tablet
W20250903 20:05:18.559020 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet a38f65e992884635aae9942681c0dc4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a38f65e992884635aae9942681c0dc4f already in progress: creating tablet
I20250903 20:05:18.559386 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_739bc0d4c1d4473caac21639a4ade167"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:18.559748 27685 ts_tablet_manager.cc:1939] T 264127511a234e9fbf409f3aa2860ccc P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
W20250903 20:05:18.559890 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_739bc0d4c1d4473caac21639a4ade167 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:18.561213 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 332) took 3282 ms. Trace:
I20250903 20:05:18.561350 27685 rpcz_store.cc:276] 0903 20:05:15.277178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.277921 (+ 743us) service_pool.cc:225] Handling call
0903 20:05:18.358566 (+3080645us) ts_tablet_manager.cc:1145] Deleting tablet 264127511a234e9fbf409f3aa2860ccc
0903 20:05:18.358577 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:18.513624 (+155047us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:18.555821 (+ 42197us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:18.560025 (+ 4204us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3080441}
I20250903 20:05:18.561887 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 264127511a234e9fbf409f3aa2860ccc (table default.Table_1ade5bba3a18432c808ee7c4b910d247 [id=d20347a8196e44459448223a3af2dead]) successfully deleted
I20250903 20:05:18.562315 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 264127511a234e9fbf409f3aa2860ccc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:18.566382 27685 tablet_replica.cc:331] T 691cb69759654fdc9223844049a06d02 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:18.566663 27685 raft_consensus.cc:2241] T 691cb69759654fdc9223844049a06d02 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:18.567893 27685 raft_consensus.cc:2270] T 691cb69759654fdc9223844049a06d02 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:18.568342 27687 ts_tablet_manager.cc:1918] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:18.568500 27687 log.cc:1199] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/2a5c74a281b44019872a6442d41bc373
I20250903 20:05:18.568740 27685 ts_tablet_manager.cc:1905] T 691cb69759654fdc9223844049a06d02 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:18.568955 27687 ts_tablet_manager.cc:1939] T 2a5c74a281b44019872a6442d41bc373 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
W20250903 20:05:18.569927 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 259bc73ca9df47e9874692725a60bb1c consensus state has no leader
I20250903 20:05:18.570243 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 2a5c74a281b44019872a6442d41bc373 (table default.Table_5c08431ef8e04a6aab9b96ab78615d2f [id=a9ddfed19dab46a7bb4e049483e3996f]) successfully deleted
W20250903 20:05:18.573498 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9cea603e8e184791a464dbbc13bbbd7c has no consensus state
I20250903 20:05:18.573873 27062 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_9502dacb275544f5abe3e80a9487371a [id=5cfc8827e4ca4f4a94424c48d6f71eb5] alter complete (version 3)
I20250903 20:05:18.580678 27685 ts_tablet_manager.cc:1918] T 691cb69759654fdc9223844049a06d02 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:18.580840 27685 log.cc:1199] T 691cb69759654fdc9223844049a06d02 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/691cb69759654fdc9223844049a06d02
I20250903 20:05:18.581241 27685 ts_tablet_manager.cc:1939] T 691cb69759654fdc9223844049a06d02 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:18.582341 27695 tablet_bootstrap.cc:492] T fbfabb6ff1714d2e921f45256b543759 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:18.582615 27695 ts_tablet_manager.cc:1397] T fbfabb6ff1714d2e921f45256b543759 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.114s user 0.011s sys 0.003s
I20250903 20:05:18.583472 27695 raft_consensus.cc:357] T fbfabb6ff1714d2e921f45256b543759 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.583711 27695 raft_consensus.cc:383] T fbfabb6ff1714d2e921f45256b543759 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:18.583820 27695 raft_consensus.cc:738] T fbfabb6ff1714d2e921f45256b543759 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:18.584057 27695 consensus_queue.cc:260] T fbfabb6ff1714d2e921f45256b543759 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.584656 27695 ts_tablet_manager.cc:1428] T fbfabb6ff1714d2e921f45256b543759 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:18.585389 27695 tablet_bootstrap.cc:492] T bf2545037c214bb1b5060575e3cdf81a P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:18.589673 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_2bbccfd73e824cdcb9d9d13e7da829c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:18.590637 27695 tablet_bootstrap.cc:654] T bf2545037c214bb1b5060575e3cdf81a P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:18.590605 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ea59a50e05934c84a8cf4e8f198d7093" } new_table_name: "default.Table_d4e288e7a9f9417288b39f130bab4289" modify_external_catalogs: true
I20250903 20:05:18.592037 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 333) took 3314 ms. Trace:
I20250903 20:05:18.592602 27685 rpcz_store.cc:276] 0903 20:05:15.277305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.278162 (+ 857us) service_pool.cc:225] Handling call
0903 20:05:18.561888 (+3283726us) ts_tablet_manager.cc:1145] Deleting tablet 691cb69759654fdc9223844049a06d02
0903 20:05:18.561898 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:18.576049 (+ 14151us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:18.580522 (+ 4473us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:18.591994 (+ 11472us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3283457}
I20250903 20:05:18.593168 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 691cb69759654fdc9223844049a06d02 (table default.Table_64c39ce0250747edacd69ba845d72455 [id=dbfd528fcaba4b519108e3c4a5a5c36f]) successfully deleted
I20250903 20:05:18.593535 27065 master_service.cc:948] ReplaceTablet: received request to replace tablet 264127511a234e9fbf409f3aa2860ccc from {username='slave'} at 127.0.0.1:45936
W20250903 20:05:18.594165 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2bbccfd73e824cdcb9d9d13e7da829c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:18.598042 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 339) took 3214 ms. Trace:
I20250903 20:05:18.600885 27685 rpcz_store.cc:276] 0903 20:05:15.383922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.383965 (+ 43us) service_pool.cc:225] Handling call
0903 20:05:18.593033 (+3209068us) ts_tablet_manager.cc:1145] Deleting tablet 9685cba2f73b412382a69dab2c7f50c1
0903 20:05:18.597855 (+ 4822us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:18.597996 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3203675}
I20250903 20:05:18.604584 27064 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f6b26b9f6ef340989d2250b8d83647c2" } modify_external_catalogs: true
I20250903 20:05:18.604842 27064 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f6b26b9f6ef340989d2250b8d83647c2" } modify_external_catalogs: true
I20250903 20:05:18.605648 28202 tablet.cc:1722] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250903 20:05:18.606134 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet fdc77810f27a40758e063b8b1150ab71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:18.606915 27687 tablet_replica.cc:331] T 31342583c026438cbbae14106e75bbe5 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
W20250903 20:05:18.608202 27050 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet cd92e3dafca6419f8eba7a4ef51a911e (table default.Table_c8d55d18c6bb4511ab10f35d2db6d2f9 [id=b15b2b0164104454a23d766feea4e0de], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet cd92e3dafca6419f8eba7a4ef51a911e consensus state has no leader
I20250903 20:05:18.609694 27687 raft_consensus.cc:2241] T 31342583c026438cbbae14106e75bbe5 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:18.610023 27687 raft_consensus.cc:2270] T 31342583c026438cbbae14106e75bbe5 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:18.610710 27687 ts_tablet_manager.cc:1905] T 31342583c026438cbbae14106e75bbe5 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:18.612017 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6238740ce1894e899be9a40321f0ab73 consensus state has no leader
I20250903 20:05:18.614171 27061 master_service.cc:948] ReplaceTablet: received request to replace tablet 7c398634d6304a12b35c0579fba4c813 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:18.615272 27692 tablet_bootstrap.cc:492] T 264127511a234e9fbf409f3aa2860ccc P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:18.615484 27692 ts_tablet_manager.cc:1397] T 264127511a234e9fbf409f3aa2860ccc P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.174s user 0.004s sys 0.007s
I20250903 20:05:18.616330 27692 raft_consensus.cc:357] T 264127511a234e9fbf409f3aa2860ccc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:18.616652 27692 raft_consensus.cc:383] T 264127511a234e9fbf409f3aa2860ccc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:18.616777 27692 raft_consensus.cc:738] T 264127511a234e9fbf409f3aa2860ccc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:18.617059 27692 consensus_queue.cc:260] T 264127511a234e9fbf409f3aa2860ccc P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:18.621382 27692 ts_tablet_manager.cc:1428] T 264127511a234e9fbf409f3aa2860ccc P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250903 20:05:18.621770 27692 tablet_bootstrap.cc:492] T a10b079e7d5a440580ac00195fe88c65 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:18.624590 27693 tablet_bootstrap.cc:492] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:18.624754 27693 ts_tablet_manager.cc:1397] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.120s user 0.011s sys 0.011s
I20250903 20:05:18.624086 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet 264127511a234e9fbf409f3aa2860ccc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:18.625614 27684 tablet_replica.cc:331] T 264127511a234e9fbf409f3aa2860ccc P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:18.625808 27684 raft_consensus.cc:2241] T 264127511a234e9fbf409f3aa2860ccc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:18.623039 27695 tablet_bootstrap.cc:492] T bf2545037c214bb1b5060575e3cdf81a P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:18.626030 27684 raft_consensus.cc:2270] T 264127511a234e9fbf409f3aa2860ccc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:18.626120 27695 ts_tablet_manager.cc:1397] T bf2545037c214bb1b5060575e3cdf81a P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.041s user 0.007s sys 0.000s
I20250903 20:05:18.626973 27695 raft_consensus.cc:357] T bf2545037c214bb1b5060575e3cdf81a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.627228 27692 tablet_bootstrap.cc:654] T a10b079e7d5a440580ac00195fe88c65 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:18.627216 27695 raft_consensus.cc:383] T bf2545037c214bb1b5060575e3cdf81a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:18.627298 27695 raft_consensus.cc:738] T bf2545037c214bb1b5060575e3cdf81a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:18.627635 27695 consensus_queue.cc:260] T bf2545037c214bb1b5060575e3cdf81a P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.627897 27684 ts_tablet_manager.cc:1905] T 264127511a234e9fbf409f3aa2860ccc P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:18.628157 27695 ts_tablet_manager.cc:1428] T bf2545037c214bb1b5060575e3cdf81a P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:18.628551 27695 tablet_bootstrap.cc:492] T 6238740ce1894e899be9a40321f0ab73 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:18.630338 27695 tablet_bootstrap.cc:654] T 6238740ce1894e899be9a40321f0ab73 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:18.631230 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet de20d18d7f264811a0631aa1144aef2c consensus state has no leader
I20250903 20:05:18.632207 27693 raft_consensus.cc:357] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.632450 27693 raft_consensus.cc:383] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:18.632534 27693 raft_consensus.cc:738] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:18.632773 27693 consensus_queue.cc:260] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.633296 27693 ts_tablet_manager.cc:1428] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250903 20:05:18.633842 27693 tablet_bootstrap.cc:492] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:18.605892 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 340) took 3221 ms. Trace:
I20250903 20:05:18.634416 27685 rpcz_store.cc:276] 0903 20:05:15.384051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.384129 (+ 78us) service_pool.cc:225] Handling call
0903 20:05:18.605717 (+3221588us) ts_tablet_manager.cc:1145] Deleting tablet 961ade36f7d14eba91bb16d5eae2e254
0903 20:05:18.605728 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:18.605849 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3215700}
W20250903 20:05:18.634970 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 961ade36f7d14eba91bb16d5eae2e254 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 961ade36f7d14eba91bb16d5eae2e254 already in progress: creating tablet
I20250903 20:05:18.636282 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 338) took 3252 ms. Trace:
I20250903 20:05:18.638851 27693 tablet_bootstrap.cc:654] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:18.639030 27687 ts_tablet_manager.cc:1918] T 31342583c026438cbbae14106e75bbe5 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:18.639216 27687 log.cc:1199] T 31342583c026438cbbae14106e75bbe5 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/31342583c026438cbbae14106e75bbe5
I20250903 20:05:18.639613 27687 ts_tablet_manager.cc:1939] T 31342583c026438cbbae14106e75bbe5 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:18.640825 27685 rpcz_store.cc:276] 0903 20:05:15.383737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.383786 (+ 49us) service_pool.cc:225] Handling call
0903 20:05:18.634729 (+3250943us) ts_tablet_manager.cc:1145] Deleting tablet 726f4d920f3e47fea41b4e12c021c988
0903 20:05:18.634738 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:18.636239 (+ 1501us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3242730}
W20250903 20:05:18.642129 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 726f4d920f3e47fea41b4e12c021c988 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 726f4d920f3e47fea41b4e12c021c988 already in progress: creating tablet
I20250903 20:05:18.642557 27685 tablet_replica.cc:331] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:18.642820 27685 raft_consensus.cc:2241] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:18.647241 27687 tablet_replica.cc:331] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:18.647431 27687 raft_consensus.cc:2241] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:18.647472 27685 pending_rounds.cc:70] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 89f8f0ee68b940db9ce250c0604f934a: Trying to abort 1 pending ops.
I20250903 20:05:18.647670 27687 pending_rounds.cc:70] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528: Trying to abort 1 pending ops.
I20250903 20:05:18.647655 27685 pending_rounds.cc:77] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 89f8f0ee68b940db9ce250c0604f934a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384933810667520 op_type: NO_OP noop_request { }
I20250903 20:05:18.647832 27687 pending_rounds.cc:77] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384944128565248 op_type: NO_OP noop_request { }
I20250903 20:05:18.647981 27687 raft_consensus.cc:2887] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:18.647889 27685 raft_consensus.cc:2887] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:18.648085 27687 raft_consensus.cc:2270] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:18.648248 27685 raft_consensus.cc:2270] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:18.649061 27685 ts_tablet_manager.cc:1905] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:18.647351 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 31342583c026438cbbae14106e75bbe5 (table default.Table_943bea896c744410a2a4d80ffb9ca439 [id=a950fbaecd7341989961ff3ea9a713e0]) successfully deleted
I20250903 20:05:18.650070 27692 tablet_bootstrap.cc:492] T a10b079e7d5a440580ac00195fe88c65 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:18.650301 27692 ts_tablet_manager.cc:1397] T a10b079e7d5a440580ac00195fe88c65 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.004s
I20250903 20:05:18.651301 27692 raft_consensus.cc:357] T a10b079e7d5a440580ac00195fe88c65 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.651517 27692 raft_consensus.cc:383] T a10b079e7d5a440580ac00195fe88c65 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:18.651597 27692 raft_consensus.cc:738] T a10b079e7d5a440580ac00195fe88c65 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:18.651786 27693 tablet_bootstrap.cc:492] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:18.651933 27693 ts_tablet_manager.cc:1397] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20250903 20:05:18.651865 27692 consensus_queue.cc:260] T a10b079e7d5a440580ac00195fe88c65 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.654606 27692 ts_tablet_manager.cc:1428] T a10b079e7d5a440580ac00195fe88c65 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:18.655333 27692 tablet_bootstrap.cc:492] T ecd339ec12e84e88acec427630cc8668 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:18.655709 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet ecd339ec12e84e88acec427630cc8668 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:18.657317 27687 ts_tablet_manager.cc:1905] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:18.657245 27693 raft_consensus.cc:357] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.657476 27693 raft_consensus.cc:383] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:18.657560 27693 raft_consensus.cc:738] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:18.658928 27695 tablet_bootstrap.cc:492] T 6238740ce1894e899be9a40321f0ab73 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:18.659337 27695 ts_tablet_manager.cc:1397] T 6238740ce1894e899be9a40321f0ab73 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.005s
I20250903 20:05:18.659516 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet a10b079e7d5a440580ac00195fe88c65 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b3bfc63aa0b4e56bec189e834dc6a8d) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:18.660459 27695 raft_consensus.cc:357] T 6238740ce1894e899be9a40321f0ab73 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.660661 27695 raft_consensus.cc:383] T 6238740ce1894e899be9a40321f0ab73 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:18.660758 27695 raft_consensus.cc:738] T 6238740ce1894e899be9a40321f0ab73 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:18.661022 27695 consensus_queue.cc:260] T 6238740ce1894e899be9a40321f0ab73 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.661710 27692 tablet_bootstrap.cc:654] T ecd339ec12e84e88acec427630cc8668 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:18.664130 27695 ts_tablet_manager.cc:1428] T 6238740ce1894e899be9a40321f0ab73 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250903 20:05:18.664471 27695 tablet_bootstrap.cc:492] T 57a1167457b84ea8a493be232b1d088a P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:18.665120 27684 ts_tablet_manager.cc:1918] T 264127511a234e9fbf409f3aa2860ccc P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:18.665295 27684 log.cc:1199] T 264127511a234e9fbf409f3aa2860ccc P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/264127511a234e9fbf409f3aa2860ccc
I20250903 20:05:18.665738 27684 ts_tablet_manager.cc:1939] T 264127511a234e9fbf409f3aa2860ccc P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:18.666877 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b1d2232da9bd4385b1b65355d3be0b4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:18.667822 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1d2232da9bd4385b1b65355d3be0b4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:18.672024 27693 consensus_queue.cc:260] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.674443 27693 ts_tablet_manager.cc:1428] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.022s user 0.003s sys 0.000s
I20250903 20:05:18.671629 27695 tablet_bootstrap.cc:654] T 57a1167457b84ea8a493be232b1d088a P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:18.675653 27693 tablet_bootstrap.cc:492] T 6ea37668d64d4feeb92a162711609c13 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:18.679569 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet a10b079e7d5a440580ac00195fe88c65 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b3bfc63aa0b4e56bec189e834dc6a8d) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:18.680564 27687 ts_tablet_manager.cc:1918] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:18.680730 27687 log.cc:1199] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/691cb69759654fdc9223844049a06d02
I20250903 20:05:18.681180 27687 ts_tablet_manager.cc:1939] T 691cb69759654fdc9223844049a06d02 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:18.681357 27693 tablet_bootstrap.cc:654] T 6ea37668d64d4feeb92a162711609c13 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:18.681422 27685 ts_tablet_manager.cc:1918] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:18.681564 27685 log.cc:1199] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/04ccb3e4a8bd4789b9f19edd4a5b6145
I20250903 20:05:18.681977 27685 ts_tablet_manager.cc:1939] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:18.683732 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 264127511a234e9fbf409f3aa2860ccc (table default.Table_1ade5bba3a18432c808ee7c4b910d247 [id=d20347a8196e44459448223a3af2dead]) successfully deleted
I20250903 20:05:18.684072 27684 tablet_replica.cc:331] T a10b079e7d5a440580ac00195fe88c65 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:18.685411 27684 raft_consensus.cc:2241] T a10b079e7d5a440580ac00195fe88c65 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:18.691691 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 464) took 1001 ms. Trace:
I20250903 20:05:18.691871 27687 rpcz_store.cc:276] 0903 20:05:17.690590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.690639 (+ 49us) service_pool.cc:225] Handling call
0903 20:05:18.647153 (+956514us) ts_tablet_manager.cc:1145] Deleting tablet 691cb69759654fdc9223844049a06d02
0903 20:05:18.647162 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:18.672509 (+ 25347us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:18.680402 (+ 7893us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:18.691605 (+ 11203us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":954346}
I20250903 20:05:18.692421 27687 tablet_replica.cc:331] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:18.692544 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 691cb69759654fdc9223844049a06d02 (table default.Table_64c39ce0250747edacd69ba845d72455 [id=dbfd528fcaba4b519108e3c4a5a5c36f]) successfully deleted
I20250903 20:05:18.692647 27687 raft_consensus.cc:2241] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:18.692852 27687 raft_consensus.cc:2270] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:18.693454 27687 ts_tablet_manager.cc:1905] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:18.693516 27693 tablet_bootstrap.cc:492] T 6ea37668d64d4feeb92a162711609c13 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:18.693641 27693 ts_tablet_manager.cc:1397] T 6ea37668d64d4feeb92a162711609c13 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.002s
I20250903 20:05:18.694715 27684 raft_consensus.cc:2270] T a10b079e7d5a440580ac00195fe88c65 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:18.696806 27057 master_service.cc:948] ReplaceTablet: received request to replace tablet 512908bc4d6941ec81b145ee5795405f from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:18.697503 27684 ts_tablet_manager.cc:1905] T a10b079e7d5a440580ac00195fe88c65 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:18.700343 27062 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet bf2545037c214bb1b5060575e3cdf81a (table default.Table_143fea9b5e3d4acbb92adc441aeede47 [id=42c554f2f532414a88b50818e2dbb458]). Expected version 1 got 0
I20250903 20:05:18.701213 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 20c25c84c6fb4609a1964a4ff3c526da
I20250903 20:05:18.702172 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet 20c25c84c6fb4609a1964a4ff3c526da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:18.704149 27210 tablet_service.cc:1515] Processing DeleteTablet for tablet 20c25c84c6fb4609a1964a4ff3c526da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.705010 27695 tablet_bootstrap.cc:492] T 57a1167457b84ea8a493be232b1d088a P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:18.705210 27695 ts_tablet_manager.cc:1397] T 57a1167457b84ea8a493be232b1d088a P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.041s user 0.001s sys 0.003s
I20250903 20:05:18.705338 27060 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 6238740ce1894e899be9a40321f0ab73 (table default.Table_6f10e1448c2c4efabebd7d91b7bf5d79 [id=53074cd97dec40c7b0e1444183d6707a]). Expected version 1 got 0
I20250903 20:05:18.705709 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 20c25c84c6fb4609a1964a4ff3c526da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:18.707896 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet ac9b5d4f39db4f05a15316585040d142
I20250903 20:05:18.708978 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet ac9b5d4f39db4f05a15316585040d142 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:18.709007 27210 tablet_service.cc:1515] Processing DeleteTablet for tablet ac9b5d4f39db4f05a15316585040d142 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.709858 27695 raft_consensus.cc:357] T 57a1167457b84ea8a493be232b1d088a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:18.710088 27695 raft_consensus.cc:383] T 57a1167457b84ea8a493be232b1d088a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:18.710153 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet ac9b5d4f39db4f05a15316585040d142 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:18.710182 27695 raft_consensus.cc:738] T 57a1167457b84ea8a493be232b1d088a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:18.710487 27695 consensus_queue.cc:260] T 57a1167457b84ea8a493be232b1d088a P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:18.711412 27493 tablet_service.cc:1468] Processing CreateTablet for tablet 64d53f2071b0449ca61caa6c264a3641 (DEFAULT_TABLE table=default.Table_4d67ac5e5c6645f69648aea15fa389ef [id=6ea10c870a1d410a9391fb921a2ef3cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.711717 27349 tablet_service.cc:1468] Processing CreateTablet for tablet 64d53f2071b0449ca61caa6c264a3641 (DEFAULT_TABLE table=default.Table_4d67ac5e5c6645f69648aea15fa389ef [id=6ea10c870a1d410a9391fb921a2ef3cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.711862 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64d53f2071b0449ca61caa6c264a3641. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.712076 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64d53f2071b0449ca61caa6c264a3641. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.713470 27695 ts_tablet_manager.cc:1428] T 57a1167457b84ea8a493be232b1d088a P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:18.713799 27695 tablet_bootstrap.cc:492] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:18.717186 27210 tablet_service.cc:1515] Processing DeleteTablet for tablet 57a1167457b84ea8a493be232b1d088a with delete_type TABLET_DATA_DELETED (replaced by tablet f326ec776ba446be8d5dda5c1dc5b6ec) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.694650 27693 raft_consensus.cc:357] T 6ea37668d64d4feeb92a162711609c13 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.722625 27693 raft_consensus.cc:383] T 6ea37668d64d4feeb92a162711609c13 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:18.722777 27693 raft_consensus.cc:738] T 6ea37668d64d4feeb92a162711609c13 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:18.725334 27687 ts_tablet_manager.cc:1918] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:18.725488 27687 log.cc:1199] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/04ccb3e4a8bd4789b9f19edd4a5b6145
I20250903 20:05:18.725870 27687 ts_tablet_manager.cc:1939] T 04ccb3e4a8bd4789b9f19edd4a5b6145 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:18.728154 27210 tablet_service.cc:1468] Processing CreateTablet for tablet 64d53f2071b0449ca61caa6c264a3641 (DEFAULT_TABLE table=default.Table_4d67ac5e5c6645f69648aea15fa389ef [id=6ea10c870a1d410a9391fb921a2ef3cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.728610 27210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64d53f2071b0449ca61caa6c264a3641. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.729941 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_35a3167eda1a45949f8cd8a281a88733" } modify_external_catalogs: true
I20250903 20:05:18.730176 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_35a3167eda1a45949f8cd8a281a88733" } modify_external_catalogs: true
I20250903 20:05:18.730336 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_83e2f4211b724a698cf478ddde974155"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:18.730963 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83e2f4211b724a698cf478ddde974155 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:18.731674 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_6353cdf28e1648ffa45cef826bdfffc3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:18.731959 27693 consensus_queue.cc:260] T 6ea37668d64d4feeb92a162711609c13 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:18.732167 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6353cdf28e1648ffa45cef826bdfffc3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:18.732575 27693 ts_tablet_manager.cc:1428] T 6ea37668d64d4feeb92a162711609c13 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.039s user 0.002s sys 0.000s
I20250903 20:05:18.733122 27695 tablet_bootstrap.cc:654] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:18.736377 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 466) took 1038 ms. Trace:
I20250903 20:05:18.736515 27687 rpcz_store.cc:276] 0903 20:05:17.697373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.697418 (+ 45us) service_pool.cc:225] Handling call
0903 20:05:18.692302 (+994884us) ts_tablet_manager.cc:1145] Deleting tablet 04ccb3e4a8bd4789b9f19edd4a5b6145
0903 20:05:18.692318 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:18.706094 (+ 13776us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:18.724760 (+ 18666us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:18.736320 (+ 11560us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":994626}
I20250903 20:05:18.736996 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 04ccb3e4a8bd4789b9f19edd4a5b6145 (table default.Table_39de38c00cdb44eb85fdb76134b7139c [id=b287b55c1542426fb4fd0372ef6bcd05]) successfully deleted
I20250903 20:05:18.737323 27693 tablet_bootstrap.cc:492] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:18.740525 27687 tablet_replica.cc:331] T 262a3549867a473aaaa7dfcc0b5872cd P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:18.740767 27687 raft_consensus.cc:2241] T 262a3549867a473aaaa7dfcc0b5872cd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:18.740943 27687 raft_consensus.cc:2270] T 262a3549867a473aaaa7dfcc0b5872cd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:18.743273 27693 tablet_bootstrap.cc:654] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:18.744088 27687 ts_tablet_manager.cc:1905] T 262a3549867a473aaaa7dfcc0b5872cd P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:18.747197 27695 tablet_bootstrap.cc:492] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:18.747355 27695 ts_tablet_manager.cc:1397] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20250903 20:05:18.748292 27695 raft_consensus.cc:357] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.748384 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 1396d4f8a93b412ca8397368fb98834e
I20250903 20:05:18.748484 27695 raft_consensus.cc:383] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:18.748571 27695 raft_consensus.cc:738] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:18.748816 27695 consensus_queue.cc:260] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.749511 27695 ts_tablet_manager.cc:1428] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:18.749838 27695 tablet_bootstrap.cc:492] T 369c750ff5bf449e989d95ffd89497d4 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:18.750029 27348 tablet_service.cc:1515] Processing DeleteTablet for tablet 1396d4f8a93b412ca8397368fb98834e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:18.750289 27210 tablet_service.cc:1515] Processing DeleteTablet for tablet 1396d4f8a93b412ca8397368fb98834e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.750891 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet 1396d4f8a93b412ca8397368fb98834e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:18.752557 27695 tablet_bootstrap.cc:654] T 369c750ff5bf449e989d95ffd89497d4 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:18.755645 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_27706ce3e18b47078bb7d3071c12a1a2" } new_table_name: "default.Table_ae457e422e0c4314aeaecd7755413366" modify_external_catalogs: true
I20250903 20:05:18.756225 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c304f6eaf4014caba10a5b376c13972d" } modify_external_catalogs: true
I20250903 20:05:18.756541 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c304f6eaf4014caba10a5b376c13972d" } modify_external_catalogs: true
I20250903 20:05:18.757213 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_3bf3d1e0cfbb43c2a04f1983c51e9192"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:18.757731 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3bf3d1e0cfbb43c2a04f1983c51e9192 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:18.758189 27210 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a76a17b64a74fbfbb4f2e92a02ae1cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.758369 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f038423efb3f4fd5bba9911713436b97" } modify_external_catalogs: true
I20250903 20:05:18.758554 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f038423efb3f4fd5bba9911713436b97" } modify_external_catalogs: true
I20250903 20:05:18.765431 27687 ts_tablet_manager.cc:1918] T 262a3549867a473aaaa7dfcc0b5872cd P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:18.765609 27687 log.cc:1199] T 262a3549867a473aaaa7dfcc0b5872cd P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/262a3549867a473aaaa7dfcc0b5872cd
I20250903 20:05:18.765936 27687 ts_tablet_manager.cc:1939] T 262a3549867a473aaaa7dfcc0b5872cd P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
W20250903 20:05:18.769737 27066 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f2629f46704e4a0fa7922533928ca2ed consensus state has no leader
I20250903 20:05:18.773723 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 0fef9aa7153a4902abce522e3ff0df3f
I20250903 20:05:18.774907 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 1223743918814d88b401782ca4b521dd
I20250903 20:05:18.775128 27210 tablet_service.cc:1515] Processing DeleteTablet for tablet 0fef9aa7153a4902abce522e3ff0df3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.776453 27348 tablet_service.cc:1515] Processing DeleteTablet for tablet 0fef9aa7153a4902abce522e3ff0df3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:18.777421 27348 tablet_service.cc:1515] Processing DeleteTablet for tablet 1223743918814d88b401782ca4b521dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:18.777459 27210 tablet_service.cc:1515] Processing DeleteTablet for tablet 1223743918814d88b401782ca4b521dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.778200 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet 0fef9aa7153a4902abce522e3ff0df3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:18.778242 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet 1223743918814d88b401782ca4b521dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:18.778771 27062 master_service.cc:948] ReplaceTablet: received request to replace tablet 6f0cba70240f4f38a4fa5ccbcd99b3b1 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:18.779142 27684 ts_tablet_manager.cc:1918] T a10b079e7d5a440580ac00195fe88c65 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:18.779286 27684 log.cc:1199] T a10b079e7d5a440580ac00195fe88c65 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/a10b079e7d5a440580ac00195fe88c65
I20250903 20:05:18.779685 27684 ts_tablet_manager.cc:1939] T a10b079e7d5a440580ac00195fe88c65 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:18.728403 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 329) took 3451 ms. Trace:
I20250903 20:05:18.785317 27685 rpcz_store.cc:276] 0903 20:05:15.276805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.412830 (+136025us) service_pool.cc:225] Handling call
0903 20:05:18.642481 (+3229651us) ts_tablet_manager.cc:1145] Deleting tablet 04ccb3e4a8bd4789b9f19edd4a5b6145
0903 20:05:18.642492 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:18.669074 (+ 26582us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:18.681262 (+ 12188us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:18.728346 (+ 47084us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3171850}
I20250903 20:05:18.786072 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 6f0cba70240f4f38a4fa5ccbcd99b3b1
I20250903 20:05:18.786650 27062 catalog_manager.cc:6652] ReplaceTablet: tablet 6f0cba70240f4f38a4fa5ccbcd99b3b1 deleted and replaced by tablet 3af0c66a8c234711a43816f248e5be03
I20250903 20:05:18.787678 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet a10b079e7d5a440580ac00195fe88c65 (table default.Table_9b20469406c94986ab496ea3e678b586 [id=6d6f5e8cadf9445d8695c075cfabb2ea]) successfully deleted
I20250903 20:05:18.787778 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f0cba70240f4f38a4fa5ccbcd99b3b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3af0c66a8c234711a43816f248e5be03) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:18.788127 27348 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f0cba70240f4f38a4fa5ccbcd99b3b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3af0c66a8c234711a43816f248e5be03) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:18.789155 27684 tablet_replica.cc:331] T 1396d4f8a93b412ca8397368fb98834e P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
W20250903 20:05:18.789999 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 20c25c84c6fb4609a1964a4ff3c526da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20c25c84c6fb4609a1964a4ff3c526da already in progress: creating tablet
I20250903 20:05:18.790146 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0c416f6e2e1b4ae4941c2e5613f16bda" } new_table_name: "default.Table_c5abd2620f3f45628ce2531cbcb81d9d" modify_external_catalogs: true
W20250903 20:05:18.790318 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet ac9b5d4f39db4f05a15316585040d142 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac9b5d4f39db4f05a15316585040d142 already in progress: creating tablet
I20250903 20:05:18.790500 27693 tablet_bootstrap.cc:492] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:18.790665 27693 ts_tablet_manager.cc:1397] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.053s user 0.000s sys 0.003s
I20250903 20:05:18.791029 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_eb74cc68ea8b4ab0bb7eb3937ac1aa4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:18.787349 27210 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f0cba70240f4f38a4fa5ccbcd99b3b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3af0c66a8c234711a43816f248e5be03) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:18.791575 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb74cc68ea8b4ab0bb7eb3937ac1aa4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:18.792363 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 04ccb3e4a8bd4789b9f19edd4a5b6145 (table default.Table_39de38c00cdb44eb85fdb76134b7139c [id=b287b55c1542426fb4fd0372ef6bcd05]) successfully deleted
I20250903 20:05:18.792832 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 351) took 3281 ms. Trace:
I20250903 20:05:18.792788 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ca35470d1cb0403089893a84293d1ea2" } new_table_name: "default.Table_b368ef35ad84486787f041c9a392c027" modify_external_catalogs: true
I20250903 20:05:18.793713 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 465) took 1096 ms. Trace:
I20250903 20:05:18.793829 27687 rpcz_store.cc:276] 0903 20:05:17.697244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.697904 (+ 660us) service_pool.cc:225] Handling call
0903 20:05:18.736818 (+1038914us) ts_tablet_manager.cc:1145] Deleting tablet 262a3549867a473aaaa7dfcc0b5872cd
0903 20:05:18.740431 (+ 3613us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:18.762100 (+ 21669us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:18.765283 (+ 3183us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:18.793648 (+ 28365us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1038747}
I20250903 20:05:18.794345 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 471) took 1009 ms. Trace:
I20250903 20:05:18.794417 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 262a3549867a473aaaa7dfcc0b5872cd (table default.Table_14af7742726b4562a02fd2c9b2895277 [id=6d72901e5102476890f466fd406a7127]) successfully deleted
I20250903 20:05:18.794446 27687 rpcz_store.cc:276] 0903 20:05:17.784628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.786272 (+ 1644us) service_pool.cc:225] Handling call
0903 20:05:18.794134 (+1007862us) ts_tablet_manager.cc:1145] Deleting tablet 6d13884fff8c43b690836c6a341a2d92
0903 20:05:18.794145 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:18.794283 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1007598}
I20250903 20:05:18.794826 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 472) took 1010 ms. Trace:
I20250903 20:05:18.794915 27687 rpcz_store.cc:276] 0903 20:05:17.784781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.786575 (+ 1794us) service_pool.cc:225] Handling call
0903 20:05:18.794691 (+1008116us) ts_tablet_manager.cc:1145] Deleting tablet 8218868ff40b405387d560f614427c08
0903 20:05:18.794703 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:18.794776 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1007958}
W20250903 20:05:18.795046 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 6d13884fff8c43b690836c6a341a2d92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d13884fff8c43b690836c6a341a2d92 already in progress: creating tablet
I20250903 20:05:18.795101 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ea37668d64d4feeb92a162711609c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:18.795317 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 473) took 1010 ms. Trace:
I20250903 20:05:18.795408 27687 rpcz_store.cc:276] 0903 20:05:17.785151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.786772 (+ 1621us) service_pool.cc:225] Handling call
0903 20:05:18.795127 (+1008355us) ts_tablet_manager.cc:1145] Deleting tablet 541278ca9ffb4e41b341139c2faa337f
0903 20:05:18.795138 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:18.795271 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1008197}
W20250903 20:05:18.795605 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 8218868ff40b405387d560f614427c08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8218868ff40b405387d560f614427c08 already in progress: creating tablet
I20250903 20:05:18.795948 27687 tablet_replica.cc:331] T c9c870fb14e3469fb2979d8462b49121 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:18.796159 27687 raft_consensus.cc:2241] T c9c870fb14e3469fb2979d8462b49121 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250903 20:05:18.796196 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 541278ca9ffb4e41b341139c2faa337f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 541278ca9ffb4e41b341139c2faa337f already in progress: creating tablet
I20250903 20:05:18.796337 27687 raft_consensus.cc:2270] T c9c870fb14e3469fb2979d8462b49121 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:18.796396 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet f9709dfd18654c5bb071e39da7c8889f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9709dfd18654c5bb071e39da7c8889f already in progress: creating tablet
W20250903 20:05:18.796620 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet b84f487773c24735a61d0fd0898b2bcb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b84f487773c24735a61d0fd0898b2bcb already in progress: creating tablet
I20250903 20:05:18.796895 27687 ts_tablet_manager.cc:1905] T c9c870fb14e3469fb2979d8462b49121 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:18.796872 27685 rpcz_store.cc:276] 0903 20:05:15.511135 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.512320 (+ 1185us) service_pool.cc:225] Handling call
0903 20:05:18.792650 (+3280330us) ts_tablet_manager.cc:1145] Deleting tablet a0f79831866b40c49c5b27ce2014c3f6
0903 20:05:18.792660 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:18.792782 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3280063}
W20250903 20:05:18.797819 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet a0f79831866b40c49c5b27ce2014c3f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0f79831866b40c49c5b27ce2014c3f6 already in progress: creating tablet
I20250903 20:05:18.798394 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 352) took 3165 ms. Trace:
W20250903 20:05:18.799221 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 200c06f296b94eab94f35f312c5a443b consensus state has no leader
I20250903 20:05:18.801030 27685 rpcz_store.cc:276] 0903 20:05:15.633437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.638802 (+ 5365us) service_pool.cc:225] Handling call
0903 20:05:18.798231 (+3159429us) ts_tablet_manager.cc:1145] Deleting tablet 0a9b7770432942da9373d5100c71bd6a
0903 20:05:18.798243 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:18.798364 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3156024}
W20250903 20:05:18.801728 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 0a9b7770432942da9373d5100c71bd6a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a9b7770432942da9373d5100c71bd6a already in progress: creating tablet
I20250903 20:05:18.802147 27685 tablet_replica.cc:331] T eced730bdbc14a8c9f6b7b7e53435fd0 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:18.802452 27685 raft_consensus.cc:2241] T eced730bdbc14a8c9f6b7b7e53435fd0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:18.803072 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 512908bc4d6941ec81b145ee5795405f
I20250903 20:05:18.804214 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 512908bc4d6941ec81b145ee5795405f with delete_type TABLET_DATA_DELETED (replaced by tablet 1598824a733a4a93bed043e81b86abf3) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:18.804731 27693 raft_consensus.cc:357] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:18.804945 27693 raft_consensus.cc:383] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:18.805022 27693 raft_consensus.cc:738] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:18.805184 27685 raft_consensus.cc:2270] T eced730bdbc14a8c9f6b7b7e53435fd0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:18.805248 27693 consensus_queue.cc:260] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:18.808341 27057 catalog_manager.cc:6652] ReplaceTablet: tablet 512908bc4d6941ec81b145ee5795405f deleted and replaced by tablet 1598824a733a4a93bed043e81b86abf3
I20250903 20:05:18.812153 27348 tablet_service.cc:1468] Processing CreateTablet for tablet 86e33bd235c14042a578a4a1c70f11b7 (DEFAULT_TABLE table=default.Table_3bcc60f1d7ed42d89cfaf71c0fc18fdd [id=14368dfbbc2a4f179dda569e352dabf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.813356 28208 raft_consensus.cc:491] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:18.813519 28208 raft_consensus.cc:513] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.792490 27692 tablet_bootstrap.cc:492] T ecd339ec12e84e88acec427630cc8668 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:18.804407 27493 tablet_service.cc:1468] Processing CreateTablet for tablet ebf0a9aaaea1408c8c9caf28f00dbff7 (DEFAULT_TABLE table=default.Table_e79d289ca0384e109873b8d5f2c78905 [id=d35bb69fba4e40c7901548fe0adff3f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.817948 27692 ts_tablet_manager.cc:1397] T ecd339ec12e84e88acec427630cc8668 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.163s user 0.015s sys 0.011s
I20250903 20:05:18.809274 27491 tablet_service.cc:1468] Processing CreateTablet for tablet 86e33bd235c14042a578a4a1c70f11b7 (DEFAULT_TABLE table=default.Table_3bcc60f1d7ed42d89cfaf71c0fc18fdd [id=14368dfbbc2a4f179dda569e352dabf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.818399 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_4a511c041bed4d8b88479f001d13c791" } new_table_name: "default.Table_d3841357472f410abd60951e763adb30" modify_external_catalogs: true
I20250903 20:05:18.810683 27496 tablet_service.cc:1468] Processing CreateTablet for tablet f36938c69f6a497f93e9458bdc9c40a0 (DEFAULT_TABLE table=default.Table_d1b7ba8a5d3440f7be2eb198bbf5e7fa [id=7badf47f3be84194b75babe421ed738a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.811378 27492 tablet_service.cc:1468] Processing CreateTablet for tablet 47500175a642457687aa4226195fce47 (DEFAULT_TABLE table=default.Table_fed56630d54540548ffb7f00228a1ee9 [id=f14a6e519cb9461286bff3839d58c50f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.818940 27692 raft_consensus.cc:357] T ecd339ec12e84e88acec427630cc8668 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.819113 27692 raft_consensus.cc:383] T ecd339ec12e84e88acec427630cc8668 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:18.812610 27488 tablet_service.cc:1468] Processing CreateTablet for tablet adea8b73e2e14cc5ae9490fc859dd0b1 (DEFAULT_TABLE table=default.Table_aef24ecf95294a9188a4338dfedc525d [id=2e5bd7fe0a5a4d38a3f8796a0f98941f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.819234 27692 raft_consensus.cc:738] T ecd339ec12e84e88acec427630cc8668 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:18.815507 27500 tablet_service.cc:1468] Processing CreateTablet for tablet 145c82e8df2c49ea89fd2f02224d8cbf (DEFAULT_TABLE table=default.Table_9abad42b86c747b68c5d9ef37d8d2788 [id=c0de66acb6ad416887e23d19cdc5434b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.819482 27692 consensus_queue.cc:260] T ecd339ec12e84e88acec427630cc8668 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.820037 27692 ts_tablet_manager.cc:1428] T ecd339ec12e84e88acec427630cc8668 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:18.820139 27685 ts_tablet_manager.cc:1905] T eced730bdbc14a8c9f6b7b7e53435fd0 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:18.816381 27497 tablet_service.cc:1468] Processing CreateTablet for tablet cf99a74acfdb4a17bbbce3a07467e9c1 (DEFAULT_TABLE table=default.Table_4f815c3438474e3ca7b8e4b9d0be87d5 [id=70128985dd71449c95ece219f0a682fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.820343 27692 tablet_bootstrap.cc:492] T fdc77810f27a40758e063b8b1150ab71 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:18.817715 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 9cea603e8e184791a464dbbc13bbbd7c (DEFAULT_TABLE table=default.Table_fd71464db83e4f1d8595472f6e3eb0b7 [id=e3996f93ec9f41b4b96701acd763f5be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.803905 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet 512908bc4d6941ec81b145ee5795405f with delete_type TABLET_DATA_DELETED (replaced by tablet 1598824a733a4a93bed043e81b86abf3) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:18.806545 27695 tablet_bootstrap.cc:492] T 369c750ff5bf449e989d95ffd89497d4 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:18.820942 27695 ts_tablet_manager.cc:1397] T 369c750ff5bf449e989d95ffd89497d4 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.071s user 0.002s sys 0.006s
I20250903 20:05:18.813450 27490 tablet_service.cc:1468] Processing CreateTablet for tablet e3e82f37c6fa4bebb0115e6087919a27 (DEFAULT_TABLE table=default.Table_480e84d976d54812a2a0ee3d43a4b08b [id=b94c022ba5184e3b9313546b163901a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.821866 27695 raft_consensus.cc:357] T 369c750ff5bf449e989d95ffd89497d4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.822093 27695 raft_consensus.cc:383] T 369c750ff5bf449e989d95ffd89497d4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:18.822190 27695 raft_consensus.cc:738] T 369c750ff5bf449e989d95ffd89497d4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:18.822476 27695 consensus_queue.cc:260] T 369c750ff5bf449e989d95ffd89497d4 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.823040 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebf0a9aaaea1408c8c9caf28f00dbff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.823060 27695 ts_tablet_manager.cc:1428] T 369c750ff5bf449e989d95ffd89497d4 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:18.823401 27695 tablet_bootstrap.cc:492] T 5fa7cc0c03234223af382982ca737e27 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:18.825176 27695 tablet_bootstrap.cc:654] T 5fa7cc0c03234223af382982ca737e27 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:18.826534 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cea603e8e184791a464dbbc13bbbd7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.810674 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet 512908bc4d6941ec81b145ee5795405f with delete_type TABLET_DATA_DELETED (replaced by tablet 1598824a733a4a93bed043e81b86abf3) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.811002 27210 tablet_service.cc:1468] Processing CreateTablet for tablet ebf0a9aaaea1408c8c9caf28f00dbff7 (DEFAULT_TABLE table=default.Table_e79d289ca0384e109873b8d5f2c78905 [id=d35bb69fba4e40c7901548fe0adff3f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.827385 27210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebf0a9aaaea1408c8c9caf28f00dbff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.829672 27692 tablet_bootstrap.cc:654] T fdc77810f27a40758e063b8b1150ab71 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:18.811038 27222 tablet_service.cc:1468] Processing CreateTablet for tablet 86e33bd235c14042a578a4a1c70f11b7 (DEFAULT_TABLE table=default.Table_3bcc60f1d7ed42d89cfaf71c0fc18fdd [id=14368dfbbc2a4f179dda569e352dabf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.830600 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86e33bd235c14042a578a4a1c70f11b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.830744 27492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47500175a642457687aa4226195fce47. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:18.831519 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 112c0dccb60c4c7c9efc164d9c8443ea consensus state has no leader
W20250903 20:05:18.834316 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4fa886f5be0f489290921e61dcaa4916 consensus state has no leader
I20250903 20:05:18.834409 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 145c82e8df2c49ea89fd2f02224d8cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.812917 27224 tablet_service.cc:1468] Processing CreateTablet for tablet 47500175a642457687aa4226195fce47 (DEFAULT_TABLE table=default.Table_fed56630d54540548ffb7f00228a1ee9 [id=f14a6e519cb9461286bff3839d58c50f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.834830 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_133bdffbe79346529dea857869c07825"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:18.835479 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47500175a642457687aa4226195fce47. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:18.835942 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_133bdffbe79346529dea857869c07825 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:18.814864 27212 tablet_service.cc:1468] Processing CreateTablet for tablet f36938c69f6a497f93e9458bdc9c40a0 (DEFAULT_TABLE table=default.Table_d1b7ba8a5d3440f7be2eb198bbf5e7fa [id=7badf47f3be84194b75babe421ed738a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.838407 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f36938c69f6a497f93e9458bdc9c40a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.789556 27684 raft_consensus.cc:2241] T 1396d4f8a93b412ca8397368fb98834e P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:18.838646 27212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f36938c69f6a497f93e9458bdc9c40a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.838740 27684 raft_consensus.cc:2270] T 1396d4f8a93b412ca8397368fb98834e P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:18.839401 27684 ts_tablet_manager.cc:1905] T 1396d4f8a93b412ca8397368fb98834e P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:18.840399 28208 leader_election.cc:290] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:18.841796 27693 ts_tablet_manager.cc:1428] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.051s user 0.001s sys 0.001s
W20250903 20:05:18.841905 27320 leader_election.cc:343] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: a10b079e7d5a440580ac00195fe88c65
I20250903 20:05:18.842442 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adea8b73e2e14cc5ae9490fc859dd0b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.815896 27216 tablet_service.cc:1468] Processing CreateTablet for tablet 145c82e8df2c49ea89fd2f02224d8cbf (DEFAULT_TABLE table=default.Table_9abad42b86c747b68c5d9ef37d8d2788 [id=c0de66acb6ad416887e23d19cdc5434b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.842887 27216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 145c82e8df2c49ea89fd2f02224d8cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.846434 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf99a74acfdb4a17bbbce3a07467e9c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.816156 27221 tablet_service.cc:1468] Processing CreateTablet for tablet e3e82f37c6fa4bebb0115e6087919a27 (DEFAULT_TABLE table=default.Table_480e84d976d54812a2a0ee3d43a4b08b [id=b94c022ba5184e3b9313546b163901a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.850584 27490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3e82f37c6fa4bebb0115e6087919a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.850765 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3e82f37c6fa4bebb0115e6087919a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.816612 27217 tablet_service.cc:1468] Processing CreateTablet for tablet cf99a74acfdb4a17bbbce3a07467e9c1 (DEFAULT_TABLE table=default.Table_4f815c3438474e3ca7b8e4b9d0be87d5 [id=70128985dd71449c95ece219f0a682fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.854472 27491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86e33bd235c14042a578a4a1c70f11b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.815901 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a10b079e7d5a440580ac00195fe88c65" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:18.815924 27227 tablet_service.cc:1468] Processing CreateTablet for tablet adea8b73e2e14cc5ae9490fc859dd0b1 (DEFAULT_TABLE table=default.Table_aef24ecf95294a9188a4338dfedc525d [id=2e5bd7fe0a5a4d38a3f8796a0f98941f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.817420 27211 tablet_service.cc:1468] Processing CreateTablet for tablet 9cea603e8e184791a464dbbc13bbbd7c (DEFAULT_TABLE table=default.Table_fd71464db83e4f1d8595472f6e3eb0b7 [id=e3996f93ec9f41b4b96701acd763f5be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.817870 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 26decef5841a4a3b85f2a5f578ae95dc (DEFAULT_TABLE table=default.Table_14e02c1be9c74bd99f4184faa6753f52 [id=a6dc7550a6d643008c92b3a59ba022ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.855537 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf99a74acfdb4a17bbbce3a07467e9c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.857419 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26decef5841a4a3b85f2a5f578ae95dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.863101 27211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cea603e8e184791a464dbbc13bbbd7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.875869 27248 raft_consensus.cc:2466] T a10b079e7d5a440580ac00195fe88c65 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:18.876264 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_86a571e18d164bd5beae624857d8526d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:18.876940 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86a571e18d164bd5beae624857d8526d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:18.877568 27489 tablet_service.cc:1468] Processing CreateTablet for tablet 849eb453d3e04b16bbb5df9017bfb2fd (DEFAULT_TABLE table=default.Table_86616565ede946a5b1043149012526f5 [id=d4f8932339764b218107c8fc674a7253]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.877703 27495 tablet_service.cc:1468] Processing CreateTablet for tablet 121a8ba102454cb4807483fac16e3781 (DEFAULT_TABLE table=default.Table_10c65228c77a4a9d8ccc535d2792f59f [id=0c587492cd8141fa97c0d6ea3faef6b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.878068 27489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 849eb453d3e04b16bbb5df9017bfb2fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.878357 27503 tablet_service.cc:1468] Processing CreateTablet for tablet 1453a8f6bd7b4b26b75d7fc08578a351 (DEFAULT_TABLE table=default.Table_e942cce7cfd5471dbbca333e31f4ba66 [id=a90b0f0944a347bf8587fb5f3ae11f3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.878729 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1453a8f6bd7b4b26b75d7fc08578a351. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.878628 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_dc971e5490844bea94ad3c6b8cf15d9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:18.879194 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc971e5490844bea94ad3c6b8cf15d9d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:18.880529 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e17243085e464908adf84082adbb3800" } new_table_name: "default.Table_6cc84d6d6ac64e75ab04f276ffbd6c79" modify_external_catalogs: true
I20250903 20:05:18.882838 27499 tablet_service.cc:1468] Processing CreateTablet for tablet 26decef5841a4a3b85f2a5f578ae95dc (DEFAULT_TABLE table=default.Table_14e02c1be9c74bd99f4184faa6753f52 [id=a6dc7550a6d643008c92b3a59ba022ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.883276 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 121a8ba102454cb4807483fac16e3781. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.884390 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26decef5841a4a3b85f2a5f578ae95dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.885721 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adea8b73e2e14cc5ae9490fc859dd0b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.886507 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_cfb651c1215b4325bcd5237fb8c1d7d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:18.887063 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cfb651c1215b4325bcd5237fb8c1d7d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:18.889433 27220 tablet_service.cc:1468] Processing CreateTablet for tablet 849eb453d3e04b16bbb5df9017bfb2fd (DEFAULT_TABLE table=default.Table_86616565ede946a5b1043149012526f5 [id=d4f8932339764b218107c8fc674a7253]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.896112 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_1c67ff00c09c4c3485ae43755a6f7c1f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:18.897548 27356 tablet_service.cc:1515] Processing DeleteTablet for tablet 5bc3b9771a0042fd87e706204dcbb712 with delete_type TABLET_DATA_DELETED (replaced by tablet 961ade36f7d14eba91bb16d5eae2e254) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:18.898927 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c67ff00c09c4c3485ae43755a6f7c1f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:18.901877 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 849eb453d3e04b16bbb5df9017bfb2fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.903537 27066 master_service.cc:948] ReplaceTablet: received request to replace tablet a38f65e992884635aae9942681c0dc4f from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:18.903599 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 121a8ba102454cb4807483fac16e3781 (DEFAULT_TABLE table=default.Table_10c65228c77a4a9d8ccc535d2792f59f [id=0c587492cd8141fa97c0d6ea3faef6b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.903743 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86e33bd235c14042a578a4a1c70f11b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.904281 28208 raft_consensus.cc:491] T 44db427ee53a4d0a907dfd63282d6d4f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:18.905542 27228 tablet_service.cc:1468] Processing CreateTablet for tablet 1453a8f6bd7b4b26b75d7fc08578a351 (DEFAULT_TABLE table=default.Table_e942cce7cfd5471dbbca333e31f4ba66 [id=a90b0f0944a347bf8587fb5f3ae11f3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.906003 27228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1453a8f6bd7b4b26b75d7fc08578a351. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.909462 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet 369c750ff5bf449e989d95ffd89497d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:18.910669 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 121a8ba102454cb4807483fac16e3781. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.917097 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 47500175a642457687aa4226195fce47 (DEFAULT_TABLE table=default.Table_fed56630d54540548ffb7f00228a1ee9 [id=f14a6e519cb9461286bff3839d58c50f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.918421 27693 tablet_bootstrap.cc:492] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:18.918752 27321 leader_election.cc:304] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:18.918685 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_c865210d32a243329721080cad1f1255"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:18.919135 27353 tablet_service.cc:1468] Processing CreateTablet for tablet f36938c69f6a497f93e9458bdc9c40a0 (DEFAULT_TABLE table=default.Table_d1b7ba8a5d3440f7be2eb198bbf5e7fa [id=7badf47f3be84194b75babe421ed738a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:18.919389 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c865210d32a243329721080cad1f1255 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:18.919507 28208 raft_consensus.cc:513] T 44db427ee53a4d0a907dfd63282d6d4f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:18.920140 28208 leader_election.cc:290] T 44db427ee53a4d0a907dfd63282d6d4f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:18.920312 28208 raft_consensus.cc:2802] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:18.920384 28208 raft_consensus.cc:491] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:18.920469 28208 raft_consensus.cc:3058] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:18.921355 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_41a441a055bb47d283b45107f9ac0385" } modify_external_catalogs: true
I20250903 20:05:18.921561 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_41a441a055bb47d283b45107f9ac0385" } modify_external_catalogs: true
I20250903 20:05:18.924847 27059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_143fea9b5e3d4acbb92adc441aeede47" } modify_external_catalogs: true
I20250903 20:05:18.925035 27059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_143fea9b5e3d4acbb92adc441aeede47" } modify_external_catalogs: true
I20250903 20:05:18.928864 27225 tablet_service.cc:1468] Processing CreateTablet for tablet c3c4f008ae4a422d9f6d0272097e9686 (DEFAULT_TABLE table=default.Table_3bf3d1e0cfbb43c2a04f1983c51e9192 [id=b2f143df31ea4e1785817dc5d75bbb9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.930974 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3c4f008ae4a422d9f6d0272097e9686. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:18.932426 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6d21f38759d743af972635f8b5b8d8d2 consensus state has no leader
I20250903 20:05:18.932612 27213 tablet_service.cc:1468] Processing CreateTablet for tablet f573a52b7d7843e486fcec3530878c65 (DEFAULT_TABLE table=default.Table_6353cdf28e1648ffa45cef826bdfffc3 [id=50a549e94cf94dceb8960c55d6042508]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.932998 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f573a52b7d7843e486fcec3530878c65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.952703 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d43fc55de38641f6946c7dab57874b91" } modify_external_catalogs: true
I20250903 20:05:18.952772 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_7713d2cb69d74ea895a8c86a04ff9c50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:18.952968 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d43fc55de38641f6946c7dab57874b91" } modify_external_catalogs: true
W20250903 20:05:18.953402 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7713d2cb69d74ea895a8c86a04ff9c50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:18.956866 27223 tablet_service.cc:1468] Processing CreateTablet for tablet 2dc0f714cdb44ece90615de5a11bc71f (DEFAULT_TABLE table=default.Table_b1d2232da9bd4385b1b65355d3be0b4d [id=058c8e9e4c3f46b8b20739a0dfac7bb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.965360 27229 tablet_service.cc:1468] Processing CreateTablet for tablet 74eab6ad2fe64b18bd00c0d5524428c6 (DEFAULT_TABLE table=default.Table_739bc0d4c1d4473caac21639a4ade167 [id=8e55d7fd93b949dd90b9aed4870d188c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.969448 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet f1c3c8e46a2b40ccb8d5bb61843cecd1
I20250903 20:05:18.972913 27219 tablet_service.cc:1468] Processing CreateTablet for tablet f1c3c8e46a2b40ccb8d5bb61843cecd1 (DEFAULT_TABLE table=default.Table_41a441a055bb47d283b45107f9ac0385 [id=0e4c47216f324dfabfa0d266eded3484]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.977428 27226 tablet_service.cc:1468] Processing CreateTablet for tablet 6853839e73e343a0a54aa400a23db35c (DEFAULT_TABLE table=default.Table_be46ef5a0c524f32b001a4fd218d497f [id=e66c3ed2e84d4d02941efee1fcbd8ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.979687 27215 tablet_service.cc:1468] Processing CreateTablet for tablet f05ef6e9eac24f7ebebd67aacd4c5c80 (DEFAULT_TABLE table=default.Table_d89675ff457643349b686394094cc04a [id=9099d7e6aff548ab8a566a091e607d5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:18.979892 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d9878f04c3404885815ae53a9054de14" } modify_external_catalogs: true
I20250903 20:05:18.980089 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d9878f04c3404885815ae53a9054de14" } modify_external_catalogs: true
I20250903 20:05:18.981503 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c865210d32a243329721080cad1f1255" } new_table_name: "default.Table_1e9db7589b3a4cbe94da871dad467381" modify_external_catalogs: true
I20250903 20:05:18.982331 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_75f638a7c40e48e584a02949e6f864fd" } modify_external_catalogs: true
I20250903 20:05:18.984364 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_09e3fcf1b8ba4f75be49736c7c4b5f30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:18.984905 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09e3fcf1b8ba4f75be49736c7c4b5f30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:18.985229 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_5d5e2d80c015418db7956702c08cac42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:18.985766 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d5e2d80c015418db7956702c08cac42 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:18.989794 27223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dc0f714cdb44ece90615de5a11bc71f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.990753 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f05ef6e9eac24f7ebebd67aacd4c5c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:18.994323 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_75f638a7c40e48e584a02949e6f864fd" } modify_external_catalogs: true
I20250903 20:05:18.995069 27229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74eab6ad2fe64b18bd00c0d5524428c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.000854 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet f1c3c8e46a2b40ccb8d5bb61843cecd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:19.001363 27219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1c3c8e46a2b40ccb8d5bb61843cecd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.001664 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6853839e73e343a0a54aa400a23db35c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.003065 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet bf2545037c214bb1b5060575e3cdf81a
I20250903 20:05:19.005515 27227 tablet_service.cc:1468] Processing CreateTablet for tablet a4fb5fb54cb740698336598c09b58699 (DEFAULT_TABLE table=default.Table_9fa3fddcea2e482baf53322b4072b3b6 [id=5d5759a8aece4433ba5544f2de1df0d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.007200 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6f10e1448c2c4efabebd7d91b7bf5d79" } new_table_name: "default.Table_0fef1d019eff4323b7ba56368614aadc" modify_external_catalogs: true
I20250903 20:05:19.009136 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 8400e23873fd4be18de2998afc64a689
I20250903 20:05:19.011823 28030 raft_consensus.cc:491] T bf2545037c214bb1b5060575e3cdf81a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:19.012037 28030 raft_consensus.cc:513] T bf2545037c214bb1b5060575e3cdf81a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.015758 27352 tablet_service.cc:1468] Processing CreateTablet for tablet adea8b73e2e14cc5ae9490fc859dd0b1 (DEFAULT_TABLE table=default.Table_aef24ecf95294a9188a4338dfedc525d [id=2e5bd7fe0a5a4d38a3f8796a0f98941f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.016568 28030 leader_election.cc:290] T bf2545037c214bb1b5060575e3cdf81a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:19.020915 27349 tablet_service.cc:1468] Processing CreateTablet for tablet ebf0a9aaaea1408c8c9caf28f00dbff7 (DEFAULT_TABLE table=default.Table_e79d289ca0384e109873b8d5f2c78905 [id=d35bb69fba4e40c7901548fe0adff3f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.021905 27361 tablet_service.cc:1468] Processing CreateTablet for tablet cf99a74acfdb4a17bbbce3a07467e9c1 (DEFAULT_TABLE table=default.Table_4f815c3438474e3ca7b8e4b9d0be87d5 [id=70128985dd71449c95ece219f0a682fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.023085 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 4e99665f920f4de590bd6d9e918c7182 (DEFAULT_TABLE table=default.Table_2bbccfd73e824cdcb9d9d13e7da829c3 [id=748b11b0bb3f4b13b45ce7d82cf8624f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.023121 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 145c82e8df2c49ea89fd2f02224d8cbf (DEFAULT_TABLE table=default.Table_9abad42b86c747b68c5d9ef37d8d2788 [id=c0de66acb6ad416887e23d19cdc5434b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.023660 27351 tablet_service.cc:1468] Processing CreateTablet for tablet 9cea603e8e184791a464dbbc13bbbd7c (DEFAULT_TABLE table=default.Table_fd71464db83e4f1d8595472f6e3eb0b7 [id=e3996f93ec9f41b4b96701acd763f5be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.025540 27356 tablet_service.cc:1468] Processing CreateTablet for tablet e3e82f37c6fa4bebb0115e6087919a27 (DEFAULT_TABLE table=default.Table_480e84d976d54812a2a0ee3d43a4b08b [id=b94c022ba5184e3b9313546b163901a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.025882 27212 tablet_service.cc:1468] Processing CreateTablet for tablet 3af0c66a8c234711a43816f248e5be03 (DEFAULT_TABLE table=default.Table_5ca5478ef07f4acdbbedb24af75f9e04 [id=7fb235b85bba4f7ab1b77bc5d2b9b58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.026479 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 64c4dd2645034a7eaf243f5474603f10
I20250903 20:05:19.026819 27366 tablet_service.cc:1468] Processing CreateTablet for tablet 26decef5841a4a3b85f2a5f578ae95dc (DEFAULT_TABLE table=default.Table_14e02c1be9c74bd99f4184faa6753f52 [id=a6dc7550a6d643008c92b3a59ba022ce]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:19.027273 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f31e3e5ea72940de953c7dd787515c62 has no consensus state
I20250903 20:05:19.027940 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 90baf83357494a0aae325908c118b471
I20250903 20:05:19.026530 28208 raft_consensus.cc:513] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.031719 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47500175a642457687aa4226195fce47. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:19.034701 27321 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 156648us
I20250903 20:05:19.035230 27213 tablet_service.cc:1468] Processing CreateTablet for tablet 36e4f956f26c45989ca02ee0cd02cd87 (DEFAULT_TABLE table=default.Table_c2bb0708a3bc47a3a1a3dbbae0964cd0 [id=ad28beb47f4e452a9c15a3c63d27e1e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.035763 27210 tablet_service.cc:1468] Processing CreateTablet for tablet 4789bd72aa454b639df76bd5ad379850 (DEFAULT_TABLE table=default.Table_362ac8cb413e426ebf154eae4b6b5c7e [id=10ef25498f7448dbb19cd66d4c6ba713]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.035905 27355 tablet_service.cc:1468] Processing CreateTablet for tablet 849eb453d3e04b16bbb5df9017bfb2fd (DEFAULT_TABLE table=default.Table_86616565ede946a5b1043149012526f5 [id=d4f8932339764b218107c8fc674a7253]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.036135 27216 tablet_service.cc:1468] Processing CreateTablet for tablet c6cebb39bdb148d1bcbb47c4160a7be4 (DEFAULT_TABLE table=default.Table_2b5e97597b354141839a4ec0ca6bf046 [id=ca81d81b44c4465bb6e5932bd23b089f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.036479 27228 tablet_service.cc:1468] Processing CreateTablet for tablet 6a881014b2b14d699e3405b22bdad5d1 (DEFAULT_TABLE table=default.Table_9d9f52b616d74e7396568e15ecaa1f34 [id=8e1c13edc87d447b817824aba6401a60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.036564 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a10b079e7d5a440580ac00195fe88c65" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:19.036785 27248 raft_consensus.cc:3058] T a10b079e7d5a440580ac00195fe88c65 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:19.038275 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 9a71290c27ac4a22bd9aa8e14e5834f5 (DEFAULT_TABLE table=default.Table_c4a34f406b12427ea406d72fc1be9db9 [id=880ecbdbc8714bd1bb334401f23177b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.039412 27364 tablet_service.cc:1468] Processing CreateTablet for tablet 121a8ba102454cb4807483fac16e3781 (DEFAULT_TABLE table=default.Table_10c65228c77a4a9d8ccc535d2792f59f [id=0c587492cd8141fa97c0d6ea3faef6b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.040104 27211 tablet_service.cc:1468] Processing CreateTablet for tablet e0b9172ebb484382adb86e597f5cbc3b (DEFAULT_TABLE table=default.Table_eb74cc68ea8b4ab0bb7eb3937ac1aa4d [id=81969f6cf3404aa2ac397fb44b0e615d]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:19.042754 27321 leader_election.cc:343] T 44db427ee53a4d0a907dfd63282d6d4f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 44db427ee53a4d0a907dfd63282d6d4f
I20250903 20:05:19.043478 27347 tablet_service.cc:1468] Processing CreateTablet for tablet f1c3c8e46a2b40ccb8d5bb61843cecd1 (DEFAULT_TABLE table=default.Table_41a441a055bb47d283b45107f9ac0385 [id=0e4c47216f324dfabfa0d266eded3484]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.044449 27687 ts_tablet_manager.cc:1918] T c9c870fb14e3469fb2979d8462b49121 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:19.044628 27687 log.cc:1199] T c9c870fb14e3469fb2979d8462b49121 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/c9c870fb14e3469fb2979d8462b49121
I20250903 20:05:19.046195 27359 tablet_service.cc:1468] Processing CreateTablet for tablet 6853839e73e343a0a54aa400a23db35c (DEFAULT_TABLE table=default.Table_be46ef5a0c524f32b001a4fd218d497f [id=e66c3ed2e84d4d02941efee1fcbd8ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.048015 27248 raft_consensus.cc:2466] T a10b079e7d5a440580ac00195fe88c65 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:19.048453 27357 tablet_service.cc:1468] Processing CreateTablet for tablet 2dc0f714cdb44ece90615de5a11bc71f (DEFAULT_TABLE table=default.Table_b1d2232da9bd4385b1b65355d3be0b4d [id=058c8e9e4c3f46b8b20739a0dfac7bb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.048642 27321 leader_election.cc:304] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:19.048794 27363 tablet_service.cc:1468] Processing CreateTablet for tablet 74eab6ad2fe64b18bd00c0d5524428c6 (DEFAULT_TABLE table=default.Table_739bc0d4c1d4473caac21639a4ade167 [id=8e55d7fd93b949dd90b9aed4870d188c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.049136 27350 tablet_service.cc:1468] Processing CreateTablet for tablet f05ef6e9eac24f7ebebd67aacd4c5c80 (DEFAULT_TABLE table=default.Table_d89675ff457643349b686394094cc04a [id=9099d7e6aff548ab8a566a091e607d5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.049662 27354 tablet_service.cc:1468] Processing CreateTablet for tablet 1453a8f6bd7b4b26b75d7fc08578a351 (DEFAULT_TABLE table=default.Table_e942cce7cfd5471dbbca333e31f4ba66 [id=a90b0f0944a347bf8587fb5f3ae11f3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.051857 28171 raft_consensus.cc:491] T e8bc99de4ffe43d58351d170b7c61d8a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:19.052073 28171 raft_consensus.cc:513] T e8bc99de4ffe43d58351d170b7c61d8a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.055003 28234 raft_consensus.cc:2802] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:19.055395 27362 tablet_service.cc:1468] Processing CreateTablet for tablet f573a52b7d7843e486fcec3530878c65 (DEFAULT_TABLE table=default.Table_6353cdf28e1648ffa45cef826bdfffc3 [id=50a549e94cf94dceb8960c55d6042508]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.056981 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 8330ab3e4b6f4560b9e41a9df280c211 (DEFAULT_TABLE table=default.Table_83e2f4211b724a698cf478ddde974155 [id=07e066ded3ea4623958b36fd3874a354]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.062273 27224 tablet_service.cc:1468] Processing CreateTablet for tablet 1598824a733a4a93bed043e81b86abf3 (DEFAULT_TABLE table=default.Table_d9276aa6946f41d7ac0fabc39338c23d [id=60e7bfe4938540e4a3c34b248356969f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.062824 27220 tablet_service.cc:1468] Processing CreateTablet for tablet eb878263eda9447e870a782907fe6a8d (DEFAULT_TABLE table=default.Table_647a8fb3a7d74bd48d49b76c3266b37f [id=e58c667578894d8795364da6dde5d8c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.078564 27358 tablet_service.cc:1468] Processing CreateTablet for tablet c3c4f008ae4a422d9f6d0272097e9686 (DEFAULT_TABLE table=default.Table_3bf3d1e0cfbb43c2a04f1983c51e9192 [id=b2f143df31ea4e1785817dc5d75bbb9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.081323 28208 leader_election.cc:290] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:19.081614 28208 raft_consensus.cc:491] T 264127511a234e9fbf409f3aa2860ccc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:19.084192 28208 raft_consensus.cc:513] T 264127511a234e9fbf409f3aa2860ccc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:19.084273 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_c749638286ff4a0db8b80342622dca67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:19.085086 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c749638286ff4a0db8b80342622dca67 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:19.086266 28208 leader_election.cc:290] T 264127511a234e9fbf409f3aa2860ccc P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:19.089637 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_4126233ea8a445a8837131a6052aca70" } new_table_name: "default.Table_4d303308b63c4ce09056e226d1a30498" modify_external_catalogs: true
I20250903 20:05:19.091277 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_20c48527dac44af39c438cb74d86dcc4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:19.091929 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20c48527dac44af39c438cb74d86dcc4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:19.095188 27687 ts_tablet_manager.cc:1939] T c9c870fb14e3469fb2979d8462b49121 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:19.096112 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2746c6354cbc45c296e45155908acb77" } new_table_name: "default.Table_e0f0013beb934622b6c69bbf05cc1ede" modify_external_catalogs: true
I20250903 20:05:19.096323 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 484) took 1061 ms. Trace:
I20250903 20:05:19.102638 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4fb5fb54cb740698336598c09b58699. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.106959 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_73e9595bc2864cbfa76e06557a740f77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:19.107749 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73e9595bc2864cbfa76e06557a740f77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:19.109409 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_00da0bb3fb0c4aa1b29a87d3c8844955"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:19.110018 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00da0bb3fb0c4aa1b29a87d3c8844955 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:19.111512 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb878263eda9447e870a782907fe6a8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.112569 27353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f36938c69f6a497f93e9458bdc9c40a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.113374 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3c4f008ae4a422d9f6d0272097e9686. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.114168 27355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 849eb453d3e04b16bbb5df9017bfb2fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.118446 27216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6cebb39bdb148d1bcbb47c4160a7be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.122113 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e99665f920f4de590bd6d9e918c7182. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.124096 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1598824a733a4a93bed043e81b86abf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.126926 27384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf2545037c214bb1b5060575e3cdf81a" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:19.127545 27321 leader_election.cc:343] T 264127511a234e9fbf409f3aa2860ccc P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 264127511a234e9fbf409f3aa2860ccc
I20250903 20:05:19.127933 27687 rpcz_store.cc:276] 0903 20:05:18.035028 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.035090 (+ 62us) service_pool.cc:225] Handling call
0903 20:05:18.795864 (+760774us) ts_tablet_manager.cc:1145] Deleting tablet c9c870fb14e3469fb2979d8462b49121
0903 20:05:18.795870 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:18.809572 (+ 13702us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:19.039448 (+229876us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:19.096273 (+ 56825us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":760502}
I20250903 20:05:19.128533 27687 tablet_replica.cc:331] T 44db427ee53a4d0a907dfd63282d6d4f P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:19.128832 27687 raft_consensus.cc:2241] T 44db427ee53a4d0a907dfd63282d6d4f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:19.129096 27687 raft_consensus.cc:2270] T 44db427ee53a4d0a907dfd63282d6d4f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:19.129799 27687 ts_tablet_manager.cc:1905] T 44db427ee53a4d0a907dfd63282d6d4f P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:19.130043 27354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1453a8f6bd7b4b26b75d7fc08578a351. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.130344 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f05ef6e9eac24f7ebebd67aacd4c5c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.131027 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1c3c8e46a2b40ccb8d5bb61843cecd1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:19.130734 27184 leader_election.cc:343] T bf2545037c214bb1b5060575e3cdf81a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:19.131356 27211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0b9172ebb484382adb86e597f5cbc3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.132601 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e4f956f26c45989ca02ee0cd02cd87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.133734 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adea8b73e2e14cc5ae9490fc859dd0b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.134343 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f573a52b7d7843e486fcec3530878c65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.124944 27229 tablet_service.cc:1515] Processing DeleteTablet for tablet bf2545037c214bb1b5060575e3cdf81a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:19.137056 27212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3af0c66a8c234711a43816f248e5be03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.137775 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6853839e73e343a0a54aa400a23db35c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.137997 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dc0f714cdb44ece90615de5a11bc71f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.138445 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_4d67ac5e5c6645f69648aea15fa389ef" } new_table_name: "default.Table_bf733feb53404025b54ce4e7437a2579" modify_external_catalogs: true
I20250903 20:05:19.140404 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74eab6ad2fe64b18bd00c0d5524428c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.142083 27351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cea603e8e184791a464dbbc13bbbd7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.145255 27210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4789bd72aa454b639df76bd5ad379850. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.146062 27695 tablet_bootstrap.cc:492] T 5fa7cc0c03234223af382982ca737e27 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:19.146265 27695 ts_tablet_manager.cc:1397] T 5fa7cc0c03234223af382982ca737e27 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.323s user 0.012s sys 0.056s
I20250903 20:05:19.147189 27695 raft_consensus.cc:357] T 5fa7cc0c03234223af382982ca737e27 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.147408 27695 raft_consensus.cc:383] T 5fa7cc0c03234223af382982ca737e27 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:19.147502 27695 raft_consensus.cc:738] T 5fa7cc0c03234223af382982ca737e27 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:19.147784 27695 consensus_queue.cc:260] T 5fa7cc0c03234223af382982ca737e27 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.150408 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 121a8ba102454cb4807483fac16e3781. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.150275 28208 raft_consensus.cc:491] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:19.150429 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_aa3dd415897242eb828a0b64f8fc6ad7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:19.150669 28208 raft_consensus.cc:513] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:19.151121 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa3dd415897242eb828a0b64f8fc6ad7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:19.153123 28208 leader_election.cc:290] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:19.153379 28208 raft_consensus.cc:491] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:19.153630 28208 raft_consensus.cc:513] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.154371 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3e82f37c6fa4bebb0115e6087919a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.155056 28208 leader_election.cc:290] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:19.157598 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8330ab3e4b6f4560b9e41a9df280c211. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.158625 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26decef5841a4a3b85f2a5f578ae95dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.158572 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e8bc99de4ffe43d58351d170b7c61d8a" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:19.164512 27693 tablet_bootstrap.cc:654] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:19.164927 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 145c82e8df2c49ea89fd2f02224d8cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.170598 27695 ts_tablet_manager.cc:1428] T 5fa7cc0c03234223af382982ca737e27 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250903 20:05:19.170917 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf99a74acfdb4a17bbbce3a07467e9c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.174372 27245 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecd339ec12e84e88acec427630cc8668" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:19.174928 27248 raft_consensus.cc:2466] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:19.175170 27245 raft_consensus.cc:2466] T ecd339ec12e84e88acec427630cc8668 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:19.175423 28030 raft_consensus.cc:491] T 3f78a951bb024cf290116b0fe438ddb4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:19.177402 27364 tablet_service.cc:1468] Processing CreateTablet for tablet 36e4f956f26c45989ca02ee0cd02cd87 (DEFAULT_TABLE table=default.Table_c2bb0708a3bc47a3a1a3dbbae0964cd0 [id=ad28beb47f4e452a9c15a3c63d27e1e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.178000 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e4f956f26c45989ca02ee0cd02cd87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.178094 27321 leader_election.cc:304] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:19.178485 27321 leader_election.cc:304] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:19.182662 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b60456f0147b407584bb3aa4321b1fe0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:19.183413 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b60456f0147b407584bb3aa4321b1fe0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:19.184746 27695 tablet_bootstrap.cc:492] T 07565ea2b5f34dcfb407ff3bfde17243 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:19.185854 27355 tablet_service.cc:1468] Processing CreateTablet for tablet 6a881014b2b14d699e3405b22bdad5d1 (DEFAULT_TABLE table=default.Table_9d9f52b616d74e7396568e15ecaa1f34 [id=8e1c13edc87d447b817824aba6401a60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.191551 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet c9c870fb14e3469fb2979d8462b49121 (table default.Table_8b2ba396ac1b424493dbc17297d2d133 [id=3403a412096a4ef98878426a43899d7c]) successfully deleted
I20250903 20:05:19.199429 28030 raft_consensus.cc:513] T 3f78a951bb024cf290116b0fe438ddb4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:19.199457 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet 64c4dd2645034a7eaf243f5474603f10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:19.200635 28030 leader_election.cc:290] T 3f78a951bb024cf290116b0fe438ddb4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:19.202001 27384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f78a951bb024cf290116b0fe438ddb4" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:19.202353 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet 90baf83357494a0aae325908c118b471 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:19.203889 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a71290c27ac4a22bd9aa8e14e5834f5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:19.206060 27184 leader_election.cc:343] T 3f78a951bb024cf290116b0fe438ddb4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:19.214368 27356 tablet_service.cc:1515] Processing DeleteTablet for tablet f1c3c8e46a2b40ccb8d5bb61843cecd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:19.216126 27356 tablet_service.cc:1468] Processing CreateTablet for tablet 4789bd72aa454b639df76bd5ad379850 (DEFAULT_TABLE table=default.Table_362ac8cb413e426ebf154eae4b6b5c7e [id=10ef25498f7448dbb19cd66d4c6ba713]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.217329 27355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a881014b2b14d699e3405b22bdad5d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.221859 27350 tablet_service.cc:1468] Processing CreateTablet for tablet c6cebb39bdb148d1bcbb47c4160a7be4 (DEFAULT_TABLE table=default.Table_2b5e97597b354141839a4ec0ca6bf046 [id=ca81d81b44c4465bb6e5932bd23b089f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.197829 27359 tablet_service.cc:1468] Processing CreateTablet for tablet 9a71290c27ac4a22bd9aa8e14e5834f5 (DEFAULT_TABLE table=default.Table_c4a34f406b12427ea406d72fc1be9db9 [id=880ecbdbc8714bd1bb334401f23177b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.222292 28240 raft_consensus.cc:2802] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:19.222961 28240 raft_consensus.cc:491] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:19.223181 28240 raft_consensus.cc:3058] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:19.223430 27352 tablet_service.cc:1468] Processing CreateTablet for tablet a4fb5fb54cb740698336598c09b58699 (DEFAULT_TABLE table=default.Table_9fa3fddcea2e482baf53322b4072b3b6 [id=5d5759a8aece4433ba5544f2de1df0d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.224720 27363 tablet_service.cc:1468] Processing CreateTablet for tablet 3af0c66a8c234711a43816f248e5be03 (DEFAULT_TABLE table=default.Table_5ca5478ef07f4acdbbedb24af75f9e04 [id=7fb235b85bba4f7ab1b77bc5d2b9b58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.187961 27348 tablet_service.cc:1468] Processing CreateTablet for tablet 4e99665f920f4de590bd6d9e918c7182 (DEFAULT_TABLE table=default.Table_2bbccfd73e824cdcb9d9d13e7da829c3 [id=748b11b0bb3f4b13b45ce7d82cf8624f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.245329 27695 tablet_bootstrap.cc:654] T 07565ea2b5f34dcfb407ff3bfde17243 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:19.247345 28240 raft_consensus.cc:513] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.254415 27492 tablet_service.cc:1515] Processing DeleteTablet for tablet ecd339ec12e84e88acec427630cc8668 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:19.257750 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf2545037c214bb1b5060575e3cdf81a" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:19.267839 27523 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f78a951bb024cf290116b0fe438ddb4" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:19.287230 28171 leader_election.cc:290] T e8bc99de4ffe43d58351d170b7c61d8a P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:19.289443 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 8400e23873fd4be18de2998afc64a689 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:19.290458 28030 raft_consensus.cc:491] T fbfabb6ff1714d2e921f45256b543759 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:19.290655 28030 raft_consensus.cc:513] T fbfabb6ff1714d2e921f45256b543759 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.291324 27228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a881014b2b14d699e3405b22bdad5d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.280659 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebf0a9aaaea1408c8c9caf28f00dbff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.297096 28208 raft_consensus.cc:2802] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:19.298653 27692 tablet_bootstrap.cc:492] T fdc77810f27a40758e063b8b1150ab71 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:19.298873 27692 ts_tablet_manager.cc:1397] T fdc77810f27a40758e063b8b1150ab71 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.479s user 0.069s sys 0.102s
I20250903 20:05:19.300009 27488 tablet_service.cc:1468] Processing CreateTablet for tablet c3c4f008ae4a422d9f6d0272097e9686 (DEFAULT_TABLE table=default.Table_3bf3d1e0cfbb43c2a04f1983c51e9192 [id=b2f143df31ea4e1785817dc5d75bbb9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.300068 27493 tablet_service.cc:1468] Processing CreateTablet for tablet 2dc0f714cdb44ece90615de5a11bc71f (DEFAULT_TABLE table=default.Table_b1d2232da9bd4385b1b65355d3be0b4d [id=058c8e9e4c3f46b8b20739a0dfac7bb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.300537 27496 tablet_service.cc:1468] Processing CreateTablet for tablet f573a52b7d7843e486fcec3530878c65 (DEFAULT_TABLE table=default.Table_6353cdf28e1648ffa45cef826bdfffc3 [id=50a549e94cf94dceb8960c55d6042508]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.300586 27358 tablet_service.cc:1468] Processing CreateTablet for tablet 8330ab3e4b6f4560b9e41a9df280c211 (DEFAULT_TABLE table=default.Table_83e2f4211b724a698cf478ddde974155 [id=07e066ded3ea4623958b36fd3874a354]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.300537 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dc0f714cdb44ece90615de5a11bc71f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.301256 27492 tablet_service.cc:1468] Processing CreateTablet for tablet 74eab6ad2fe64b18bd00c0d5524428c6 (DEFAULT_TABLE table=default.Table_739bc0d4c1d4473caac21639a4ade167 [id=8e55d7fd93b949dd90b9aed4870d188c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.301054 28240 leader_election.cc:290] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:19.301486 28240 raft_consensus.cc:491] T eced730bdbc14a8c9f6b7b7e53435fd0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:19.301584 27492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74eab6ad2fe64b18bd00c0d5524428c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.301642 28240 raft_consensus.cc:513] T eced730bdbc14a8c9f6b7b7e53435fd0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.302415 28240 leader_election.cc:290] T eced730bdbc14a8c9f6b7b7e53435fd0 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:19.302587 28240 raft_consensus.cc:491] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:19.302711 28240 raft_consensus.cc:513] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.303309 28240 leader_election.cc:290] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:19.303440 28240 raft_consensus.cc:491] T 6ea37668d64d4feeb92a162711609c13 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:19.303557 28240 raft_consensus.cc:513] T 6ea37668d64d4feeb92a162711609c13 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.304078 28240 leader_election.cc:290] T 6ea37668d64d4feeb92a162711609c13 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:19.306499 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f573a52b7d7843e486fcec3530878c65. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:19.306427 27320 leader_election.cc:343] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: a10b079e7d5a440580ac00195fe88c65
I20250903 20:05:19.306496 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3af0c66a8c234711a43816f248e5be03. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:19.306871 27320 leader_election.cc:343] T 44db427ee53a4d0a907dfd63282d6d4f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 44db427ee53a4d0a907dfd63282d6d4f
I20250903 20:05:19.307019 27320 leader_election.cc:304] T 44db427ee53a4d0a907dfd63282d6d4f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:19.307243 27320 raft_consensus.cc:2684] T 44db427ee53a4d0a907dfd63282d6d4f P 51578fb7bceb4e119b3ab1e58c4f5528: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250903 20:05:19.308065 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3c4f008ae4a422d9f6d0272097e9686. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.310400 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8330ab3e4b6f4560b9e41a9df280c211. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.314549 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4fb5fb54cb740698336598c09b58699. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.314994 27492 tablet_service.cc:1468] Processing CreateTablet for tablet 6853839e73e343a0a54aa400a23db35c (DEFAULT_TABLE table=default.Table_be46ef5a0c524f32b001a4fd218d497f [id=e66c3ed2e84d4d02941efee1fcbd8ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.315438 27492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6853839e73e343a0a54aa400a23db35c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.318626 27493 tablet_service.cc:1468] Processing CreateTablet for tablet f1c3c8e46a2b40ccb8d5bb61843cecd1 (DEFAULT_TABLE table=default.Table_41a441a055bb47d283b45107f9ac0385 [id=0e4c47216f324dfabfa0d266eded3484]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.320134 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1c3c8e46a2b40ccb8d5bb61843cecd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.322490 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4789bd72aa454b639df76bd5ad379850. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.328641 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e99665f920f4de590bd6d9e918c7182. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.330824 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6cebb39bdb148d1bcbb47c4160a7be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.331931 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a71290c27ac4a22bd9aa8e14e5834f5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:19.339350 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6238740ce1894e899be9a40321f0ab73 consensus state has no leader
I20250903 20:05:19.345064 27354 tablet_service.cc:1468] Processing CreateTablet for tablet 1598824a733a4a93bed043e81b86abf3 (DEFAULT_TABLE table=default.Table_d9276aa6946f41d7ac0fabc39338c23d [id=60e7bfe4938540e4a3c34b248356969f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:19.345885 27051 consensus_peers.cc:489] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 -> Peer 35dd69f3fa7a4fd8843b97f8a5d68296 (127.26.65.125:44059): Couldn't send request to peer 35dd69f3fa7a4fd8843b97f8a5d68296. Status: Network error: Client connection negotiation failed: client connection to 127.26.65.125:44059: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250903 20:05:19.345949 27347 tablet_service.cc:1468] Processing CreateTablet for tablet eb878263eda9447e870a782907fe6a8d (DEFAULT_TABLE table=default.Table_647a8fb3a7d74bd48d49b76c3266b37f [id=e58c667578894d8795364da6dde5d8c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.346649 28208 raft_consensus.cc:491] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250903 20:05:19.350420 27183 leader_election.cc:343] T bf2545037c214bb1b5060575e3cdf81a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:19.354233 27245 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e8bc99de4ffe43d58351d170b7c61d8a" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:19.354475 27245 raft_consensus.cc:3058] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:19.351385 28030 leader_election.cc:290] T fbfabb6ff1714d2e921f45256b543759 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:19.356650 27183 leader_election.cc:304] T bf2545037c214bb1b5060575e3cdf81a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:19.360105 28030 raft_consensus.cc:491] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:19.364305 27354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1598824a733a4a93bed043e81b86abf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.364342 27384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fbfabb6ff1714d2e921f45256b543759" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:19.364373 28208 raft_consensus.cc:3058] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:19.366041 27184 leader_election.cc:343] T fbfabb6ff1714d2e921f45256b543759 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:19.366477 27183 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 99544.5us
W20250903 20:05:19.366925 27183 leader_election.cc:343] T 3f78a951bb024cf290116b0fe438ddb4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:19.367434 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 4e99665f920f4de590bd6d9e918c7182 (DEFAULT_TABLE table=default.Table_2bbccfd73e824cdcb9d9d13e7da829c3 [id=748b11b0bb3f4b13b45ce7d82cf8624f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.368008 27183 leader_election.cc:304] T 3f78a951bb024cf290116b0fe438ddb4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:19.369867 28030 raft_consensus.cc:513] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.370605 28030 leader_election.cc:290] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:19.373373 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb878263eda9447e870a782907fe6a8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.374033 27523 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fbfabb6ff1714d2e921f45256b543759" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:19.375218 28030 raft_consensus.cc:2747] T bf2545037c214bb1b5060575e3cdf81a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:19.375468 28030 raft_consensus.cc:2747] T 3f78a951bb024cf290116b0fe438ddb4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:19.376268 27495 tablet_service.cc:1468] Processing CreateTablet for tablet 6a881014b2b14d699e3405b22bdad5d1 (DEFAULT_TABLE table=default.Table_9d9f52b616d74e7396568e15ecaa1f34 [id=8e1c13edc87d447b817824aba6401a60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.377321 27685 ts_tablet_manager.cc:1918] T eced730bdbc14a8c9f6b7b7e53435fd0 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:19.378070 27500 tablet_service.cc:1468] Processing CreateTablet for tablet f05ef6e9eac24f7ebebd67aacd4c5c80 (DEFAULT_TABLE table=default.Table_d89675ff457643349b686394094cc04a [id=9099d7e6aff548ab8a566a091e607d5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.382421 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e99665f920f4de590bd6d9e918c7182. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.382718 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a881014b2b14d699e3405b22bdad5d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.385397 27693 tablet_bootstrap.cc:492] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:19.385601 27693 ts_tablet_manager.cc:1397] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.543s user 0.005s sys 0.018s
I20250903 20:05:19.385895 27384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a76a17b64a74fbfbb4f2e92a02ae1cd" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:19.385982 27490 tablet_service.cc:1468] Processing CreateTablet for tablet 9a71290c27ac4a22bd9aa8e14e5834f5 (DEFAULT_TABLE table=default.Table_c4a34f406b12427ea406d72fc1be9db9 [id=880ecbdbc8714bd1bb334401f23177b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.386004 27499 tablet_service.cc:1468] Processing CreateTablet for tablet 36e4f956f26c45989ca02ee0cd02cd87 (DEFAULT_TABLE table=default.Table_c2bb0708a3bc47a3a1a3dbbae0964cd0 [id=ad28beb47f4e452a9c15a3c63d27e1e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.389242 27684 ts_tablet_manager.cc:1918] T 1396d4f8a93b412ca8397368fb98834e P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:19.389607 27684 log.cc:1199] T 1396d4f8a93b412ca8397368fb98834e P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/1396d4f8a93b412ca8397368fb98834e
I20250903 20:05:19.390995 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f05ef6e9eac24f7ebebd67aacd4c5c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.399899 27684 ts_tablet_manager.cc:1939] T 1396d4f8a93b412ca8397368fb98834e P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:19.403098 27352 tablet_service.cc:1468] Processing CreateTablet for tablet e0b9172ebb484382adb86e597f5cbc3b (DEFAULT_TABLE table=default.Table_eb74cc68ea8b4ab0bb7eb3937ac1aa4d [id=81969f6cf3404aa2ac397fb44b0e615d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.404245 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0b9172ebb484382adb86e597f5cbc3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.408968 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet bf2545037c214bb1b5060575e3cdf81a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:19.409538 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 8400e23873fd4be18de2998afc64a689 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:19.410652 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 64c4dd2645034a7eaf243f5474603f10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:19.411332 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 90baf83357494a0aae325908c118b471 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:19.417032 27692 raft_consensus.cc:357] T fdc77810f27a40758e063b8b1150ab71 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.417342 27490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a71290c27ac4a22bd9aa8e14e5834f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.417311 27692 raft_consensus.cc:383] T fdc77810f27a40758e063b8b1150ab71 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:19.422107 27692 raft_consensus.cc:738] T fdc77810f27a40758e063b8b1150ab71 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:19.423676 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e4f956f26c45989ca02ee0cd02cd87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.423591 27692 consensus_queue.cc:260] T fdc77810f27a40758e063b8b1150ab71 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.424203 27692 ts_tablet_manager.cc:1428] T fdc77810f27a40758e063b8b1150ab71 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.125s user 0.002s sys 0.000s
I20250903 20:05:19.424559 27692 tablet_bootstrap.cc:492] T 6ea37668d64d4feeb92a162711609c13 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:19.426424 27692 tablet_bootstrap.cc:654] T 6ea37668d64d4feeb92a162711609c13 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:19.432875 27693 raft_consensus.cc:357] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.433096 27693 raft_consensus.cc:383] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:19.433187 27693 raft_consensus.cc:738] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:19.433427 27693 consensus_queue.cc:260] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.433969 27693 ts_tablet_manager.cc:1428] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.048s user 0.001s sys 0.001s
I20250903 20:05:19.434303 27693 tablet_bootstrap.cc:492] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:19.434568 27684 tablet_replica.cc:331] T ecd339ec12e84e88acec427630cc8668 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:19.434844 27684 raft_consensus.cc:2241] T ecd339ec12e84e88acec427630cc8668 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:19.435094 27684 raft_consensus.cc:2270] T ecd339ec12e84e88acec427630cc8668 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:19.435848 27684 ts_tablet_manager.cc:1905] T ecd339ec12e84e88acec427630cc8668 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:19.436180 27693 tablet_bootstrap.cc:654] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:19.449853 27523 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e8bc99de4ffe43d58351d170b7c61d8a" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:19.450166 27523 raft_consensus.cc:2466] T e8bc99de4ffe43d58351d170b7c61d8a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:19.450419 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ea37668d64d4feeb92a162711609c13" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:19.450636 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e8bc99de4ffe43d58351d170b7c61d8a" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:19.450822 27522 raft_consensus.cc:3058] T e8bc99de4ffe43d58351d170b7c61d8a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:19.452304 27491 tablet_service.cc:1468] Processing CreateTablet for tablet 3af0c66a8c234711a43816f248e5be03 (DEFAULT_TABLE table=default.Table_5ca5478ef07f4acdbbedb24af75f9e04 [id=7fb235b85bba4f7ab1b77bc5d2b9b58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.452768 27491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3af0c66a8c234711a43816f248e5be03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.453784 27522 raft_consensus.cc:2466] T e8bc99de4ffe43d58351d170b7c61d8a P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
W20250903 20:05:19.454310 27320 leader_election.cc:343] T 264127511a234e9fbf409f3aa2860ccc P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 264127511a234e9fbf409f3aa2860ccc
I20250903 20:05:19.454353 27521 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fdc77810f27a40758e063b8b1150ab71" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:19.454493 27320 leader_election.cc:304] T 264127511a234e9fbf409f3aa2860ccc P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:19.454573 27521 raft_consensus.cc:2466] T fdc77810f27a40758e063b8b1150ab71 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:19.454747 27496 tablet_service.cc:1468] Processing CreateTablet for tablet 1598824a733a4a93bed043e81b86abf3 (DEFAULT_TABLE table=default.Table_d9276aa6946f41d7ac0fabc39338c23d [id=60e7bfe4938540e4a3c34b248356969f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.450470 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecd339ec12e84e88acec427630cc8668" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:19.455152 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1598824a733a4a93bed043e81b86abf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.455667 27488 tablet_service.cc:1468] Processing CreateTablet for tablet 8330ab3e4b6f4560b9e41a9df280c211 (DEFAULT_TABLE table=default.Table_83e2f4211b724a698cf478ddde974155 [id=07e066ded3ea4623958b36fd3874a354]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.456108 27503 tablet_service.cc:1468] Processing CreateTablet for tablet e0b9172ebb484382adb86e597f5cbc3b (DEFAULT_TABLE table=default.Table_eb74cc68ea8b4ab0bb7eb3937ac1aa4d [id=81969f6cf3404aa2ac397fb44b0e615d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.456262 27489 tablet_service.cc:1515] Processing DeleteTablet for tablet 8400e23873fd4be18de2998afc64a689 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:19.456341 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet 64c4dd2645034a7eaf243f5474603f10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:19.456460 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0b9172ebb484382adb86e597f5cbc3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.456763 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet 90baf83357494a0aae325908c118b471 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:19.457337 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8330ab3e4b6f4560b9e41a9df280c211. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.459012 27492 tablet_service.cc:1468] Processing CreateTablet for tablet eb878263eda9447e870a782907fe6a8d (DEFAULT_TABLE table=default.Table_647a8fb3a7d74bd48d49b76c3266b37f [id=e58c667578894d8795364da6dde5d8c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.459443 27492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb878263eda9447e870a782907fe6a8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.460021 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 4789bd72aa454b639df76bd5ad379850 (DEFAULT_TABLE table=default.Table_362ac8cb413e426ebf154eae4b6b5c7e [id=10ef25498f7448dbb19cd66d4c6ba713]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.460445 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4789bd72aa454b639df76bd5ad379850. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.462492 28244 raft_consensus.cc:2747] T 264127511a234e9fbf409f3aa2860ccc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:19.473737 27692 tablet_bootstrap.cc:492] T 6ea37668d64d4feeb92a162711609c13 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:19.473948 27692 ts_tablet_manager.cc:1397] T 6ea37668d64d4feeb92a162711609c13 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.049s user 0.000s sys 0.006s
I20250903 20:05:19.474028 27384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e8bc99de4ffe43d58351d170b7c61d8a" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:19.474382 27384 raft_consensus.cc:2391] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
I20250903 20:05:19.474908 27692 raft_consensus.cc:357] T 6ea37668d64d4feeb92a162711609c13 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.475102 27692 raft_consensus.cc:383] T 6ea37668d64d4feeb92a162711609c13 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:19.481032 27692 raft_consensus.cc:738] T 6ea37668d64d4feeb92a162711609c13 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:19.478569 27499 tablet_service.cc:1468] Processing CreateTablet for tablet c6cebb39bdb148d1bcbb47c4160a7be4 (DEFAULT_TABLE table=default.Table_2b5e97597b354141839a4ec0ca6bf046 [id=ca81d81b44c4465bb6e5932bd23b089f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.481770 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6cebb39bdb148d1bcbb47c4160a7be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.482340 27692 consensus_queue.cc:260] T 6ea37668d64d4feeb92a162711609c13 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.483001 27692 ts_tablet_manager.cc:1428] T 6ea37668d64d4feeb92a162711609c13 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20250903 20:05:19.483350 27692 tablet_bootstrap.cc:492] T c9129a3e773a4fecb28ceee27778a294 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:19.485159 27692 tablet_bootstrap.cc:654] T c9129a3e773a4fecb28ceee27778a294 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:19.486742 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet f1c3c8e46a2b40ccb8d5bb61843cecd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:19.457134 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet bf2545037c214bb1b5060575e3cdf81a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:19.505962 27687 ts_tablet_manager.cc:1918] T 44db427ee53a4d0a907dfd63282d6d4f P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:19.513307 27692 tablet_bootstrap.cc:492] T c9129a3e773a4fecb28ceee27778a294 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:19.513499 27692 ts_tablet_manager.cc:1397] T c9129a3e773a4fecb28ceee27778a294 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.001s
I20250903 20:05:19.514353 27692 raft_consensus.cc:357] T c9129a3e773a4fecb28ceee27778a294 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.514568 27692 raft_consensus.cc:383] T c9129a3e773a4fecb28ceee27778a294 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:19.514676 27692 raft_consensus.cc:738] T c9129a3e773a4fecb28ceee27778a294 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:19.514909 27692 consensus_queue.cc:260] T c9129a3e773a4fecb28ceee27778a294 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.520032 28208 raft_consensus.cc:513] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.520994 28208 leader_election.cc:290] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:19.522063 27692 ts_tablet_manager.cc:1428] T c9129a3e773a4fecb28ceee27778a294 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
W20250903 20:05:19.524127 27320 leader_election.cc:343] T 6ea37668d64d4feeb92a162711609c13 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:19.524602 27320 leader_election.cc:304] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:19.525095 28208 raft_consensus.cc:2802] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:19.525296 28208 raft_consensus.cc:695] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:19.526458 27687 log.cc:1199] T 44db427ee53a4d0a907dfd63282d6d4f P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/44db427ee53a4d0a907dfd63282d6d4f
I20250903 20:05:19.544147 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 1396d4f8a93b412ca8397368fb98834e (table default.Table_35a3167eda1a45949f8cd8a281a88733 [id=da438a4eb783471ea0b09fe1117ec822]) successfully deleted
W20250903 20:05:19.544484 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 0fef9aa7153a4902abce522e3ff0df3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0fef9aa7153a4902abce522e3ff0df3f already in progress: creating tablet
W20250903 20:05:19.544713 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 1223743918814d88b401782ca4b521dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1223743918814d88b401782ca4b521dd already in progress: creating tablet
W20250903 20:05:19.544909 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 6f0cba70240f4f38a4fa5ccbcd99b3b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f0cba70240f4f38a4fa5ccbcd99b3b1 already in progress: creating tablet
W20250903 20:05:19.545099 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 512908bc4d6941ec81b145ee5795405f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 512908bc4d6941ec81b145ee5795405f already in progress: creating tablet
I20250903 20:05:19.545320 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecd339ec12e84e88acec427630cc8668" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:19.546504 27692 tablet_bootstrap.cc:492] T 5bc3b9771a0042fd87e706204dcbb712 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:19.525679 28208 consensus_queue.cc:237] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:19.550740 27320 leader_election.cc:343] T eced730bdbc14a8c9f6b7b7e53435fd0 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: eced730bdbc14a8c9f6b7b7e53435fd0
I20250903 20:05:19.551226 27320 leader_election.cc:304] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
W20250903 20:05:19.551553 27320 leader_election.cc:343] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: Tablet not RUNNING: STOPPED
W20250903 20:05:19.551849 27320 leader_election.cc:343] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: Tablet not RUNNING: STOPPED
I20250903 20:05:19.552479 28240 raft_consensus.cc:2802] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:19.552567 28240 raft_consensus.cc:491] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:19.552664 28240 raft_consensus.cc:3058] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:19.555732 28240 raft_consensus.cc:513] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.556432 28240 leader_election.cc:290] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:19.556594 28184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250903 20:05:19.557018 28184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 20:05:19.557143 28184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250903 20:05:19.558048 27492 tablet_service.cc:1468] Processing CreateTablet for tablet 3d7aac88e92d4a45ab2ea694b7eaaaac (DEFAULT_TABLE table=default.Table_1c67ff00c09c4c3485ae43755a6f7c1f [id=bae69f4ce4da474f93b45bbeb58e0115]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.559446 27492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d7aac88e92d4a45ab2ea694b7eaaaac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.560993 27692 tablet_bootstrap.cc:654] T 5bc3b9771a0042fd87e706204dcbb712 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:19.561615 27495 tablet_service.cc:1468] Processing CreateTablet for tablet a4fb5fb54cb740698336598c09b58699 (DEFAULT_TABLE table=default.Table_9fa3fddcea2e482baf53322b4072b3b6 [id=5d5759a8aece4433ba5544f2de1df0d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.565384 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fdc77810f27a40758e063b8b1150ab71" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:19.565660 27527 raft_consensus.cc:3058] T fdc77810f27a40758e063b8b1150ab71 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:19.568663 27491 tablet_service.cc:1468] Processing CreateTablet for tablet 17e7deb74389448db37289563b86c253 (DEFAULT_TABLE table=default.Table_dc971e5490844bea94ad3c6b8cf15d9d [id=5cdd26db1c56496b807e8cd941cdb678]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.569363 27496 tablet_service.cc:1468] Processing CreateTablet for tablet 7e7390249fd045f8aec9a8e4e9573c7a (DEFAULT_TABLE table=default.Table_cfb651c1215b4325bcd5237fb8c1d7d4 [id=52585446937b46dc9a69a935254bb010]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.570267 27687 ts_tablet_manager.cc:1939] T 44db427ee53a4d0a907dfd63282d6d4f P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:19.570334 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4fb5fb54cb740698336598c09b58699. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.574950 27491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17e7deb74389448db37289563b86c253. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:19.575770 28184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 20:05:19.575918 28184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 20:05:19.576045 28184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 20:05:19.576167 28184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 20:05:19.578181 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 485) took 1466 ms. Trace:
I20250903 20:05:19.578158 27488 tablet_service.cc:1468] Processing CreateTablet for tablet 6820c9a583af4bf68f9e63ae06e6ec64 (DEFAULT_TABLE table=default.Table_86a571e18d164bd5beae624857d8526d [id=b6906abcba134b2799d9a8e793f228a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.578404 27687 rpcz_store.cc:276] 0903 20:05:18.111178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.111252 (+ 74us) service_pool.cc:225] Handling call
0903 20:05:19.128452 (+1017200us) ts_tablet_manager.cc:1145] Deleting tablet 44db427ee53a4d0a907dfd63282d6d4f
0903 20:05:19.128467 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:19.369736 (+241269us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:19.505772 (+136036us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:19.577171 (+ 71399us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1016893}
I20250903 20:05:19.578881 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 487) took 1384 ms. Trace:
I20250903 20:05:19.578940 27503 tablet_service.cc:1468] Processing CreateTablet for tablet f31e3e5ea72940de953c7dd787515c62 (DEFAULT_TABLE table=default.Table_1e9db7589b3a4cbe94da871dad467381 [id=c3a218a895744161a2100e5ea01a6131]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.579053 27687 rpcz_store.cc:276] 0903 20:05:18.194255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.194439 (+ 184us) service_pool.cc:225] Handling call
0903 20:05:19.578701 (+1384262us) ts_tablet_manager.cc:1145] Deleting tablet e2441aa3c2c748bba5360259fb663c81
0903 20:05:19.578710 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:19.578829 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1344478}
I20250903 20:05:19.579373 27687 tablet_replica.cc:331] T 73f65e7515994b898eec2ea23903aff4 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:19.579655 27687 raft_consensus.cc:2241] T 73f65e7515994b898eec2ea23903aff4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:19.580161 27687 raft_consensus.cc:2270] T 73f65e7515994b898eec2ea23903aff4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:19.581844 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e7390249fd045f8aec9a8e4e9573c7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.582474 27687 ts_tablet_manager.cc:1905] T 73f65e7515994b898eec2ea23903aff4 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:19.583559 27497 tablet_service.cc:1468] Processing CreateTablet for tablet ee6d475d2c2941c0b8f271b9a1072d5e (DEFAULT_TABLE table=default.Table_133bdffbe79346529dea857869c07825 [id=6cf5cb06af38416d9d91861b5bfe233e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.584717 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f31e3e5ea72940de953c7dd787515c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.586632 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6820c9a583af4bf68f9e63ae06e6ec64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.587496 27357 tablet_service.cc:1468] Processing CreateTablet for tablet 17e7deb74389448db37289563b86c253 (DEFAULT_TABLE table=default.Table_dc971e5490844bea94ad3c6b8cf15d9d [id=5cdd26db1c56496b807e8cd941cdb678]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.587889 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17e7deb74389448db37289563b86c253. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.587931 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee6d475d2c2941c0b8f271b9a1072d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.589720 27359 tablet_service.cc:1468] Processing CreateTablet for tablet 7e7390249fd045f8aec9a8e4e9573c7a (DEFAULT_TABLE table=default.Table_cfb651c1215b4325bcd5237fb8c1d7d4 [id=52585446937b46dc9a69a935254bb010]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.590375 27693 tablet_bootstrap.cc:492] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:19.592482 28184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.65.84:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.26.65.126:39015,127.26.65.125:44059,127.26.65.124:40911
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.65.125:44059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.26.65.125
--webserver_port=41613
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 19:52:33 UTC on 24a791456cd2
build id 7884
ASAN enabled
I20250903 20:05:19.593971 27360 tablet_service.cc:1468] Processing CreateTablet for tablet ee6d475d2c2941c0b8f271b9a1072d5e (DEFAULT_TABLE table=default.Table_133bdffbe79346529dea857869c07825 [id=6cf5cb06af38416d9d91861b5bfe233e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.594483 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 6820c9a583af4bf68f9e63ae06e6ec64 (DEFAULT_TABLE table=default.Table_86a571e18d164bd5beae624857d8526d [id=b6906abcba134b2799d9a8e793f228a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.594844 28184 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 20:05:19.595019 27693 ts_tablet_manager.cc:1397] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.156s user 0.004s sys 0.000s
I20250903 20:05:19.597400 28184 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250903 20:05:19.598524 27350 tablet_service.cc:1468] Processing CreateTablet for tablet f31e3e5ea72940de953c7dd787515c62 (DEFAULT_TABLE table=default.Table_1e9db7589b3a4cbe94da871dad467381 [id=c3a218a895744161a2100e5ea01a6131]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.600975 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 44db427ee53a4d0a907dfd63282d6d4f (table default.Table_8e482428f5d8407795dcd3111aefc596 [id=5c647f2080d043f3851e5767b1efd161]) successfully deleted
I20250903 20:05:19.602514 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e7390249fd045f8aec9a8e4e9573c7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.603750 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee6d475d2c2941c0b8f271b9a1072d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.608438 27527 raft_consensus.cc:2466] T fdc77810f27a40758e063b8b1150ab71 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:19.610561 27355 tablet_service.cc:1468] Processing CreateTablet for tablet 3d7aac88e92d4a45ab2ea694b7eaaaac (DEFAULT_TABLE table=default.Table_1c67ff00c09c4c3485ae43755a6f7c1f [id=bae69f4ce4da474f93b45bbeb58e0115]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.611088 27320 leader_election.cc:304] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:19.615285 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6820c9a583af4bf68f9e63ae06e6ec64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.617954 28237 raft_consensus.cc:2802] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:19.618278 28237 raft_consensus.cc:695] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
W20250903 20:05:19.619954 27051 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet a9f43781299e47a8aecc37a0b73d7a64 (table default.Table_6ab02d2c339e463fbed5c3429c7c4ec9 [id=f70d7ea93e154e6199f6766316353862], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet a9f43781299e47a8aecc37a0b73d7a64 consensus state has no leader
I20250903 20:05:19.620482 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f31e3e5ea72940de953c7dd787515c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.633100 28237 consensus_queue.cc:237] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.635066 27355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d7aac88e92d4a45ab2ea694b7eaaaac. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:19.636058 27184 leader_election.cc:343] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:19.636291 27184 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 185840us
I20250903 20:05:19.637251 27213 tablet_service.cc:1468] Processing CreateTablet for tablet 3d7aac88e92d4a45ab2ea694b7eaaaac (DEFAULT_TABLE table=default.Table_1c67ff00c09c4c3485ae43755a6f7c1f [id=bae69f4ce4da474f93b45bbeb58e0115]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.637429 27222 tablet_service.cc:1468] Processing CreateTablet for tablet 7e7390249fd045f8aec9a8e4e9573c7a (DEFAULT_TABLE table=default.Table_cfb651c1215b4325bcd5237fb8c1d7d4 [id=52585446937b46dc9a69a935254bb010]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.637492 27220 tablet_service.cc:1468] Processing CreateTablet for tablet f31e3e5ea72940de953c7dd787515c62 (DEFAULT_TABLE table=default.Table_1e9db7589b3a4cbe94da871dad467381 [id=c3a218a895744161a2100e5ea01a6131]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.642990 27685 log.cc:1199] T eced730bdbc14a8c9f6b7b7e53435fd0 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/eced730bdbc14a8c9f6b7b7e53435fd0
I20250903 20:05:19.648658 27684 ts_tablet_manager.cc:1918] T ecd339ec12e84e88acec427630cc8668 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:19.653575 27684 log.cc:1199] T ecd339ec12e84e88acec427630cc8668 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/ecd339ec12e84e88acec427630cc8668
I20250903 20:05:19.654291 27684 ts_tablet_manager.cc:1939] T ecd339ec12e84e88acec427630cc8668 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
W20250903 20:05:19.654850 28234 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.600s user 0.000s sys 0.021s
I20250903 20:05:19.657894 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet ecd339ec12e84e88acec427630cc8668 (table default.Table_54bf0e4b392545b08001021008e93985 [id=a803b1a5d2404f74aa4671eebd23e8ae]) successfully deleted
I20250903 20:05:19.657878 27229 tablet_service.cc:1468] Processing CreateTablet for tablet ee6d475d2c2941c0b8f271b9a1072d5e (DEFAULT_TABLE table=default.Table_133bdffbe79346529dea857869c07825 [id=6cf5cb06af38416d9d91861b5bfe233e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.658563 27685 ts_tablet_manager.cc:1939] T eced730bdbc14a8c9f6b7b7e53435fd0 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
W20250903 20:05:19.659597 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 8400e23873fd4be18de2998afc64a689 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8400e23873fd4be18de2998afc64a689 already in progress: creating tablet
I20250903 20:05:19.659653 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d7aac88e92d4a45ab2ea694b7eaaaac. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:19.660594 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 64c4dd2645034a7eaf243f5474603f10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64c4dd2645034a7eaf243f5474603f10 already in progress: creating tablet
W20250903 20:05:19.660913 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 90baf83357494a0aae325908c118b471 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90baf83357494a0aae325908c118b471 already in progress: creating tablet
W20250903 20:05:19.661229 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet f1c3c8e46a2b40ccb8d5bb61843cecd1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1c3c8e46a2b40ccb8d5bb61843cecd1 already in progress: creating tablet
W20250903 20:05:19.661545 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet bf2545037c214bb1b5060575e3cdf81a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf2545037c214bb1b5060575e3cdf81a already in progress: creating tablet
W20250903 20:05:19.662057 28234 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.607s user 0.000s sys 0.021s
I20250903 20:05:19.662376 28234 raft_consensus.cc:695] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:19.661393 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 17e7deb74389448db37289563b86c253 (DEFAULT_TABLE table=default.Table_dc971e5490844bea94ad3c6b8cf15d9d [id=5cdd26db1c56496b807e8cd941cdb678]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.665452 28030 raft_consensus.cc:491] T 6238740ce1894e899be9a40321f0ab73 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250903 20:05:19.661868 27322 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.608s user 0.008s sys 0.014s
W20250903 20:05:19.665608 27322 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.612s user 0.008s sys 0.014s
I20250903 20:05:19.665603 28030 raft_consensus.cc:513] T 6238740ce1894e899be9a40321f0ab73 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.667263 28030 leader_election.cc:290] T 6238740ce1894e899be9a40321f0ab73 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:19.668447 28208 raft_consensus.cc:491] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:19.668696 28208 raft_consensus.cc:513] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:19.670008 28208 leader_election.cc:290] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:19.670573 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bc3b9771a0042fd87e706204dcbb712" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:19.670923 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6238740ce1894e899be9a40321f0ab73" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:19.671197 27320 leader_election.cc:343] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:19.664228 28234 consensus_queue.cc:237] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.675907 27224 tablet_service.cc:1468] Processing CreateTablet for tablet 6820c9a583af4bf68f9e63ae06e6ec64 (DEFAULT_TABLE table=default.Table_86a571e18d164bd5beae624857d8526d [id=b6906abcba134b2799d9a8e793f228a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:19.677102 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e7390249fd045f8aec9a8e4e9573c7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.677961 27229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee6d475d2c2941c0b8f271b9a1072d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.678675 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e8bc99de4ffe43d58351d170b7c61d8a" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:19.675480 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 353) took 4020 ms. Trace:
I20250903 20:05:19.679157 27685 rpcz_store.cc:276] 0903 20:05:15.638743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.642787 (+ 4044us) service_pool.cc:225] Handling call
0903 20:05:18.802072 (+3159285us) ts_tablet_manager.cc:1145] Deleting tablet eced730bdbc14a8c9f6b7b7e53435fd0
0903 20:05:18.802084 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:19.127451 (+325367us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:19.377172 (+249721us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:19.658863 (+281691us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3159023}
I20250903 20:05:19.679385 27462 leader_election.cc:304] T e8bc99de4ffe43d58351d170b7c61d8a P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:19.679728 28242 raft_consensus.cc:2747] T e8bc99de4ffe43d58351d170b7c61d8a P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:19.680702 27695 tablet_bootstrap.cc:492] T 07565ea2b5f34dcfb407ff3bfde17243 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:19.681066 27695 ts_tablet_manager.cc:1397] T 07565ea2b5f34dcfb407ff3bfde17243 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.510s user 0.036s sys 0.028s
I20250903 20:05:19.682461 27245 raft_consensus.cc:2466] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
W20250903 20:05:19.686075 27184 leader_election.cc:343] T 6238740ce1894e899be9a40321f0ab73 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:19.686334 27695 raft_consensus.cc:357] T 07565ea2b5f34dcfb407ff3bfde17243 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:19.686249 27183 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 319071us
I20250903 20:05:19.690656 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a76a17b64a74fbfbb4f2e92a02ae1cd" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:19.689646 27695 raft_consensus.cc:383] T 07565ea2b5f34dcfb407ff3bfde17243 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:19.691268 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17e7deb74389448db37289563b86c253. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.691859 27692 tablet_bootstrap.cc:492] T 5bc3b9771a0042fd87e706204dcbb712 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:19.691867 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6820c9a583af4bf68f9e63ae06e6ec64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.691916 27685 tablet_replica.cc:331] T 6ea37668d64d4feeb92a162711609c13 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:19.692018 27692 ts_tablet_manager.cc:1397] T 5bc3b9771a0042fd87e706204dcbb712 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.170s user 0.006s sys 0.007s
I20250903 20:05:19.692122 27685 raft_consensus.cc:2241] T 6ea37668d64d4feeb92a162711609c13 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:19.691924 27695 raft_consensus.cc:738] T 07565ea2b5f34dcfb407ff3bfde17243 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:19.692147 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f31e3e5ea72940de953c7dd787515c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:19.693112 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6238740ce1894e899be9a40321f0ab73" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:19.693512 27692 raft_consensus.cc:357] T 5bc3b9771a0042fd87e706204dcbb712 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:19.693765 27692 raft_consensus.cc:383] T 5bc3b9771a0042fd87e706204dcbb712 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:19.693887 27692 raft_consensus.cc:738] T 5bc3b9771a0042fd87e706204dcbb712 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:19.694156 27692 consensus_queue.cc:260] T 5bc3b9771a0042fd87e706204dcbb712 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:19.694875 27692 ts_tablet_manager.cc:1428] T 5bc3b9771a0042fd87e706204dcbb712 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:19.695919 27692 tablet_bootstrap.cc:492] T deb84fc3465443d99c11097e53f5f575 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:19.699566 27692 tablet_bootstrap.cc:654] T deb84fc3465443d99c11097e53f5f575 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:19.703634 27245 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ea37668d64d4feeb92a162711609c13" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:19.704299 27321 leader_election.cc:343] T eced730bdbc14a8c9f6b7b7e53435fd0 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: eced730bdbc14a8c9f6b7b7e53435fd0
I20250903 20:05:19.704466 27321 leader_election.cc:304] T eced730bdbc14a8c9f6b7b7e53435fd0 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:19.704905 28208 raft_consensus.cc:2747] T eced730bdbc14a8c9f6b7b7e53435fd0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:19.705546 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecd339ec12e84e88acec427630cc8668" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:19.708997 27695 consensus_queue.cc:260] T 07565ea2b5f34dcfb407ff3bfde17243 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.712841 27693 raft_consensus.cc:357] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.713100 27693 raft_consensus.cc:383] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:19.713181 27693 raft_consensus.cc:738] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:19.713449 27693 consensus_queue.cc:260] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.713950 27693 ts_tablet_manager.cc:1428] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.119s user 0.001s sys 0.001s
I20250903 20:05:19.714337 27693 tablet_bootstrap.cc:492] T 8b3121fce1c44d16a0b083313dc67b7a P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:19.708918 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fdc77810f27a40758e063b8b1150ab71" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:19.720520 27695 ts_tablet_manager.cc:1428] T 07565ea2b5f34dcfb407ff3bfde17243 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.028s user 0.002s sys 0.001s
W20250903 20:05:19.720681 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet e2441aa3c2c748bba5360259fb663c81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2441aa3c2c748bba5360259fb663c81 already in progress: creating tablet
I20250903 20:05:19.723857 27685 raft_consensus.cc:2270] T 6ea37668d64d4feeb92a162711609c13 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:19.724572 27685 ts_tablet_manager.cc:1905] T 6ea37668d64d4feeb92a162711609c13 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:19.725284 27692 tablet_bootstrap.cc:492] T deb84fc3465443d99c11097e53f5f575 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:19.725526 27692 ts_tablet_manager.cc:1397] T deb84fc3465443d99c11097e53f5f575 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
W20250903 20:05:19.729918 27183 leader_election.cc:343] T fbfabb6ff1714d2e921f45256b543759 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:19.724750 28249 raft_consensus.cc:491] T 57a1167457b84ea8a493be232b1d088a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:19.730187 27246 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fdc77810f27a40758e063b8b1150ab71" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:19.733793 27183 leader_election.cc:304] T fbfabb6ff1714d2e921f45256b543759 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:19.741171 27249 raft_consensus.cc:3058] T ecd339ec12e84e88acec427630cc8668 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:19.741212 28245 raft_consensus.cc:491] T 369c750ff5bf449e989d95ffd89497d4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:19.742579 28030 raft_consensus.cc:2747] T fbfabb6ff1714d2e921f45256b543759 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:19.731143 28249 raft_consensus.cc:513] T 57a1167457b84ea8a493be232b1d088a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:19.744872 27695 tablet_bootstrap.cc:492] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:19.745630 27248 raft_consensus.cc:2466] T fdc77810f27a40758e063b8b1150ab71 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:19.746655 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet eced730bdbc14a8c9f6b7b7e53435fd0 (table default.Table_b520cd881f58424dadf3fa4a013812ee [id=2c179236954b41cfa9242e16946ef9bb]) successfully deleted
I20250903 20:05:19.746754 27692 raft_consensus.cc:357] T deb84fc3465443d99c11097e53f5f575 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.747437 27692 raft_consensus.cc:383] T deb84fc3465443d99c11097e53f5f575 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:19.747686 27692 raft_consensus.cc:738] T deb84fc3465443d99c11097e53f5f575 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:19.748193 27692 consensus_queue.cc:260] T deb84fc3465443d99c11097e53f5f575 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.748481 28245 raft_consensus.cc:513] T 369c750ff5bf449e989d95ffd89497d4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.730862 27244 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bc3b9771a0042fd87e706204dcbb712" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:19.751389 27692 ts_tablet_manager.cc:1428] T deb84fc3465443d99c11097e53f5f575 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20250903 20:05:19.751725 27692 tablet_bootstrap.cc:492] T 8b3121fce1c44d16a0b083313dc67b7a P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:19.755985 27244 raft_consensus.cc:2466] T 5bc3b9771a0042fd87e706204dcbb712 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
W20250903 20:05:19.760671 27183 leader_election.cc:343] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:19.761052 28245 leader_election.cc:290] T 369c750ff5bf449e989d95ffd89497d4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:19.761608 27183 leader_election.cc:304] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:19.761945 28249 leader_election.cc:290] T 57a1167457b84ea8a493be232b1d088a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:19.763078 27692 tablet_bootstrap.cc:654] T 8b3121fce1c44d16a0b083313dc67b7a P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:19.765019 27249 raft_consensus.cc:2466] T ecd339ec12e84e88acec427630cc8668 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
W20250903 20:05:19.765074 27183 consensus_peers.cc:489] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1396d4f8a93b412ca8397368fb98834e. This is attempt 1: this message will repeat every 5th retry.
W20250903 20:05:19.765331 27183 leader_election.cc:343] T 6238740ce1894e899be9a40321f0ab73 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:19.768551 28249 raft_consensus.cc:2747] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:19.773207 27695 tablet_bootstrap.cc:654] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:19.776867 27183 leader_election.cc:304] T 6238740ce1894e899be9a40321f0ab73 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:19.778877 27692 tablet_bootstrap.cc:492] T 8b3121fce1c44d16a0b083313dc67b7a P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
W20250903 20:05:19.780864 27321 leader_election.cc:343] T 6ea37668d64d4feeb92a162711609c13 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:19.782524 27692 ts_tablet_manager.cc:1397] T 8b3121fce1c44d16a0b083313dc67b7a P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20250903 20:05:19.787307 27692 raft_consensus.cc:357] T 8b3121fce1c44d16a0b083313dc67b7a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.787642 27692 raft_consensus.cc:383] T 8b3121fce1c44d16a0b083313dc67b7a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:19.787751 27692 raft_consensus.cc:738] T 8b3121fce1c44d16a0b083313dc67b7a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:19.788771 27692 consensus_queue.cc:260] T 8b3121fce1c44d16a0b083313dc67b7a P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.790004 27692 ts_tablet_manager.cc:1428] T 8b3121fce1c44d16a0b083313dc67b7a P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250903 20:05:19.790372 27692 tablet_bootstrap.cc:492] T 1fef9400227449ed892daff588999caa P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:19.798951 27321 leader_election.cc:304] T 6ea37668d64d4feeb92a162711609c13 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:19.799432 27321 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 59424.1us
I20250903 20:05:19.802821 27321 leader_election.cc:304] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:19.805112 27692 tablet_bootstrap.cc:654] T 1fef9400227449ed892daff588999caa P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:19.809106 28249 raft_consensus.cc:2747] T 6238740ce1894e899be9a40321f0ab73 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:19.809451 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "369c750ff5bf449e989d95ffd89497d4" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:19.809814 28234 raft_consensus.cc:2802] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250903 20:05:19.810307 27184 leader_election.cc:343] T 369c750ff5bf449e989d95ffd89497d4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:19.811831 28234 raft_consensus.cc:491] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:19.811959 28234 raft_consensus.cc:3058] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:19.820967 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "57a1167457b84ea8a493be232b1d088a" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:19.821549 27321 leader_election.cc:304] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:19.823318 27184 leader_election.cc:343] T 57a1167457b84ea8a493be232b1d088a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:19.827142 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 516830fc42e84af693b3af52caee30ee consensus state has no leader
W20250903 20:05:19.830971 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4bcd65022b7b4502983b6b58809047e3 consensus state has no leader
W20250903 20:05:19.844185 27059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 64d53f2071b0449ca61caa6c264a3641 consensus state has no leader
I20250903 20:05:19.855620 27685 ts_tablet_manager.cc:1918] T 6ea37668d64d4feeb92a162711609c13 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:19.855945 27685 log.cc:1199] T 6ea37668d64d4feeb92a162711609c13 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/6ea37668d64d4feeb92a162711609c13
I20250903 20:05:19.857635 27685 ts_tablet_manager.cc:1939] T 6ea37668d64d4feeb92a162711609c13 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:19.860210 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_fc325fc838f44caa818ddcddb486d85f" } modify_external_catalogs: true
I20250903 20:05:19.860420 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_fc325fc838f44caa818ddcddb486d85f" } modify_external_catalogs: true
I20250903 20:05:19.860776 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_480e84d976d54812a2a0ee3d43a4b08b" } modify_external_catalogs: true
I20250903 20:05:19.860958 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_480e84d976d54812a2a0ee3d43a4b08b" } modify_external_catalogs: true
I20250903 20:05:19.862107 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1e9db7589b3a4cbe94da871dad467381" } modify_external_catalogs: true
I20250903 20:05:19.862139 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_aaea28edb28e4ee991b380a66d0297d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:19.862304 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1e9db7589b3a4cbe94da871dad467381" } modify_external_catalogs: true
W20250903 20:05:19.862820 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aaea28edb28e4ee991b380a66d0297d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:19.863474 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 355) took 4208 ms. Trace:
I20250903 20:05:19.863754 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e79b97db68cd4a7d827d56a72a6d15e9" } modify_external_catalogs: true
I20250903 20:05:19.863914 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e79b97db68cd4a7d827d56a72a6d15e9" } modify_external_catalogs: true
I20250903 20:05:19.863735 27685 rpcz_store.cc:276] 0903 20:05:15.649388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.649552 (+ 164us) service_pool.cc:225] Handling call
0903 20:05:19.679464 (+4029912us) ts_tablet_manager.cc:1145] Deleting tablet 6ea37668d64d4feeb92a162711609c13
0903 20:05:19.691040 (+ 11576us) spinlock_profiling.cc:243] Waited 5.83 ms on lock 0x61800000d518. stack: 00007f589abe0dc6 00007f589abe0b80 00007f5899bc83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f589abc96cd 00007f589abca73e 00007f58ae2928ed 00007f58ae2141d3 00007f58aa45e676 00007f58aa45f9f9 00007f58aa492344 00007f58aa473da0 00007f58aa473b2d 0000000000a774dc
0903 20:05:19.691053 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:19.822157 (+131104us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:19.854346 (+ 32189us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:19.858154 (+ 3808us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6307,"spinlock_wait_cycles":13419776,"tablet-delete.queue_time_us":4029632}
I20250903 20:05:19.864007 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_24fa04bd95714589b18eb17010dbcec2" } modify_external_catalogs: true
I20250903 20:05:19.864174 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_24fa04bd95714589b18eb17010dbcec2" } modify_external_catalogs: true
I20250903 20:05:19.864307 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 6ea37668d64d4feeb92a162711609c13 (table default.Table_cc799e62ff6b47e1af28b7006551bbf5 [id=d97016a400e34272830add1e220c65ae]) successfully deleted
I20250903 20:05:19.864707 27685 tablet_replica.cc:331] T a10b079e7d5a440580ac00195fe88c65 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:19.865393 27685 raft_consensus.cc:2241] T a10b079e7d5a440580ac00195fe88c65 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:19.865607 27685 raft_consensus.cc:2270] T a10b079e7d5a440580ac00195fe88c65 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:19.867271 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_1310291579564dd18d0b47d1050cf905"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:19.867866 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1310291579564dd18d0b47d1050cf905 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:19.868525 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_654e50df426048f8bc1852c3015cb4f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:19.869031 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_654e50df426048f8bc1852c3015cb4f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:19.869483 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_9d4921e7acb249219b9c88ec3615a646" } new_table_name: "default.Table_b838fda30d4043a5ad578ec527c1d105" modify_external_catalogs: true
I20250903 20:05:19.870775 27685 ts_tablet_manager.cc:1905] T a10b079e7d5a440580ac00195fe88c65 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:19.879317 28208 raft_consensus.cc:2802] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:19.879321 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_1b5f4328e47f4393a36d5ed02c2614c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:19.879518 28208 raft_consensus.cc:695] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
W20250903 20:05:19.879927 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b5f4328e47f4393a36d5ed02c2614c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:19.881098 28208 consensus_queue.cc:237] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.884142 28236 raft_consensus.cc:2747] T 6ea37668d64d4feeb92a162711609c13 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:19.885040 28234 raft_consensus.cc:513] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:19.885689 28234 leader_election.cc:290] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:19.885947 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet 5bc3b9771a0042fd87e706204dcbb712 with delete_type TABLET_DATA_DELETED (replaced by tablet 961ade36f7d14eba91bb16d5eae2e254) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:19.886459 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bc3b9771a0042fd87e706204dcbb712" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:19.886744 27249 raft_consensus.cc:3058] T 5bc3b9771a0042fd87e706204dcbb712 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:19.886842 27684 tablet_replica.cc:331] T 5bc3b9771a0042fd87e706204dcbb712 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:19.887044 27684 raft_consensus.cc:2241] T 5bc3b9771a0042fd87e706204dcbb712 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:19.887457 27684 raft_consensus.cc:2270] T 5bc3b9771a0042fd87e706204dcbb712 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:19.887684 27496 tablet_service.cc:1515] Processing DeleteTablet for tablet 8b3121fce1c44d16a0b083313dc67b7a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:19.888175 27488 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ea37668d64d4feeb92a162711609c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:10 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:19.888342 27503 tablet_service.cc:1515] Processing DeleteTablet for tablet deb84fc3465443d99c11097e53f5f575 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:19.888450 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet fdc77810f27a40758e063b8b1150ab71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:19.889003 27684 ts_tablet_manager.cc:1905] T 5bc3b9771a0042fd87e706204dcbb712 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:19.892398 27249 raft_consensus.cc:2466] T 5bc3b9771a0042fd87e706204dcbb712 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:19.893419 27321 leader_election.cc:304] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:19.894155 28264 raft_consensus.cc:491] T 5fa7cc0c03234223af382982ca737e27 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:19.894507 28234 raft_consensus.cc:2802] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:19.894381 28264 raft_consensus.cc:513] T 5fa7cc0c03234223af382982ca737e27 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.894660 28234 raft_consensus.cc:695] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:19.895282 28264 leader_election.cc:290] T 5fa7cc0c03234223af382982ca737e27 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:19.895272 28234 consensus_queue.cc:237] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:19.895783 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5fa7cc0c03234223af382982ca737e27" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:19.895910 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bc3b9771a0042fd87e706204dcbb712" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
W20250903 20:05:19.896874 27184 leader_election.cc:343] T 5fa7cc0c03234223af382982ca737e27 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:19.897090 27320 leader_election.cc:343] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: Tablet not RUNNING: STOPPED
I20250903 20:05:19.898288 27684 ts_tablet_manager.cc:1918] T 5bc3b9771a0042fd87e706204dcbb712 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:19.898453 27684 log.cc:1199] T 5bc3b9771a0042fd87e706204dcbb712 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/5bc3b9771a0042fd87e706204dcbb712
I20250903 20:05:19.898850 27684 ts_tablet_manager.cc:1939] T 5bc3b9771a0042fd87e706204dcbb712 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:19.900002 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 5bc3b9771a0042fd87e706204dcbb712 (table default.Table_459903503b5148d09d3cb8f2f5519fab [id=3c69612aaf164dc899dcc252d85636f6]) successfully deleted
I20250903 20:05:19.902676 27684 tablet_replica.cc:331] T 6ea37668d64d4feeb92a162711609c13 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:19.903108 27684 raft_consensus.cc:2241] T 6ea37668d64d4feeb92a162711609c13 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:19.903388 27684 raft_consensus.cc:2270] T 6ea37668d64d4feeb92a162711609c13 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:19.907177 27684 ts_tablet_manager.cc:1905] T 6ea37668d64d4feeb92a162711609c13 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:19.910398 27687 ts_tablet_manager.cc:1918] T 73f65e7515994b898eec2ea23903aff4 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:19.910614 27687 log.cc:1199] T 73f65e7515994b898eec2ea23903aff4 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/73f65e7515994b898eec2ea23903aff4
I20250903 20:05:19.910971 27695 tablet_bootstrap.cc:492] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:19.911155 27695 ts_tablet_manager.cc:1397] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.190s user 0.007s sys 0.010s
I20250903 20:05:19.912124 27695 raft_consensus.cc:357] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.912322 27695 raft_consensus.cc:383] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:19.912431 27695 raft_consensus.cc:738] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:19.912706 27695 consensus_queue.cc:260] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.913056 27693 tablet_bootstrap.cc:654] T 8b3121fce1c44d16a0b083313dc67b7a P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:19.913347 27695 ts_tablet_manager.cc:1428] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:19.915419 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "57a1167457b84ea8a493be232b1d088a" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:19.915810 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "369c750ff5bf449e989d95ffd89497d4" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:19.916204 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5fa7cc0c03234223af382982ca737e27" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:19.943182 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet f31e3e5ea72940de953c7dd787515c62
I20250903 20:05:19.944211 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet f31e3e5ea72940de953c7dd787515c62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:19 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:19.944419 27496 tablet_service.cc:1515] Processing DeleteTablet for tablet f31e3e5ea72940de953c7dd787515c62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:19 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:19.944846 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet e3e82f37c6fa4bebb0115e6087919a27
I20250903 20:05:19.945684 27066 master_service.cc:948] ReplaceTablet: received request to replace tablet 514e382479014fa7bf1f404efff11c86 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:19.945736 27496 tablet_service.cc:1515] Processing DeleteTablet for tablet e3e82f37c6fa4bebb0115e6087919a27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:19 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:19.945884 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet e3e82f37c6fa4bebb0115e6087919a27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:19 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:19.946709 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet e3e82f37c6fa4bebb0115e6087919a27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:19 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:19.945168 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet f31e3e5ea72940de953c7dd787515c62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:19 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:19.948807 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 3919f7f95a954f59b68f56d0051c0109
I20250903 20:05:19.949173 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3bcc60f1d7ed42d89cfaf71c0fc18fdd" } new_table_name: "default.Table_7f9b2d482da241399d11bfa5a80b6729" modify_external_catalogs: true
I20250903 20:05:19.949790 27496 tablet_service.cc:1515] Processing DeleteTablet for tablet 3919f7f95a954f59b68f56d0051c0109 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:19 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:19.949697 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_77c890250aaf48f4b658f0f126941bc7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:19.950086 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet 3919f7f95a954f59b68f56d0051c0109 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:19 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:19.950292 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77c890250aaf48f4b658f0f126941bc7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:19.953403 27687 ts_tablet_manager.cc:1939] T 73f65e7515994b898eec2ea23903aff4 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:19.953455 27061 master_service.cc:948] ReplaceTablet: received request to replace tablet 68cff0e416a54a5eb37071f5a12f07ab from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:19.953452 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 3919f7f95a954f59b68f56d0051c0109 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:19 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:19.954023 27061 master_service.cc:948] ReplaceTablet: received request to replace tablet 68cff0e416a54a5eb37071f5a12f07ab from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:19.953296 27684 ts_tablet_manager.cc:1918] T 6ea37668d64d4feeb92a162711609c13 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:19.954530 27061 master_service.cc:948] ReplaceTablet: received request to replace tablet ed096b452ba747f89e91edc15e6e3ac1 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:19.954413 27684 log.cc:1199] T 6ea37668d64d4feeb92a162711609c13 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/6ea37668d64d4feeb92a162711609c13
I20250903 20:05:19.955088 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet bc98762790f441038972366b1cc2c55b
I20250903 20:05:19.955114 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 486) took 1811 ms. Trace:
I20250903 20:05:19.955219 27687 rpcz_store.cc:276] 0903 20:05:18.143538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.160398 (+ 16860us) service_pool.cc:225] Handling call
0903 20:05:19.579281 (+1418883us) ts_tablet_manager.cc:1145] Deleting tablet 73f65e7515994b898eec2ea23903aff4
0903 20:05:19.579291 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:19.800889 (+221598us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:19.909700 (+108811us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:19.955067 (+ 45367us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1341490}
I20250903 20:05:19.955762 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 73f65e7515994b898eec2ea23903aff4 (table default.Table_d5626bea3c88471b846f37ae4ed79715 [id=75e5cb0bab9b46a6b5caec7c99efc7b8]) successfully deleted
I20250903 20:05:19.955863 27496 tablet_service.cc:1515] Processing DeleteTablet for tablet bc98762790f441038972366b1cc2c55b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:19 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:19.956336 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet bc98762790f441038972366b1cc2c55b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:19 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:19.956815 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet bc98762790f441038972366b1cc2c55b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:19 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:19.959430 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 314856bf269f4df7aa25f5a30cec0d5b
I20250903 20:05:19.958799 27684 ts_tablet_manager.cc:1939] T 6ea37668d64d4feeb92a162711609c13 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:19.960680 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet 314856bf269f4df7aa25f5a30cec0d5b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:19 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:19.960261 27496 tablet_service.cc:1515] Processing DeleteTablet for tablet 314856bf269f4df7aa25f5a30cec0d5b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:19 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:19.962337 27352 tablet_service.cc:1515] Processing DeleteTablet for tablet 314856bf269f4df7aa25f5a30cec0d5b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:19 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:19.973881 27685 ts_tablet_manager.cc:1918] T a10b079e7d5a440580ac00195fe88c65 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:19.974229 27685 log.cc:1199] T a10b079e7d5a440580ac00195fe88c65 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/a10b079e7d5a440580ac00195fe88c65
W20250903 20:05:19.974354 27059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 548b50819f694fa3ad508da9f442a99c consensus state has no leader
I20250903 20:05:19.974753 27685 ts_tablet_manager.cc:1939] T a10b079e7d5a440580ac00195fe88c65 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:19.976295 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 6ea37668d64d4feeb92a162711609c13 (table default.Table_cc799e62ff6b47e1af28b7006551bbf5 [id=d97016a400e34272830add1e220c65ae]) successfully deleted
W20250903 20:05:19.977156 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 86e33bd235c14042a578a4a1c70f11b7 consensus state has no leader
I20250903 20:05:19.977087 27693 tablet_bootstrap.cc:492] T 8b3121fce1c44d16a0b083313dc67b7a P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:19.979434 27693 ts_tablet_manager.cc:1397] T 8b3121fce1c44d16a0b083313dc67b7a P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.265s user 0.003s sys 0.003s
I20250903 20:05:19.979766 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 514e382479014fa7bf1f404efff11c86
I20250903 20:05:19.980235 27066 catalog_manager.cc:6652] ReplaceTablet: tablet 514e382479014fa7bf1f404efff11c86 deleted and replaced by tablet 07439db2c3b8470eb2b797cc68376aa7
I20250903 20:05:19.981000 27496 tablet_service.cc:1515] Processing DeleteTablet for tablet 514e382479014fa7bf1f404efff11c86 with delete_type TABLET_DATA_DELETED (replaced by tablet 07439db2c3b8470eb2b797cc68376aa7) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:19.981261 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet 514e382479014fa7bf1f404efff11c86 with delete_type TABLET_DATA_DELETED (replaced by tablet 07439db2c3b8470eb2b797cc68376aa7) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:19.981775 27352 tablet_service.cc:1515] Processing DeleteTablet for tablet 514e382479014fa7bf1f404efff11c86 with delete_type TABLET_DATA_DELETED (replaced by tablet 07439db2c3b8470eb2b797cc68376aa7) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:19.983029 27183 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 58371.8us
I20250903 20:05:19.983264 27684 tablet_replica.cc:331] T deb84fc3465443d99c11097e53f5f575 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:19.983305 27066 master_service.cc:948] ReplaceTablet: received request to replace tablet 541278ca9ffb4e41b341139c2faa337f from {username='slave'} at 127.0.0.1:45936
W20250903 20:05:19.983362 27183 leader_election.cc:343] T 57a1167457b84ea8a493be232b1d088a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:19.983517 27684 raft_consensus.cc:2241] T deb84fc3465443d99c11097e53f5f575 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:19.983503 27183 leader_election.cc:304] T 57a1167457b84ea8a493be232b1d088a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:19.983758 27684 raft_consensus.cc:2270] T deb84fc3465443d99c11097e53f5f575 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:19.983800 28245 raft_consensus.cc:2747] T 57a1167457b84ea8a493be232b1d088a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:19.983992 27183 leader_election.cc:343] T 369c750ff5bf449e989d95ffd89497d4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:19.984113 27183 leader_election.cc:304] T 369c750ff5bf449e989d95ffd89497d4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:19.984376 28245 raft_consensus.cc:2747] T 369c750ff5bf449e989d95ffd89497d4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:19.984534 27183 leader_election.cc:343] T 5fa7cc0c03234223af382982ca737e27 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:19.984608 27684 ts_tablet_manager.cc:1905] T deb84fc3465443d99c11097e53f5f575 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:19.984655 27183 leader_election.cc:304] T 5fa7cc0c03234223af382982ca737e27 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:19.984876 28245 raft_consensus.cc:2747] T 5fa7cc0c03234223af382982ca737e27 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:19.985646 27693 raft_consensus.cc:357] T 8b3121fce1c44d16a0b083313dc67b7a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.985883 27693 raft_consensus.cc:383] T 8b3121fce1c44d16a0b083313dc67b7a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:19.985994 27693 raft_consensus.cc:738] T 8b3121fce1c44d16a0b083313dc67b7a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:19.986368 27693 consensus_queue.cc:260] T 8b3121fce1c44d16a0b083313dc67b7a P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:19.988183 27063 catalog_manager.cc:5582] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66). New cstate: current_term: 1 leader_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:19.990847 27693 ts_tablet_manager.cc:1428] T 8b3121fce1c44d16a0b083313dc67b7a P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250903 20:05:19.994053 27693 tablet_bootstrap.cc:492] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:19.999812 27693 tablet_bootstrap.cc:654] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:20.000627 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 488) took 1744 ms. Trace:
I20250903 20:05:20.000803 27687 rpcz_store.cc:276] 0903 20:05:18.256096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.256154 (+ 58us) service_pool.cc:225] Handling call
0903 20:05:20.000378 (+1744224us) ts_tablet_manager.cc:1145] Deleting tablet fa05f54ebc324b40855fa3288760e9e9
0903 20:05:20.000395 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:20.000563 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1743748}
W20250903 20:05:20.001511 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet fa05f54ebc324b40855fa3288760e9e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fa05f54ebc324b40855fa3288760e9e9 already in progress: creating tablet
I20250903 20:05:20.002070 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 489) took 1745 ms. Trace:
I20250903 20:05:20.002328 27687 rpcz_store.cc:276] 0903 20:05:18.256267 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.256412 (+ 145us) service_pool.cc:225] Handling call
0903 20:05:20.001874 (+1745462us) ts_tablet_manager.cc:1145] Deleting tablet aa9f5e0964b348b7af184f60cd145666
0903 20:05:20.001888 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:20.002008 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1745184}
I20250903 20:05:20.002759 27695 tablet_bootstrap.cc:492] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
W20250903 20:05:20.002962 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet aa9f5e0964b348b7af184f60cd145666 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa9f5e0964b348b7af184f60cd145666 already in progress: creating tablet
I20250903 20:05:20.003587 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 494) took 1603 ms. Trace:
I20250903 20:05:20.004050 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 356) took 4354 ms. Trace:
I20250903 20:05:20.003862 27687 rpcz_store.cc:276] 0903 20:05:18.400301 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.400361 (+ 60us) service_pool.cc:225] Handling call
0903 20:05:20.003328 (+1602967us) ts_tablet_manager.cc:1145] Deleting tablet 0ca6e2165a9d41299c1fe0bfa9475915
0903 20:05:20.003366 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:20.003514 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1602706}
I20250903 20:05:20.004187 27685 rpcz_store.cc:276] 0903 20:05:15.649499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.650079 (+ 580us) service_pool.cc:225] Handling call
0903 20:05:19.864623 (+4214544us) ts_tablet_manager.cc:1145] Deleting tablet a10b079e7d5a440580ac00195fe88c65
0903 20:05:19.864642 (+ 19us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:19.969976 (+105334us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:19.973719 (+ 3743us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:20.003963 (+ 30244us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4214301}
W20250903 20:05:20.004626 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 0ca6e2165a9d41299c1fe0bfa9475915 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ca6e2165a9d41299c1fe0bfa9475915 already in progress: creating tablet
I20250903 20:05:20.004789 27685 tablet_replica.cc:331] T fdc77810f27a40758e063b8b1150ab71 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:20.004873 27695 tablet_bootstrap.cc:654] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:20.004904 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet a10b079e7d5a440580ac00195fe88c65 (table default.Table_9b20469406c94986ab496ea3e678b586 [id=6d6f5e8cadf9445d8695c075cfabb2ea]) successfully deleted
I20250903 20:05:20.005026 27685 raft_consensus.cc:2241] T fdc77810f27a40758e063b8b1150ab71 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:20.005264 27685 raft_consensus.cc:2270] T fdc77810f27a40758e063b8b1150ab71 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:20.005545 27687 tablet_replica.cc:331] T eced730bdbc14a8c9f6b7b7e53435fd0 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:20.005929 27685 ts_tablet_manager.cc:1905] T fdc77810f27a40758e063b8b1150ab71 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:20.005982 27687 raft_consensus.cc:2241] T eced730bdbc14a8c9f6b7b7e53435fd0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:20.006726 27687 raft_consensus.cc:2270] T eced730bdbc14a8c9f6b7b7e53435fd0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:20.007223 27063 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet e8bc99de4ffe43d58351d170b7c61d8a (table default.Table_c5256f043b61433b8debe9e13625bd0d [id=416ff29f66d14564826a6734dde1c604]). Expected version 2 got 0
I20250903 20:05:20.008355 27684 ts_tablet_manager.cc:1918] T deb84fc3465443d99c11097e53f5f575 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:20.008621 27684 log.cc:1199] T deb84fc3465443d99c11097e53f5f575 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/deb84fc3465443d99c11097e53f5f575
I20250903 20:05:20.009083 27684 ts_tablet_manager.cc:1939] T deb84fc3465443d99c11097e53f5f575 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:20.010125 27684 tablet_replica.cc:331] T fdc77810f27a40758e063b8b1150ab71 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:20.010516 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet deb84fc3465443d99c11097e53f5f575 (table default.Table_a642c53ef0da461ba2f33dd4010e7129 [id=95919dc5590942bd8e47047598b6d6bc]) successfully deleted
I20250903 20:05:20.011139 27692 tablet_bootstrap.cc:492] T 1fef9400227449ed892daff588999caa P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:20.011370 27692 ts_tablet_manager.cc:1397] T 1fef9400227449ed892daff588999caa P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.221s user 0.023s sys 0.019s
I20250903 20:05:20.011348 27687 ts_tablet_manager.cc:1905] T eced730bdbc14a8c9f6b7b7e53435fd0 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:20.012243 27692 raft_consensus.cc:357] T 1fef9400227449ed892daff588999caa P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:20.012459 27692 raft_consensus.cc:383] T 1fef9400227449ed892daff588999caa P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:20.012539 27692 raft_consensus.cc:738] T 1fef9400227449ed892daff588999caa P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:20.012841 27692 consensus_queue.cc:260] T 1fef9400227449ed892daff588999caa P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:20.015636 27692 ts_tablet_manager.cc:1428] T 1fef9400227449ed892daff588999caa P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250903 20:05:20.015972 27692 tablet_bootstrap.cc:492] T 3f78a951bb024cf290116b0fe438ddb4 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:20.016923 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet 07565ea2b5f34dcfb407ff3bfde17243 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:20.017231 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 0bdcbaa0169c4ab7ba6ec038b50482fe with delete_type TABLET_DATA_DELETED (replaced by tablet 1bcb9d5f010f4297bab78c0183889238) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:20.017611 27496 tablet_service.cc:1515] Processing DeleteTablet for tablet 1fef9400227449ed892daff588999caa with delete_type TABLET_DATA_DELETED (replaced by tablet 67ca1dcf6fc7435db555392235f9900f) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:20.020879 28260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 20:05:20.022536 27692 tablet_bootstrap.cc:654] T 3f78a951bb024cf290116b0fe438ddb4 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:20.010452 27684 raft_consensus.cc:2241] T fdc77810f27a40758e063b8b1150ab71 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:20.023744 27685 ts_tablet_manager.cc:1918] T fdc77810f27a40758e063b8b1150ab71 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:20.023893 27685 log.cc:1199] T fdc77810f27a40758e063b8b1150ab71 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/fdc77810f27a40758e063b8b1150ab71
I20250903 20:05:20.024101 27684 raft_consensus.cc:2270] T fdc77810f27a40758e063b8b1150ab71 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:20.025717 27685 ts_tablet_manager.cc:1939] T fdc77810f27a40758e063b8b1150ab71 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:20.026430 27684 ts_tablet_manager.cc:1905] T fdc77810f27a40758e063b8b1150ab71 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:20.026633 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 354) took 4377 ms. Trace:
I20250903 20:05:20.026762 27685 rpcz_store.cc:276] 0903 20:05:15.649262 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.650365 (+ 1103us) service_pool.cc:225] Handling call
0903 20:05:20.004446 (+4354081us) ts_tablet_manager.cc:1145] Deleting tablet fdc77810f27a40758e063b8b1150ab71
0903 20:05:20.004690 (+ 244us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:20.018301 (+ 13611us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:20.023584 (+ 5283us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:20.026576 (+ 2992us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4353885}
I20250903 20:05:20.027251 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet fdc77810f27a40758e063b8b1150ab71 (table default.Table_9c5c22a18c1d4913abdfab7d6a589058 [id=84da55ea3b4946f0b87d8d6389c63403]) successfully deleted
I20250903 20:05:20.030820 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 357) took 4343 ms. Trace:
I20250903 20:05:20.031744 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_191beed070214f6eac98ee16e25e34f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:20.030965 27685 rpcz_store.cc:276] 0903 20:05:15.686973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.687067 (+ 94us) service_pool.cc:225] Handling call
0903 20:05:20.030645 (+4343578us) ts_tablet_manager.cc:1145] Deleting tablet 1bad3c509460422396a17a3c4ae3d1e8
0903 20:05:20.030657 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:20.030783 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4343259}
W20250903 20:05:20.032413 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 1bad3c509460422396a17a3c4ae3d1e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1bad3c509460422396a17a3c4ae3d1e8 already in progress: creating tablet
W20250903 20:05:20.032801 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_191beed070214f6eac98ee16e25e34f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.033083 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_83be2fe3517249a39f2aa821588442b9" } new_table_name: "default.Table_0b5d0984523e4daba55e75727cdf4487" modify_external_catalogs: true
I20250903 20:05:20.033936 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 358) took 4336 ms. Trace:
I20250903 20:05:20.034052 27685 rpcz_store.cc:276] 0903 20:05:15.697096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.697771 (+ 675us) service_pool.cc:225] Handling call
0903 20:05:20.033783 (+4336012us) ts_tablet_manager.cc:1145] Deleting tablet ed096b452ba747f89e91edc15e6e3ac1
0903 20:05:20.033794 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:20.033905 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4335752}
W20250903 20:05:20.034503 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet ed096b452ba747f89e91edc15e6e3ac1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed096b452ba747f89e91edc15e6e3ac1 already in progress: creating tablet
I20250903 20:05:20.035454 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 359) took 4338 ms. Trace:
I20250903 20:05:20.036052 27685 rpcz_store.cc:276] 0903 20:05:15.697216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.698070 (+ 854us) service_pool.cc:225] Handling call
0903 20:05:20.035301 (+4337231us) ts_tablet_manager.cc:1145] Deleting tablet e9eb24dd5a6243c49a3389320a3f78a0
0903 20:05:20.035310 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:20.035406 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4336964}
I20250903 20:05:20.036619 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 360) took 4339 ms. Trace:
W20250903 20:05:20.036629 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet e9eb24dd5a6243c49a3389320a3f78a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9eb24dd5a6243c49a3389320a3f78a0 already in progress: creating tablet
I20250903 20:05:20.036814 27695 tablet_bootstrap.cc:492] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:20.037266 27695 ts_tablet_manager.cc:1397] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.003s
I20250903 20:05:20.038034 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_217752fec9f1486c99d277d26ede63ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:20.038240 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_20274d6f2af744d485064ee2ce5d22d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:20.038705 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_217752fec9f1486c99d277d26ede63ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.036924 27685 rpcz_store.cc:276] 0903 20:05:15.697365 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.698382 (+ 1017us) service_pool.cc:225] Handling call
0903 20:05:20.036393 (+4338011us) ts_tablet_manager.cc:1145] Deleting tablet 1fbe54dc40dd47daa109ceeb07546b0a
0903 20:05:20.036401 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:20.036587 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4337832}
I20250903 20:05:20.039173 27685 tablet_replica.cc:331] T 5bc3b9771a0042fd87e706204dcbb712 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
W20250903 20:05:20.039444 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 1fbe54dc40dd47daa109ceeb07546b0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fbe54dc40dd47daa109ceeb07546b0a already in progress: creating tablet
I20250903 20:05:20.039467 27685 raft_consensus.cc:2241] T 5bc3b9771a0042fd87e706204dcbb712 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:20.039763 27685 raft_consensus.cc:2270] T 5bc3b9771a0042fd87e706204dcbb712 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:20.038713 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20274d6f2af744d485064ee2ce5d22d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.040468 27685 ts_tablet_manager.cc:1905] T 5bc3b9771a0042fd87e706204dcbb712 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:20.038012 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_73e9595bc2864cbfa76e06557a740f77" } new_table_name: "default.Table_b68da0901fe14d3198aeb59b315c7acd" modify_external_catalogs: true
I20250903 20:05:20.042419 27693 tablet_bootstrap.cc:492] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:20.042615 27693 ts_tablet_manager.cc:1397] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.049s user 0.006s sys 0.000s
W20250903 20:05:20.042706 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a253e3399622409e92299069b19d3698 consensus state has no leader
I20250903 20:05:20.043612 27693 raft_consensus.cc:357] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:20.043815 27693 raft_consensus.cc:383] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:20.043880 27693 raft_consensus.cc:738] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:20.046717 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_0eded5731b1944de87e65ac1b3298371"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:20.047446 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0eded5731b1944de87e65ac1b3298371 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.049129 27057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_060249fde8eb4bd88887054d8691e260" } new_table_name: "default.Table_1d4af1f6213b4b4db3bfaee9e1ede2d0" modify_external_catalogs: true
I20250903 20:05:20.049620 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_76100d98b0774693ab71b084505d007a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:20.050187 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_76100d98b0774693ab71b084505d007a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.045987 27695 raft_consensus.cc:357] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:20.053977 27695 raft_consensus.cc:383] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:20.054093 27695 raft_consensus.cc:738] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:20.054786 27695 consensus_queue.cc:260] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:20.056926 27693 consensus_queue.cc:260] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:20.058296 27693 ts_tablet_manager.cc:1428] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
W20250903 20:05:20.059218 27066 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 18231d53e9db46059da678f3f8b67a52 has no consensus state
I20250903 20:05:20.063730 27693 tablet_bootstrap.cc:492] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:20.064267 27695 ts_tablet_manager.cc:1428] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20250903 20:05:20.067335 27693 tablet_bootstrap.cc:654] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:20.069787 27692 tablet_bootstrap.cc:492] T 3f78a951bb024cf290116b0fe438ddb4 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:20.069955 27692 ts_tablet_manager.cc:1397] T 3f78a951bb024cf290116b0fe438ddb4 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.054s user 0.006s sys 0.000s
I20250903 20:05:20.070757 27692 raft_consensus.cc:357] T 3f78a951bb024cf290116b0fe438ddb4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:20.071338 27695 tablet_bootstrap.cc:492] T e9eb24dd5a6243c49a3389320a3f78a0 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:20.070950 27692 raft_consensus.cc:383] T 3f78a951bb024cf290116b0fe438ddb4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:20.073350 27684 ts_tablet_manager.cc:1918] T fdc77810f27a40758e063b8b1150ab71 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:20.073714 27684 log.cc:1199] T fdc77810f27a40758e063b8b1150ab71 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/fdc77810f27a40758e063b8b1150ab71
I20250903 20:05:20.074131 27684 ts_tablet_manager.cc:1939] T fdc77810f27a40758e063b8b1150ab71 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:20.075265 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet fdc77810f27a40758e063b8b1150ab71 (table default.Table_9c5c22a18c1d4913abdfab7d6a589058 [id=84da55ea3b4946f0b87d8d6389c63403]) successfully deleted
I20250903 20:05:20.073349 27692 raft_consensus.cc:738] T 3f78a951bb024cf290116b0fe438ddb4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:20.075783 27692 consensus_queue.cc:260] T 3f78a951bb024cf290116b0fe438ddb4 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:20.077080 27057 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 13496efe34944b7280e7a684f180e6e0 consensus state has no leader
I20250903 20:05:20.077873 27066 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet de8ac9f984be46ec8f021928de12a090 (table default.Table_eaff993b7bb7437580aae238faa74b3e [id=2650c8259ef4489695c56b6a2ba898d3]). Expected version 2 got 0
I20250903 20:05:20.078614 27496 tablet_service.cc:1468] Processing CreateTablet for tablet 19f65041c35f447a962d6365d0eaf3a6 (DEFAULT_TABLE table=default.Table_aa3dd415897242eb828a0b64f8fc6ad7 [id=c24968f3a270497bbf6bd82a0ec71f3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.079080 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19f65041c35f447a962d6365d0eaf3a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.081441 27685 ts_tablet_manager.cc:1918] T 5bc3b9771a0042fd87e706204dcbb712 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:20.081638 27685 log.cc:1199] T 5bc3b9771a0042fd87e706204dcbb712 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/5bc3b9771a0042fd87e706204dcbb712
I20250903 20:05:20.082147 27685 ts_tablet_manager.cc:1939] T 5bc3b9771a0042fd87e706204dcbb712 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:20.083010 27350 tablet_service.cc:1515] Processing DeleteTablet for tablet deb84fc3465443d99c11097e53f5f575 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:20.083225 27695 tablet_bootstrap.cc:654] T e9eb24dd5a6243c49a3389320a3f78a0 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:20.086182 27495 tablet_service.cc:1468] Processing CreateTablet for tablet ef7dd1ec6fcc47ec81facd6ae0207b97 (DEFAULT_TABLE table=default.Table_00da0bb3fb0c4aa1b29a87d3c8844955 [id=1c8c21dceeff4f1493623a901e857456]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.086683 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef7dd1ec6fcc47ec81facd6ae0207b97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.087018 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 19f65041c35f447a962d6365d0eaf3a6 (DEFAULT_TABLE table=default.Table_aa3dd415897242eb828a0b64f8fc6ad7 [id=c24968f3a270497bbf6bd82a0ec71f3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.087453 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19f65041c35f447a962d6365d0eaf3a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.090698 27692 ts_tablet_manager.cc:1428] T 3f78a951bb024cf290116b0fe438ddb4 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.021s user 0.003s sys 0.000s
I20250903 20:05:20.090978 27350 tablet_service.cc:1468] Processing CreateTablet for tablet 19f65041c35f447a962d6365d0eaf3a6 (DEFAULT_TABLE table=default.Table_aa3dd415897242eb828a0b64f8fc6ad7 [id=c24968f3a270497bbf6bd82a0ec71f3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.091420 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19f65041c35f447a962d6365d0eaf3a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.093649 27692 tablet_bootstrap.cc:492] T 57a1167457b84ea8a493be232b1d088a P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:20.096143 27692 tablet_bootstrap.cc:654] T 57a1167457b84ea8a493be232b1d088a P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:20.097081 27684 tablet_replica.cc:331] T 8b3121fce1c44d16a0b083313dc67b7a P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:20.097352 27684 raft_consensus.cc:2241] T 8b3121fce1c44d16a0b083313dc67b7a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:20.097574 27684 raft_consensus.cc:2270] T 8b3121fce1c44d16a0b083313dc67b7a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:20.098512 27684 ts_tablet_manager.cc:1905] T 8b3121fce1c44d16a0b083313dc67b7a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:20.098663 27693 tablet_bootstrap.cc:492] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:20.098825 27693 ts_tablet_manager.cc:1397] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20250903 20:05:20.099709 27693 raft_consensus.cc:357] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:20.099907 27693 raft_consensus.cc:383] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:20.100011 27693 raft_consensus.cc:738] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:20.100083 27355 tablet_service.cc:1468] Processing CreateTablet for tablet ef7dd1ec6fcc47ec81facd6ae0207b97 (DEFAULT_TABLE table=default.Table_00da0bb3fb0c4aa1b29a87d3c8844955 [id=1c8c21dceeff4f1493623a901e857456]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.100256 27693 consensus_queue.cc:260] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:20.100515 27355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef7dd1ec6fcc47ec81facd6ae0207b97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.100786 27693 ts_tablet_manager.cc:1428] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250903 20:05:20.101987 27693 tablet_bootstrap.cc:492] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:20.102751 27220 tablet_service.cc:1468] Processing CreateTablet for tablet ef7dd1ec6fcc47ec81facd6ae0207b97 (DEFAULT_TABLE table=default.Table_00da0bb3fb0c4aa1b29a87d3c8844955 [id=1c8c21dceeff4f1493623a901e857456]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.102994 27495 tablet_service.cc:1468] Processing CreateTablet for tablet 18231d53e9db46059da678f3f8b67a52 (DEFAULT_TABLE table=default.Table_b68da0901fe14d3198aeb59b315c7acd [id=fe24472bb8e74884b9bf90ccdefd6d89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.103153 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef7dd1ec6fcc47ec81facd6ae0207b97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.103397 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18231d53e9db46059da678f3f8b67a52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.103885 27693 tablet_bootstrap.cc:654] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:20.106025 27684 ts_tablet_manager.cc:1918] T 8b3121fce1c44d16a0b083313dc67b7a P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:20.106163 27684 log.cc:1199] T 8b3121fce1c44d16a0b083313dc67b7a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/8b3121fce1c44d16a0b083313dc67b7a
I20250903 20:05:20.109081 27692 tablet_bootstrap.cc:492] T 57a1167457b84ea8a493be232b1d088a P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:20.109236 27692 ts_tablet_manager.cc:1397] T 57a1167457b84ea8a493be232b1d088a P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20250903 20:05:20.110035 27692 raft_consensus.cc:357] T 57a1167457b84ea8a493be232b1d088a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:20.110337 27692 raft_consensus.cc:383] T 57a1167457b84ea8a493be232b1d088a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:20.110446 27692 raft_consensus.cc:738] T 57a1167457b84ea8a493be232b1d088a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:20.110800 27692 consensus_queue.cc:260] T 57a1167457b84ea8a493be232b1d088a P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:20.113930 27352 tablet_service.cc:1468] Processing CreateTablet for tablet 18231d53e9db46059da678f3f8b67a52 (DEFAULT_TABLE table=default.Table_b68da0901fe14d3198aeb59b315c7acd [id=fe24472bb8e74884b9bf90ccdefd6d89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.114421 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18231d53e9db46059da678f3f8b67a52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.116575 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 361) took 4414 ms. Trace:
I20250903 20:05:20.116688 27685 rpcz_store.cc:276] 0903 20:05:15.701900 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.701947 (+ 47us) service_pool.cc:225] Handling call
0903 20:05:20.039108 (+4337161us) ts_tablet_manager.cc:1145] Deleting tablet 5bc3b9771a0042fd87e706204dcbb712
0903 20:05:20.039116 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:20.044135 (+ 5019us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:20.081263 (+ 37128us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:20.116511 (+ 35248us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4336863}
I20250903 20:05:20.117321 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 5bc3b9771a0042fd87e706204dcbb712 (table default.Table_459903503b5148d09d3cb8f2f5519fab [id=3c69612aaf164dc899dcc252d85636f6]) successfully deleted
I20250903 20:05:20.117627 27692 ts_tablet_manager.cc:1428] T 57a1167457b84ea8a493be232b1d088a P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20250903 20:05:20.118606 27224 tablet_service.cc:1468] Processing CreateTablet for tablet 18231d53e9db46059da678f3f8b67a52 (DEFAULT_TABLE table=default.Table_b68da0901fe14d3198aeb59b315c7acd [id=fe24472bb8e74884b9bf90ccdefd6d89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.119038 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18231d53e9db46059da678f3f8b67a52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.119254 27496 tablet_service.cc:1468] Processing CreateTablet for tablet 5175dbfc0c8a4c438524cff8556dded0 (DEFAULT_TABLE table=default.Table_7713d2cb69d74ea895a8c86a04ff9c50 [id=243455314e1f4b3fbfef1f2db2648b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.119679 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5175dbfc0c8a4c438524cff8556dded0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.119864 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet 3f78a951bb024cf290116b0fe438ddb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:20.122697 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 5175dbfc0c8a4c438524cff8556dded0 (DEFAULT_TABLE table=default.Table_7713d2cb69d74ea895a8c86a04ff9c50 [id=243455314e1f4b3fbfef1f2db2648b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.123126 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5175dbfc0c8a4c438524cff8556dded0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.125629 27222 tablet_service.cc:1468] Processing CreateTablet for tablet 5175dbfc0c8a4c438524cff8556dded0 (DEFAULT_TABLE table=default.Table_7713d2cb69d74ea895a8c86a04ff9c50 [id=243455314e1f4b3fbfef1f2db2648b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.126097 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5175dbfc0c8a4c438524cff8556dded0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.130678 27692 tablet_bootstrap.cc:492] T 4c543ac4df33429b9787c6d32bf1ebeb P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:20.132581 27350 tablet_service.cc:1515] Processing DeleteTablet for tablet 8b3121fce1c44d16a0b083313dc67b7a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:20.132891 27692 tablet_bootstrap.cc:654] T 4c543ac4df33429b9787c6d32bf1ebeb P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:20.134245 27355 tablet_service.cc:1515] Processing DeleteTablet for tablet 1fef9400227449ed892daff588999caa with delete_type TABLET_DATA_DELETED (replaced by tablet 67ca1dcf6fc7435db555392235f9900f) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:20.134430 27684 ts_tablet_manager.cc:1939] T 8b3121fce1c44d16a0b083313dc67b7a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:20.135663 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 8b3121fce1c44d16a0b083313dc67b7a (table default.Table_41538df113834b45ae2213b105fb9706 [id=158a2fc69fce4c54917277cc90439126]) successfully deleted
I20250903 20:05:20.135916 27693 tablet_bootstrap.cc:492] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:20.136085 27693 ts_tablet_manager.cc:1397] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.002s
W20250903 20:05:20.136729 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet f31e3e5ea72940de953c7dd787515c62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f31e3e5ea72940de953c7dd787515c62 already in progress: creating tablet
I20250903 20:05:20.136921 27693 raft_consensus.cc:357] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:20.137197 27693 raft_consensus.cc:383] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:20.137171 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet 57a1167457b84ea8a493be232b1d088a with delete_type TABLET_DATA_DELETED (replaced by tablet f326ec776ba446be8d5dda5c1dc5b6ec) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:20.137285 27693 raft_consensus.cc:738] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:20.137566 27693 consensus_queue.cc:260] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:20.137676 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet e3e82f37c6fa4bebb0115e6087919a27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3e82f37c6fa4bebb0115e6087919a27 already in progress: creating tablet
W20250903 20:05:20.138384 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 3919f7f95a954f59b68f56d0051c0109 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3919f7f95a954f59b68f56d0051c0109 already in progress: creating tablet
I20250903 20:05:20.139101 27693 ts_tablet_manager.cc:1428] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20250903 20:05:20.139647 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet bc98762790f441038972366b1cc2c55b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc98762790f441038972366b1cc2c55b already in progress: creating tablet
I20250903 20:05:20.139822 27496 tablet_service.cc:1468] Processing CreateTablet for tablet cac38daf002c4ae698d72903a277a587 (DEFAULT_TABLE table=default.Table_b60456f0147b407584bb3aa4321b1fe0 [id=81a5ab39f95046b4a9b757c49529fcd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.140123 27495 tablet_service.cc:1468] Processing CreateTablet for tablet 8e55dc800c234dffb135257722ccfba6 (DEFAULT_TABLE table=default.Table_20c48527dac44af39c438cb74d86dcc4 [id=7c7d1a7f3a094d1a97e3fff60f777359]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.140203 27693 tablet_bootstrap.cc:492] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:20.140364 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cac38daf002c4ae698d72903a277a587. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:20.140730 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 314856bf269f4df7aa25f5a30cec0d5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 314856bf269f4df7aa25f5a30cec0d5b already in progress: creating tablet
I20250903 20:05:20.141038 27355 tablet_service.cc:1468] Processing CreateTablet for tablet 8e55dc800c234dffb135257722ccfba6 (DEFAULT_TABLE table=default.Table_20c48527dac44af39c438cb74d86dcc4 [id=7c7d1a7f3a094d1a97e3fff60f777359]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.141469 27355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e55dc800c234dffb135257722ccfba6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:20.143246 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 514e382479014fa7bf1f404efff11c86 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 514e382479014fa7bf1f404efff11c86 already in progress: creating tablet
I20250903 20:05:20.143591 27684 tablet_replica.cc:331] T 1fef9400227449ed892daff588999caa P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:20.143746 27693 tablet_bootstrap.cc:654] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:20.143957 27684 raft_consensus.cc:2241] T 1fef9400227449ed892daff588999caa P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:20.144234 27684 raft_consensus.cc:2270] T 1fef9400227449ed892daff588999caa P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:20.145347 27684 ts_tablet_manager.cc:1905] T 1fef9400227449ed892daff588999caa P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:20.146579 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e55dc800c234dffb135257722ccfba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.146916 27685 tablet_replica.cc:331] T fbfabb6ff1714d2e921f45256b543759 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:20.147550 27059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_362ac8cb413e426ebf154eae4b6b5c7e" } modify_external_catalogs: true
I20250903 20:05:20.147784 27059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_362ac8cb413e426ebf154eae4b6b5c7e" } modify_external_catalogs: true
I20250903 20:05:20.149608 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_359258cdcd734ef4b3463c80fa1c582a" } modify_external_catalogs: true
I20250903 20:05:20.149824 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_359258cdcd734ef4b3463c80fa1c582a" } modify_external_catalogs: true
I20250903 20:05:20.149998 27685 raft_consensus.cc:2241] T fbfabb6ff1714d2e921f45256b543759 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:20.154498 27685 raft_consensus.cc:2270] T fbfabb6ff1714d2e921f45256b543759 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:20.154762 27222 tablet_service.cc:1468] Processing CreateTablet for tablet 8e55dc800c234dffb135257722ccfba6 (DEFAULT_TABLE table=default.Table_20c48527dac44af39c438cb74d86dcc4 [id=7c7d1a7f3a094d1a97e3fff60f777359]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.155243 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_c891a3989f9a42dbb10ac8788775a83a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:20.155602 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e55dc800c234dffb135257722ccfba6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:20.156687 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c891a3989f9a42dbb10ac8788775a83a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.162637 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3dee346162e54d5190c8e658e9483c9a" } modify_external_catalogs: true
I20250903 20:05:20.163193 27065 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_bf733feb53404025b54ce4e7437a2579" } modify_external_catalogs: true
I20250903 20:05:20.164341 27064 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_8a37ddf23e9041d59958722b82a977f9" } modify_external_catalogs: true
I20250903 20:05:20.165030 27684 ts_tablet_manager.cc:1918] T 1fef9400227449ed892daff588999caa P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:20.165192 27684 log.cc:1199] T 1fef9400227449ed892daff588999caa P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/1fef9400227449ed892daff588999caa
I20250903 20:05:20.165611 27684 ts_tablet_manager.cc:1939] T 1fef9400227449ed892daff588999caa P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:20.168793 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 1fef9400227449ed892daff588999caa (table default.Table_bd23bbbddf114a4286a815ee4468be81 [id=5568f788d1414c83b4359b5adf28632a]) successfully deleted
I20250903 20:05:20.169453 27684 tablet_replica.cc:331] T 3f78a951bb024cf290116b0fe438ddb4 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:20.169926 27684 raft_consensus.cc:2241] T 3f78a951bb024cf290116b0fe438ddb4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:20.170389 27684 raft_consensus.cc:2270] T 3f78a951bb024cf290116b0fe438ddb4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:20.171573 27684 ts_tablet_manager.cc:1905] T 3f78a951bb024cf290116b0fe438ddb4 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:20.173019 28234 raft_consensus.cc:491] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:20.173247 28234 raft_consensus.cc:513] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:20.173946 28234 leader_election.cc:290] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:20.174791 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "57a1167457b84ea8a493be232b1d088a" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:20.175074 27522 raft_consensus.cc:2466] T 57a1167457b84ea8a493be232b1d088a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:20.175606 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_dbaa47c36c6f43fab8828747862ebbaa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:20.176257 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbaa47c36c6f43fab8828747862ebbaa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.176605 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c750811a8d6c4069a3192ca531cf5712" } new_table_name: "default.Table_1dd1b4a75c914e6a88058266d2a7e98a" modify_external_catalogs: true
I20250903 20:05:20.177531 27320 leader_election.cc:304] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:20.177994 28234 raft_consensus.cc:2802] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:20.178115 28234 raft_consensus.cc:491] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:20.178234 28234 raft_consensus.cc:3058] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:20.179733 27350 tablet_service.cc:1468] Processing CreateTablet for tablet cac38daf002c4ae698d72903a277a587 (DEFAULT_TABLE table=default.Table_b60456f0147b407584bb3aa4321b1fe0 [id=81a5ab39f95046b4a9b757c49529fcd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.180033 27693 tablet_bootstrap.cc:492] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:20.180485 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cac38daf002c4ae698d72903a277a587. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.183141 27684 ts_tablet_manager.cc:1918] T 3f78a951bb024cf290116b0fe438ddb4 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:20.183296 27684 log.cc:1199] T 3f78a951bb024cf290116b0fe438ddb4 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/3f78a951bb024cf290116b0fe438ddb4
I20250903 20:05:20.183468 27693 ts_tablet_manager.cc:1397] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.002s
I20250903 20:05:20.185840 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_faaa45179ec149bcb4e47fb312ee6549"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:20.186501 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 5f2eefeebbfc47249e2f72b7c52e905d
W20250903 20:05:20.186616 28281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 20:05:20.187348 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ef2b637034674c86aa8127fec9c7fc7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:20.135669 27063 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet de8ac9f984be46ec8f021928de12a090 (table default.Table_eaff993b7bb7437580aae238faa74b3e [id=2650c8259ef4489695c56b6a2ba898d3]). Expected version 2 got 0
I20250903 20:05:20.187736 27684 ts_tablet_manager.cc:1939] T 3f78a951bb024cf290116b0fe438ddb4 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
W20250903 20:05:20.187908 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef2b637034674c86aa8127fec9c7fc7b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.188328 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_20c48527dac44af39c438cb74d86dcc4" } new_table_name: "default.Table_fc0b0b9655f24509a2ebf80e1f026277" modify_external_catalogs: true
I20250903 20:05:20.189239 27692 tablet_bootstrap.cc:492] T 4c543ac4df33429b9787c6d32bf1ebeb P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:20.189417 27692 ts_tablet_manager.cc:1397] T 4c543ac4df33429b9787c6d32bf1ebeb P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.060s user 0.002s sys 0.012s
I20250903 20:05:20.189728 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 4789bd72aa454b639df76bd5ad379850
I20250903 20:05:20.190382 27692 raft_consensus.cc:357] T 4c543ac4df33429b9787c6d32bf1ebeb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:20.190574 27692 raft_consensus.cc:383] T 4c543ac4df33429b9787c6d32bf1ebeb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:20.190673 27692 raft_consensus.cc:738] T 4c543ac4df33429b9787c6d32bf1ebeb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:20.190593 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_de42081b245243918dff0bc0301b027a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:20.190959 27692 consensus_queue.cc:260] T 4c543ac4df33429b9787c6d32bf1ebeb P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:20.191201 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de42081b245243918dff0bc0301b027a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.191561 27692 ts_tablet_manager.cc:1428] T 4c543ac4df33429b9787c6d32bf1ebeb P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250903 20:05:20.192476 27065 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_bf733feb53404025b54ce4e7437a2579" } modify_external_catalogs: true
I20250903 20:05:20.193727 27503 tablet_service.cc:1515] Processing DeleteTablet for tablet 4789bd72aa454b639df76bd5ad379850 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:20.194178 27488 tablet_service.cc:1468] Processing CreateTablet for tablet c0ea33c5c2444c57a86086d49a85db2b (DEFAULT_TABLE table=default.Table_5d5e2d80c015418db7956702c08cac42 [id=ea9e6eb03085487a910dc84894196d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.194473 27692 tablet_bootstrap.cc:492] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:20.196617 27692 tablet_bootstrap.cc:654] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:20.199142 27496 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f2eefeebbfc47249e2f72b7c52e905d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:20.194178 27491 tablet_service.cc:1468] Processing CreateTablet for tablet 153b9fef87f24b96a97d33a7c14f662e (DEFAULT_TABLE table=default.Table_c749638286ff4a0db8b80342622dca67 [id=2c0e64114adc44dab5c2271bde165bc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.200114 27503 tablet_service.cc:1468] Processing CreateTablet for tablet cc50edd0891f4e48bc0136c12cde45da (DEFAULT_TABLE table=default.Table_09e3fcf1b8ba4f75be49736c7c4b5f30 [id=d317d133362c402db605e373d094f49d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.200843 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0ea33c5c2444c57a86086d49a85db2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.193756 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3dee346162e54d5190c8e658e9483c9a" } modify_external_catalogs: true
I20250903 20:05:20.221993 28184 server_base.cc:1047] running on GCE node
W20250903 20:05:20.222869 28292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 20:05:20.216363 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 64d53f2071b0449ca61caa6c264a3641
I20250903 20:05:20.225464 27692 tablet_bootstrap.cc:492] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:20.225633 27692 ts_tablet_manager.cc:1397] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.008s
I20250903 20:05:20.219933 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 3f78a951bb024cf290116b0fe438ddb4 (table default.Table_835ced1f8b194192809c3b6fbf4d1d96 [id=23f448924ec04ef08a541a14bd2ce5df]) successfully deleted
I20250903 20:05:20.193812 27064 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_8a37ddf23e9041d59958722b82a977f9" } modify_external_catalogs: true
I20250903 20:05:20.226493 27692 raft_consensus.cc:357] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:20.226701 27692 raft_consensus.cc:383] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:20.226783 27692 raft_consensus.cc:738] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:20.227005 27496 tablet_service.cc:1515] Processing DeleteTablet for tablet 64d53f2071b0449ca61caa6c264a3641 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:20.227030 27692 consensus_queue.cc:260] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:20.227566 27692 ts_tablet_manager.cc:1428] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250903 20:05:20.229061 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 249362a0b8304719863bcb82c47273b3 consensus state has no leader
I20250903 20:05:20.230360 27496 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c543ac4df33429b9787c6d32bf1ebeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:20.193547 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_faaa45179ec149bcb4e47fb312ee6549 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.234306 27685 ts_tablet_manager.cc:1905] T fbfabb6ff1714d2e921f45256b543759 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:20.235724 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8e55dc800c234dffb135257722ccfba6 consensus state has no leader
I20250903 20:05:20.241843 28234 raft_consensus.cc:513] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:20.243466 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "57a1167457b84ea8a493be232b1d088a" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:20.243671 27522 raft_consensus.cc:3058] T 57a1167457b84ea8a493be232b1d088a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:20.244313 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet 4789bd72aa454b639df76bd5ad379850 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:20.246739 27217 tablet_service.cc:1468] Processing CreateTablet for tablet cac38daf002c4ae698d72903a277a587 (DEFAULT_TABLE table=default.Table_b60456f0147b407584bb3aa4321b1fe0 [id=81a5ab39f95046b4a9b757c49529fcd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.247155 27522 raft_consensus.cc:2466] T 57a1167457b84ea8a493be232b1d088a P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:20.247641 27057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b86a1d18f92c4844997ebd323da582b6" } new_table_name: "default.Table_cbe4f0209a3345bc9ae4a50a260346c8" modify_external_catalogs: true
I20250903 20:05:20.246264 28184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 20:05:20.251744 27229 tablet_service.cc:1515] Processing DeleteTablet for tablet 64d53f2071b0449ca61caa6c264a3641 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:20.251986 27224 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f2eefeebbfc47249e2f72b7c52e905d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:20.252230 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cac38daf002c4ae698d72903a277a587. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.252202 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 153b9fef87f24b96a97d33a7c14f662e (DEFAULT_TABLE table=default.Table_c749638286ff4a0db8b80342622dca67 [id=2c0e64114adc44dab5c2271bde165bc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.252326 27212 tablet_service.cc:1468] Processing CreateTablet for tablet d93258c032524b71997389b81846a5be (DEFAULT_TABLE table=default.Table_654e50df426048f8bc1852c3015cb4f5 [id=1b0ce78b0e3d40b0ab9a49662d56701f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.254403 27496 tablet_service.cc:1515] Processing DeleteTablet for tablet e9eb24dd5a6243c49a3389320a3f78a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:20.254637 27244 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "57a1167457b84ea8a493be232b1d088a" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:20.254863 27244 raft_consensus.cc:3058] T 57a1167457b84ea8a493be232b1d088a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:20.256228 27499 tablet_service.cc:1468] Processing CreateTablet for tablet e6d019ca2fcd41309e74cdd3a1e20daf (DEFAULT_TABLE table=default.Table_1b5f4328e47f4393a36d5ed02c2614c4 [id=d2b53e6b021b4506829c4a2d36e7c635]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.256680 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6d019ca2fcd41309e74cdd3a1e20daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.257629 27244 raft_consensus.cc:2466] T 57a1167457b84ea8a493be232b1d088a P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:20.258352 27213 tablet_service.cc:1468] Processing CreateTablet for tablet c0ea33c5c2444c57a86086d49a85db2b (DEFAULT_TABLE table=default.Table_5d5e2d80c015418db7956702c08cac42 [id=ea9e6eb03085487a910dc84894196d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.258535 27497 tablet_service.cc:1468] Processing CreateTablet for tablet 3d875d29e66d48ed872e38d20f30d9c0 (DEFAULT_TABLE table=default.Table_77c890250aaf48f4b658f0f126941bc7 [id=f4642be57818412a9607dc534d31f23f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.258920 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d875d29e66d48ed872e38d20f30d9c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.259057 27496 tablet_service.cc:1468] Processing CreateTablet for tablet d93258c032524b71997389b81846a5be (DEFAULT_TABLE table=default.Table_654e50df426048f8bc1852c3015cb4f5 [id=1b0ce78b0e3d40b0ab9a49662d56701f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.259482 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d93258c032524b71997389b81846a5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.259981 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 1bf570855b204b96acd070587c434463
I20250903 20:05:20.262414 27492 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bf570855b204b96acd070587c434463 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:20.258246 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "57a1167457b84ea8a493be232b1d088a" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:20.263319 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ef2b637034674c86aa8127fec9c7fc7b" } new_table_name: "default.Table_f42f6303480447ddac82d66378c1ce36" modify_external_catalogs: true
I20250903 20:05:20.263633 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_acf2f9a7259946b3bfe30be00f5f0a66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:20.263823 27220 tablet_service.cc:1468] Processing CreateTablet for tablet cc50edd0891f4e48bc0136c12cde45da (DEFAULT_TABLE table=default.Table_09e3fcf1b8ba4f75be49736c7c4b5f30 [id=d317d133362c402db605e373d094f49d]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:20.264310 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_acf2f9a7259946b3bfe30be00f5f0a66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.265702 27249 raft_consensus.cc:2374] T 57a1167457b84ea8a493be232b1d088a P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1. Re-sending same reply.
I20250903 20:05:20.268108 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 259bc73ca9df47e9874692725a60bb1c
I20250903 20:05:20.264883 27229 tablet_service.cc:1468] Processing CreateTablet for tablet e6d019ca2fcd41309e74cdd3a1e20daf (DEFAULT_TABLE table=default.Table_1b5f4328e47f4393a36d5ed02c2614c4 [id=d2b53e6b021b4506829c4a2d36e7c635]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.273258 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 153b9fef87f24b96a97d33a7c14f662e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.278683 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bf570855b204b96acd070587c434463 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:20.278975 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e019b2e0ac534620ad809b330da7038b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:20.279690 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e019b2e0ac534620ad809b330da7038b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.285640 27224 tablet_service.cc:1468] Processing CreateTablet for tablet 3d875d29e66d48ed872e38d20f30d9c0 (DEFAULT_TABLE table=default.Table_77c890250aaf48f4b658f0f126941bc7 [id=f4642be57818412a9607dc534d31f23f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.287019 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 0906165a50b047c8a776b30832149964 (DEFAULT_TABLE table=default.Table_20274d6f2af744d485064ee2ce5d22d6 [id=9c8ccd24b88048aea95c867aae5fcb23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.287500 27227 tablet_service.cc:1468] Processing CreateTablet for tablet 07439db2c3b8470eb2b797cc68376aa7 (DEFAULT_TABLE table=default.Table_b0a0a2dfe31e4961871f334372541d05 [id=30918e7a5f1d43aab5eaddef7c485cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.287923 27215 tablet_service.cc:1468] Processing CreateTablet for tablet c532bb70298547b58a5b88b95d0934a9 (DEFAULT_TABLE table=default.Table_217752fec9f1486c99d277d26ede63ba [id=20fca17dfad1467ebcffefacf4068cbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.288417 27211 tablet_service.cc:1468] Processing CreateTablet for tablet 421dc8af40c94a89bc372e8779872a4e (DEFAULT_TABLE table=default.Table_76100d98b0774693ab71b084505d007a [id=f78623e5b0444dac9c633558aba62f5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.289301 27695 tablet_bootstrap.cc:492] T e9eb24dd5a6243c49a3389320a3f78a0 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:20.289759 28184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 20:05:20.290455 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet 259bc73ca9df47e9874692725a60bb1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:20.290633 27695 ts_tablet_manager.cc:1397] T e9eb24dd5a6243c49a3389320a3f78a0 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.218s user 0.007s sys 0.025s
I20250903 20:05:20.292138 27210 tablet_service.cc:1468] Processing CreateTablet for tablet 8292ae4c5bdd469681f3a3b295b24349 (DEFAULT_TABLE table=default.Table_1310291579564dd18d0b47d1050cf905 [id=c232ef71f26b42779d5849f8bb49bbe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.292527 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc50edd0891f4e48bc0136c12cde45da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.293049 27695 raft_consensus.cc:357] T e9eb24dd5a6243c49a3389320a3f78a0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:20.293128 27228 tablet_service.cc:1468] Processing CreateTablet for tablet 91567f10d90142acb7798757262218f0 (DEFAULT_TABLE table=default.Table_aaea28edb28e4ee991b380a66d0297d3 [id=9e129548408242a49193a98a1a340b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.293655 27222 tablet_service.cc:1468] Processing CreateTablet for tablet ce3be9d44147420d953e716a347352c9 (DEFAULT_TABLE table=default.Table_191beed070214f6eac98ee16e25e34f8 [id=98b7dc72c66e4cfb92bd24ae61c6605f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.294153 27216 tablet_service.cc:1468] Processing CreateTablet for tablet f64ab5cf99af4608906441df4fb6e86c (DEFAULT_TABLE table=default.Table_0eded5731b1944de87e65ac1b3298371 [id=d131083310924d1b9318affd49a69afe]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:20.294786 27057 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b7a6d5d8a97a4769a81ec7224d591f8c consensus state has no leader
W20250903 20:05:20.298349 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aa31f2ddb9f84a8093aa1c6f64b278c9 has no consensus state
I20250903 20:05:20.299528 27493 tablet_service.cc:1468] Processing CreateTablet for tablet 91567f10d90142acb7798757262218f0 (DEFAULT_TABLE table=default.Table_aaea28edb28e4ee991b380a66d0297d3 [id=9e129548408242a49193a98a1a340b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.300064 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91567f10d90142acb7798757262218f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.300839 27211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 421dc8af40c94a89bc372e8779872a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.300849 27695 raft_consensus.cc:383] T e9eb24dd5a6243c49a3389320a3f78a0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:20.301587 27695 raft_consensus.cc:738] T e9eb24dd5a6243c49a3389320a3f78a0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:20.301914 27695 consensus_queue.cc:260] T e9eb24dd5a6243c49a3389320a3f78a0 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:20.302789 27500 tablet_service.cc:1515] Processing DeleteTablet for tablet 259bc73ca9df47e9874692725a60bb1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:20.302965 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce3be9d44147420d953e716a347352c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.305160 27228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91567f10d90142acb7798757262218f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.305963 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0ea33c5c2444c57a86086d49a85db2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.306337 27210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8292ae4c5bdd469681f3a3b295b24349. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.306777 27695 ts_tablet_manager.cc:1428] T e9eb24dd5a6243c49a3389320a3f78a0 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20250903 20:05:20.306777 27492 tablet_service.cc:1468] Processing CreateTablet for tablet ce3be9d44147420d953e716a347352c9 (DEFAULT_TABLE table=default.Table_191beed070214f6eac98ee16e25e34f8 [id=98b7dc72c66e4cfb92bd24ae61c6605f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.307008 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0906165a50b047c8a776b30832149964. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.307226 27492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce3be9d44147420d953e716a347352c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.307237 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c532bb70298547b58a5b88b95d0934a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.307977 27229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6d019ca2fcd41309e74cdd3a1e20daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.309072 28184 hybrid_clock.cc:648] HybridClock initialized: now 1756929920308988 us; error 76 us; skew 500 ppm
I20250903 20:05:20.309620 28184 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 20:05:20.309765 27216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f64ab5cf99af4608906441df4fb6e86c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.299520 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 8292ae4c5bdd469681f3a3b295b24349 (DEFAULT_TABLE table=default.Table_1310291579564dd18d0b47d1050cf905 [id=c232ef71f26b42779d5849f8bb49bbe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.312501 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07439db2c3b8470eb2b797cc68376aa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.313889 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d875d29e66d48ed872e38d20f30d9c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.314513 27695 tablet_bootstrap.cc:492] T ac1b0dc95fca49b4af6697c0a70e9730 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:20.315531 27212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d93258c032524b71997389b81846a5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.316864 27695 tablet_bootstrap.cc:654] T ac1b0dc95fca49b4af6697c0a70e9730 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:20.332193 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8292ae4c5bdd469681f3a3b295b24349. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.337266 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a88f7945592647468a4f4b0da26f5a84" } new_table_name: "default.Table_db5fb46856fa43dc9e96685bc4d691c3" modify_external_catalogs: true
I20250903 20:05:20.338101 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_3c841526bf5745998631d5afd01f5147"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:20.338523 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet e9eb24dd5a6243c49a3389320a3f78a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:20.338697 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c841526bf5745998631d5afd01f5147 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.338940 27490 tablet_service.cc:1468] Processing CreateTablet for tablet 0906165a50b047c8a776b30832149964 (DEFAULT_TABLE table=default.Table_20274d6f2af744d485064ee2ce5d22d6 [id=9c8ccd24b88048aea95c867aae5fcb23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.339002 27489 tablet_service.cc:1468] Processing CreateTablet for tablet c532bb70298547b58a5b88b95d0934a9 (DEFAULT_TABLE table=default.Table_217752fec9f1486c99d277d26ede63ba [id=20fca17dfad1467ebcffefacf4068cbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.339409 27502 tablet_service.cc:1468] Processing CreateTablet for tablet 07439db2c3b8470eb2b797cc68376aa7 (DEFAULT_TABLE table=default.Table_b0a0a2dfe31e4961871f334372541d05 [id=30918e7a5f1d43aab5eaddef7c485cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.339771 27500 tablet_service.cc:1468] Processing CreateTablet for tablet 421dc8af40c94a89bc372e8779872a4e (DEFAULT_TABLE table=default.Table_76100d98b0774693ab71b084505d007a [id=f78623e5b0444dac9c633558aba62f5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.337078 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_d608d44f1a1447488bb7c52ee1456fdb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:20.340269 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d608d44f1a1447488bb7c52ee1456fdb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.340518 27490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0906165a50b047c8a776b30832149964. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.342643 28249 raft_consensus.cc:491] T 07565ea2b5f34dcfb407ff3bfde17243 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:20.342844 28249 raft_consensus.cc:513] T 07565ea2b5f34dcfb407ff3bfde17243 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:20.343712 28249 leader_election.cc:290] T 07565ea2b5f34dcfb407ff3bfde17243 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:20.344645 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07565ea2b5f34dcfb407ff3bfde17243" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:20.345291 27684 tablet_replica.cc:331] T 57a1167457b84ea8a493be232b1d088a P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
W20250903 20:05:20.345250 27183 leader_election.cc:343] T 07565ea2b5f34dcfb407ff3bfde17243 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:20.345517 27684 raft_consensus.cc:2241] T 57a1167457b84ea8a493be232b1d088a P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:20.345690 27684 raft_consensus.cc:2270] T 57a1167457b84ea8a493be232b1d088a P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:20.346096 27491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 153b9fef87f24b96a97d33a7c14f662e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.350695 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc50edd0891f4e48bc0136c12cde45da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.351215 27496 tablet_service.cc:1468] Processing CreateTablet for tablet f64ab5cf99af4608906441df4fb6e86c (DEFAULT_TABLE table=default.Table_0eded5731b1944de87e65ac1b3298371 [id=d131083310924d1b9318affd49a69afe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.351703 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f64ab5cf99af4608906441df4fb6e86c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.370644 28242 raft_consensus.cc:491] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:20.370612 27220 tablet_service.cc:1468] Processing CreateTablet for tablet 197ccb3d4bc54e8f9202fc6e7ae1426c (DEFAULT_TABLE table=default.Table_faaa45179ec149bcb4e47fb312ee6549 [id=d87bd84debbd47efbf7173b191095fe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.371050 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 197ccb3d4bc54e8f9202fc6e7ae1426c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.372186 27692 tablet_bootstrap.cc:492] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:20.372400 27684 ts_tablet_manager.cc:1905] T 57a1167457b84ea8a493be232b1d088a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:20.381767 27685 ts_tablet_manager.cc:1918] T fbfabb6ff1714d2e921f45256b543759 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:20.381935 27685 log.cc:1199] T fbfabb6ff1714d2e921f45256b543759 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/fbfabb6ff1714d2e921f45256b543759
I20250903 20:05:20.382694 28242 raft_consensus.cc:513] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:20.384182 28249 raft_consensus.cc:491] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:20.384366 28249 raft_consensus.cc:513] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:20.385033 28249 leader_election.cc:290] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:20.385104 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9eb24dd5a6243c49a3389320a3f78a0" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:20.385347 27249 raft_consensus.cc:2466] T e9eb24dd5a6243c49a3389320a3f78a0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
W20250903 20:05:20.387370 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d859c0222c3647dab3dddd105e4cf0fc consensus state has no leader
I20250903 20:05:20.386113 27685 ts_tablet_manager.cc:1939] T fbfabb6ff1714d2e921f45256b543759 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:20.388737 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 362) took 4680 ms. Trace:
I20250903 20:05:20.389194 27685 rpcz_store.cc:276] 0903 20:05:15.707714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.707770 (+ 56us) service_pool.cc:225] Handling call
0903 20:05:20.146820 (+4439050us) ts_tablet_manager.cc:1145] Deleting tablet fbfabb6ff1714d2e921f45256b543759
0903 20:05:20.146832 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:20.278616 (+131784us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:20.381616 (+103000us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:20.388610 (+ 6994us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4438790}
I20250903 20:05:20.390424 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet fbfabb6ff1714d2e921f45256b543759 (table default.Table_aab58ad5394041179974552994469305 [id=1889dee4122f4e69ac2e3d22fb4126a6]) successfully deleted
I20250903 20:05:20.391745 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_196cfa99ce59433b85ad81b45d853081" } modify_external_catalogs: true
I20250903 20:05:20.391949 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_196cfa99ce59433b85ad81b45d853081" } modify_external_catalogs: true
I20250903 20:05:20.391358 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 363) took 4669 ms. Trace:
I20250903 20:05:20.395633 27685 rpcz_store.cc:276] 0903 20:05:15.721922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.722172 (+ 250us) service_pool.cc:225] Handling call
0903 20:05:20.390931 (+4668759us) ts_tablet_manager.cc:1145] Deleting tablet 5002b7e8118c49b6a247cb8dc4ca1aea
0903 20:05:20.390943 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:20.391286 (+ 343us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4668436}
I20250903 20:05:20.396306 28234 leader_election.cc:290] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:20.383445 28242 leader_election.cc:290] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:20.396622 28237 raft_consensus.cc:491] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:20.397130 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_58f40fbfe6a946eab6b621584922f4a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:20.397502 28237 raft_consensus.cc:513] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:20.397769 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58f40fbfe6a946eab6b621584922f4a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.398926 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0bdcbaa0169c4ab7ba6ec038b50482fe" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:20.400168 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07439db2c3b8470eb2b797cc68376aa7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:20.400969 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 5002b7e8118c49b6a247cb8dc4ca1aea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5002b7e8118c49b6a247cb8dc4ca1aea already in progress: creating tablet
I20250903 20:05:20.401136 27065 master_service.cc:948] ReplaceTablet: received request to replace tablet a38f65e992884635aae9942681c0dc4f from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:20.401785 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ae457e422e0c4314aeaecd7755413366" } modify_external_catalogs: true
I20250903 20:05:20.402022 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ae457e422e0c4314aeaecd7755413366" } modify_external_catalogs: true
I20250903 20:05:20.398115 28237 leader_election.cc:290] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:20.402449 27320 leader_election.cc:304] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
W20250903 20:05:20.402694 27320 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 155087us
I20250903 20:05:20.403221 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 421dc8af40c94a89bc372e8779872a4e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:20.403795 27321 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 145678us
I20250903 20:05:20.404438 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 3f78a951bb024cf290116b0fe438ddb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:20.406309 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07565ea2b5f34dcfb407ff3bfde17243" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:20.406366 27382 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0bdcbaa0169c4ab7ba6ec038b50482fe" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:20.407114 28208 raft_consensus.cc:491] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:20.406394 28234 raft_consensus.cc:2802] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:20.409094 27489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c532bb70298547b58a5b88b95d0934a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.409703 28208 raft_consensus.cc:513] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:20.410409 28208 leader_election.cc:290] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:20.411301 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9eb24dd5a6243c49a3389320a3f78a0" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:20.414309 28237 raft_consensus.cc:491] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:20.414482 28237 raft_consensus.cc:513] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:20.415112 28237 leader_election.cc:290] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:20.421383 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_34e4bbb060e8432e89a9dfb67a1af00c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:20.421500 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_026e7093a60a495e95489df73d689967"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:20.422122 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34e4bbb060e8432e89a9dfb67a1af00c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.422389 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_39c530a33b00492fbae7db9de5ea26c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:20.422894 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39c530a33b00492fbae7db9de5ea26c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:20.422124 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_026e7093a60a495e95489df73d689967 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.423375 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f2eefeebbfc47249e2f72b7c52e905d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:20.424647 27352 tablet_service.cc:1515] Processing DeleteTablet for tablet 4789bd72aa454b639df76bd5ad379850 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:20.424803 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 847d0a0a3d2b45938cec9058c77f6231
I20250903 20:05:20.426258 27693 raft_consensus.cc:357] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:20.426400 27462 leader_election.cc:304] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:20.426466 27693 raft_consensus.cc:383] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:20.427412 27693 raft_consensus.cc:738] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:20.427682 27693 consensus_queue.cc:260] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:20.430106 27693 ts_tablet_manager.cc:1428] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.245s user 0.001s sys 0.001s
I20250903 20:05:20.430491 27693 tablet_bootstrap.cc:492] T e9eb24dd5a6243c49a3389320a3f78a0 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:20.431171 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_3453a2165fe44a3d93deee7b39853d3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:20.431434 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_50f6b8731a2c4ccb8bf58597acb5b760" } modify_external_catalogs: true
I20250903 20:05:20.431655 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_50f6b8731a2c4ccb8bf58597acb5b760" } modify_external_catalogs: true
I20250903 20:05:20.433427 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet 64d53f2071b0449ca61caa6c264a3641 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:20.433851 27359 tablet_service.cc:1468] Processing CreateTablet for tablet d93258c032524b71997389b81846a5be (DEFAULT_TABLE table=default.Table_654e50df426048f8bc1852c3015cb4f5 [id=1b0ce78b0e3d40b0ab9a49662d56701f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.433996 27347 tablet_service.cc:1468] Processing CreateTablet for tablet e6d019ca2fcd41309e74cdd3a1e20daf (DEFAULT_TABLE table=default.Table_1b5f4328e47f4393a36d5ed02c2614c4 [id=d2b53e6b021b4506829c4a2d36e7c635]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.434321 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bf570855b204b96acd070587c434463 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:20.434412 27354 tablet_service.cc:1468] Processing CreateTablet for tablet 3d875d29e66d48ed872e38d20f30d9c0 (DEFAULT_TABLE table=default.Table_77c890250aaf48f4b658f0f126941bc7 [id=f4642be57818412a9607dc534d31f23f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.434682 27348 tablet_service.cc:1468] Processing CreateTablet for tablet ce3be9d44147420d953e716a347352c9 (DEFAULT_TABLE table=default.Table_191beed070214f6eac98ee16e25e34f8 [id=98b7dc72c66e4cfb92bd24ae61c6605f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.435118 27358 tablet_service.cc:1468] Processing CreateTablet for tablet 8292ae4c5bdd469681f3a3b295b24349 (DEFAULT_TABLE table=default.Table_1310291579564dd18d0b47d1050cf905 [id=c232ef71f26b42779d5849f8bb49bbe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.435204 27490 tablet_service.cc:1468] Processing CreateTablet for tablet 931404d9b3844d908baaed3190d57b56 (DEFAULT_TABLE table=default.Table_dbaa47c36c6f43fab8828747862ebbaa [id=523a2be18abd4730951758e0b8b91a2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.435534 27495 tablet_service.cc:1468] Processing CreateTablet for tablet 197ccb3d4bc54e8f9202fc6e7ae1426c (DEFAULT_TABLE table=default.Table_faaa45179ec149bcb4e47fb312ee6549 [id=d87bd84debbd47efbf7173b191095fe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.435696 27364 tablet_service.cc:1468] Processing CreateTablet for tablet 91567f10d90142acb7798757262218f0 (DEFAULT_TABLE table=default.Table_aaea28edb28e4ee991b380a66d0297d3 [id=9e129548408242a49193a98a1a340b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.435685 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ef539157e90548248025f3570f2bab2d" } new_table_name: "default.Table_78e7a5b76ebc469396d8a3d670ab63ee" modify_external_catalogs: true
I20250903 20:05:20.435932 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 197ccb3d4bc54e8f9202fc6e7ae1426c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.436264 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet 259bc73ca9df47e9874692725a60bb1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:20.436731 27357 tablet_service.cc:1468] Processing CreateTablet for tablet 421dc8af40c94a89bc372e8779872a4e (DEFAULT_TABLE table=default.Table_76100d98b0774693ab71b084505d007a [id=f78623e5b0444dac9c633558aba62f5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.437021 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d93258c032524b71997389b81846a5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.437290 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8292ae4c5bdd469681f3a3b295b24349. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.437520 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91567f10d90142acb7798757262218f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.437717 28242 raft_consensus.cc:2802] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:20.437821 28242 raft_consensus.cc:491] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:20.438059 28242 raft_consensus.cc:3058] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:20.442494 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6d019ca2fcd41309e74cdd3a1e20daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.405108 27244 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9129a3e773a4fecb28ceee27778a294" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:20.445246 27244 raft_consensus.cc:2466] T c9129a3e773a4fecb28ceee27778a294 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:20.398478 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 364) took 4676 ms. Trace:
W20250903 20:05:20.400146 27183 leader_election.cc:343] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:20.445612 27183 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 45476us
I20250903 20:05:20.445431 27685 rpcz_store.cc:276] 0903 20:05:15.722113 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.722165 (+ 52us) service_pool.cc:225] Handling call
0903 20:05:20.397942 (+4675777us) ts_tablet_manager.cc:1145] Deleting tablet cb4d99139516405bb65c7480ddf259f1
0903 20:05:20.397956 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:20.398398 (+ 442us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4675325}
I20250903 20:05:20.447113 27685 tablet_replica.cc:331] T 8b3121fce1c44d16a0b083313dc67b7a P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:20.447623 27685 raft_consensus.cc:2241] T 8b3121fce1c44d16a0b083313dc67b7a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
W20250903 20:05:20.447365 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet cb4d99139516405bb65c7480ddf259f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb4d99139516405bb65c7480ddf259f1 already in progress: creating tablet
I20250903 20:05:20.448419 27685 raft_consensus.cc:2270] T 8b3121fce1c44d16a0b083313dc67b7a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:20.435343 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3453a2165fe44a3d93deee7b39853d3e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.448809 27224 tablet_service.cc:1515] Processing DeleteTablet for tablet 847d0a0a3d2b45938cec9058c77f6231 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:20.449663 27685 ts_tablet_manager.cc:1905] T 8b3121fce1c44d16a0b083313dc67b7a P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:20.451551 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0b5d0984523e4daba55e75727cdf4487" } new_table_name: "default.Table_8bbb5561118c4a68af214f8d319d2caf" modify_external_catalogs: true
I20250903 20:05:20.451155 27695 tablet_bootstrap.cc:492] T ac1b0dc95fca49b4af6697c0a70e9730 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:20.451671 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_6bb2706114f147fa85c1389ab0b3e752"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:20.452126 27695 ts_tablet_manager.cc:1397] T ac1b0dc95fca49b4af6697c0a70e9730 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.138s user 0.002s sys 0.012s
W20250903 20:05:20.452313 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6bb2706114f147fa85c1389ab0b3e752 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.453148 27695 raft_consensus.cc:357] T ac1b0dc95fca49b4af6697c0a70e9730 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:20.453626 27695 raft_consensus.cc:383] T ac1b0dc95fca49b4af6697c0a70e9730 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:20.453778 27695 raft_consensus.cc:738] T ac1b0dc95fca49b4af6697c0a70e9730 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:20.454104 27695 consensus_queue.cc:260] T ac1b0dc95fca49b4af6697c0a70e9730 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:20.455685 27490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 931404d9b3844d908baaed3190d57b56. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.463227 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c5abd2620f3f45628ce2531cbcb81d9d" } modify_external_catalogs: true
I20250903 20:05:20.463552 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c5abd2620f3f45628ce2531cbcb81d9d" } modify_external_catalogs: true
I20250903 20:05:20.469539 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 153b9fef87f24b96a97d33a7c14f662e (DEFAULT_TABLE table=default.Table_c749638286ff4a0db8b80342622dca67 [id=2c0e64114adc44dab5c2271bde165bc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.470997 27352 tablet_service.cc:1468] Processing CreateTablet for tablet c0ea33c5c2444c57a86086d49a85db2b (DEFAULT_TABLE table=default.Table_5d5e2d80c015418db7956702c08cac42 [id=ea9e6eb03085487a910dc84894196d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.472990 27355 tablet_service.cc:1468] Processing CreateTablet for tablet cc50edd0891f4e48bc0136c12cde45da (DEFAULT_TABLE table=default.Table_09e3fcf1b8ba4f75be49736c7c4b5f30 [id=d317d133362c402db605e373d094f49d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.475129 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce3be9d44147420d953e716a347352c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.477660 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 0906165a50b047c8a776b30832149964 (DEFAULT_TABLE table=default.Table_20274d6f2af744d485064ee2ce5d22d6 [id=9c8ccd24b88048aea95c867aae5fcb23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.474177 27505 tablet_service.cc:1515] Processing DeleteTablet for tablet 847d0a0a3d2b45938cec9058c77f6231 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:20.474726 27692 tablet_bootstrap.cc:654] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:20.473542 27684 ts_tablet_manager.cc:1918] T 57a1167457b84ea8a493be232b1d088a P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:20.479743 27354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d875d29e66d48ed872e38d20f30d9c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.479976 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 153b9fef87f24b96a97d33a7c14f662e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.480727 27349 tablet_service.cc:1468] Processing CreateTablet for tablet 07439db2c3b8470eb2b797cc68376aa7 (DEFAULT_TABLE table=default.Table_b0a0a2dfe31e4961871f334372541d05 [id=30918e7a5f1d43aab5eaddef7c485cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.481045 27366 tablet_service.cc:1468] Processing CreateTablet for tablet f64ab5cf99af4608906441df4fb6e86c (DEFAULT_TABLE table=default.Table_0eded5731b1944de87e65ac1b3298371 [id=d131083310924d1b9318affd49a69afe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.484493 27363 tablet_service.cc:1468] Processing CreateTablet for tablet 197ccb3d4bc54e8f9202fc6e7ae1426c (DEFAULT_TABLE table=default.Table_faaa45179ec149bcb4e47fb312ee6549 [id=d87bd84debbd47efbf7173b191095fe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.485651 27356 tablet_service.cc:1468] Processing CreateTablet for tablet 931404d9b3844d908baaed3190d57b56 (DEFAULT_TABLE table=default.Table_dbaa47c36c6f43fab8828747862ebbaa [id=523a2be18abd4730951758e0b8b91a2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.487597 27351 tablet_service.cc:1515] Processing DeleteTablet for tablet 57a1167457b84ea8a493be232b1d088a with delete_type TABLET_DATA_DELETED (replaced by tablet f326ec776ba446be8d5dda5c1dc5b6ec) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:20.488222 27687 ts_tablet_manager.cc:1918] T eced730bdbc14a8c9f6b7b7e53435fd0 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250903 20:05:20.492802 27184 leader_election.cc:343] T 07565ea2b5f34dcfb407ff3bfde17243 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:20.492964 27184 leader_election.cc:304] T 07565ea2b5f34dcfb407ff3bfde17243 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:20.493461 27184 leader_election.cc:343] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:20.493573 27184 leader_election.cc:304] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:20.493853 28264 raft_consensus.cc:2747] T 07565ea2b5f34dcfb407ff3bfde17243 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:20.494092 28245 raft_consensus.cc:2747] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:20.494880 27693 tablet_bootstrap.cc:654] T e9eb24dd5a6243c49a3389320a3f78a0 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:20.495555 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 847d0a0a3d2b45938cec9058c77f6231 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:20.491098 28242 raft_consensus.cc:513] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:20.496945 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0ea33c5c2444c57a86086d49a85db2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.497200 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet f2629f46704e4a0fa7922533928ca2ed
I20250903 20:05:20.497852 27064 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b60456f0147b407584bb3aa4321b1fe0" } modify_external_catalogs: true
I20250903 20:05:20.498049 27064 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b60456f0147b407584bb3aa4321b1fe0" } modify_external_catalogs: true
I20250903 20:05:20.500483 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 421dc8af40c94a89bc372e8779872a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.500828 27355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc50edd0891f4e48bc0136c12cde45da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.504364 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 4e9a6a6e05464b7b97ae624f51402cf0
I20250903 20:05:20.505851 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 931404d9b3844d908baaed3190d57b56. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:20.506114 27059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a253e3399622409e92299069b19d3698 consensus state has no leader
I20250903 20:05:20.506490 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f64ab5cf99af4608906441df4fb6e86c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.508121 27496 tablet_service.cc:1515] Processing DeleteTablet for tablet f2629f46704e4a0fa7922533928ca2ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:20.508317 28242 leader_election.cc:290] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:20.505824 27687 log.cc:1199] T eced730bdbc14a8c9f6b7b7e53435fd0 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/eced730bdbc14a8c9f6b7b7e53435fd0
I20250903 20:05:20.510349 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07439db2c3b8470eb2b797cc68376aa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.510510 27224 tablet_service.cc:1515] Processing DeleteTablet for tablet f2629f46704e4a0fa7922533928ca2ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:20.510646 27687 ts_tablet_manager.cc:1939] T eced730bdbc14a8c9f6b7b7e53435fd0 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:20.511680 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 495) took 2102 ms. Trace:
I20250903 20:05:20.511811 27687 rpcz_store.cc:276] 0903 20:05:18.409496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.409557 (+ 61us) service_pool.cc:225] Handling call
0903 20:05:20.005427 (+1595870us) ts_tablet_manager.cc:1145] Deleting tablet eced730bdbc14a8c9f6b7b7e53435fd0
0903 20:05:20.005436 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:20.408236 (+402800us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:20.488103 (+ 79867us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:20.511581 (+ 23478us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1595507}
I20250903 20:05:20.512406 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 496) took 2078 ms. Trace:
I20250903 20:05:20.512521 27687 rpcz_store.cc:276] 0903 20:05:18.434237 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.434312 (+ 75us) service_pool.cc:225] Handling call
0903 20:05:20.512121 (+2077809us) ts_tablet_manager.cc:1145] Deleting tablet 63a1f26fac3b4b939ebb79c6a07d8f8e
0903 20:05:20.512138 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:20.512339 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2077554}
I20250903 20:05:20.512964 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 497) took 2008 ms. Trace:
I20250903 20:05:20.513000 27684 log.cc:1199] T 57a1167457b84ea8a493be232b1d088a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/57a1167457b84ea8a493be232b1d088a
I20250903 20:05:20.513095 27687 rpcz_store.cc:276] 0903 20:05:18.504061 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.504126 (+ 65us) service_pool.cc:225] Handling call
0903 20:05:20.512773 (+2008647us) ts_tablet_manager.cc:1145] Deleting tablet f0efbb7eefb84ab6bee5b9625b91e298
0903 20:05:20.512783 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:20.512912 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2008394}
I20250903 20:05:20.513409 27684 ts_tablet_manager.cc:1939] T 57a1167457b84ea8a493be232b1d088a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:20.513535 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 498) took 1974 ms. Trace:
I20250903 20:05:20.513634 27687 rpcz_store.cc:276] 0903 20:05:18.539388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.539647 (+ 259us) service_pool.cc:225] Handling call
0903 20:05:20.513367 (+1973720us) ts_tablet_manager.cc:1145] Deleting tablet f4dec89fc6bb436ba6542c267eaf9c15
0903 20:05:20.513377 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:20.513462 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1973399}
I20250903 20:05:20.513995 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 499) took 1974 ms. Trace:
I20250903 20:05:20.514390 27212 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e9a6a6e05464b7b97ae624f51402cf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:20.514744 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 197ccb3d4bc54e8f9202fc6e7ae1426c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.515031 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 57a1167457b84ea8a493be232b1d088a (table default.Table_b2a82a2c310140caaeac5d0dcb4bb61a [id=e2548ad469e54a41af7c0181357d2467]) successfully deleted
I20250903 20:05:20.516872 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0906165a50b047c8a776b30832149964. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.518482 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e9a6a6e05464b7b97ae624f51402cf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:20.518934 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fc634116a78f45dd833034ddda90e9bf consensus state has no leader
I20250903 20:05:20.523451 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet eced730bdbc14a8c9f6b7b7e53435fd0 (table default.Table_b520cd881f58424dadf3fa4a013812ee [id=2c179236954b41cfa9242e16946ef9bb]) successfully deleted
W20250903 20:05:20.523762 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet f0efbb7eefb84ab6bee5b9625b91e298 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0efbb7eefb84ab6bee5b9625b91e298 already in progress: creating tablet
W20250903 20:05:20.523931 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet f4dec89fc6bb436ba6542c267eaf9c15 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4dec89fc6bb436ba6542c267eaf9c15 already in progress: creating tablet
I20250903 20:05:20.514096 27687 rpcz_store.cc:276] 0903 20:05:18.539567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.540016 (+ 449us) service_pool.cc:225] Handling call
0903 20:05:20.513832 (+1973816us) ts_tablet_manager.cc:1145] Deleting tablet a38f65e992884635aae9942681c0dc4f
0903 20:05:20.513839 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:20.513936 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1973622}
I20250903 20:05:20.533643 27350 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e9a6a6e05464b7b97ae624f51402cf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:20.534850 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet a38f65e992884635aae9942681c0dc4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a38f65e992884635aae9942681c0dc4f already in progress: creating tablet
I20250903 20:05:20.536438 28249 raft_consensus.cc:491] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:20.536617 28249 raft_consensus.cc:513] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:20.537365 28249 leader_election.cc:290] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:20.538091 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de8ac9f984be46ec8f021928de12a090" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:20.538612 27183 leader_election.cc:343] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:20.538983 28299 raft_consensus.cc:491] T c9129a3e773a4fecb28ceee27778a294 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:20.539857 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d8651e0216bf42e08446c9cdc2ed5b0a" } new_table_name: "default.Table_30f001eba314494d93e8ecac2f0bc3dc" modify_external_catalogs: true
I20250903 20:05:20.541460 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1b5f4328e47f4393a36d5ed02c2614c4" } new_table_name: "default.Table_1afb82628b3049949cc9dfdd46071a9e" modify_external_catalogs: true
W20250903 20:05:20.542579 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 4789bd72aa454b639df76bd5ad379850 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4789bd72aa454b639df76bd5ad379850 already in progress: creating tablet
I20250903 20:05:20.542932 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_a7492190504c4a388eea5ab3ebfe7b83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:20.543520 27695 ts_tablet_manager.cc:1428] T ac1b0dc95fca49b4af6697c0a70e9730 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.091s user 0.002s sys 0.000s
W20250903 20:05:20.543619 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7492190504c4a388eea5ab3ebfe7b83 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.544019 28299 raft_consensus.cc:513] T c9129a3e773a4fecb28ceee27778a294 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:20.547187 27364 tablet_service.cc:1468] Processing CreateTablet for tablet c532bb70298547b58a5b88b95d0934a9 (DEFAULT_TABLE table=default.Table_217752fec9f1486c99d277d26ede63ba [id=20fca17dfad1467ebcffefacf4068cbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.548062 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de8ac9f984be46ec8f021928de12a090" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:20.548532 27184 leader_election.cc:343] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:20.548666 27184 leader_election.cc:304] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:20.549064 28249 raft_consensus.cc:2747] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:20.550535 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 5f2eefeebbfc47249e2f72b7c52e905d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f2eefeebbfc47249e2f72b7c52e905d already in progress: creating tablet
I20250903 20:05:20.553296 27695 tablet_bootstrap.cc:492] T 4be9bdc390de4a74a396003c9c3c2b71 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
W20250903 20:05:20.540292 27462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 113487us
W20250903 20:05:20.554004 27462 leader_election.cc:343] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:20.555133 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 64d53f2071b0449ca61caa6c264a3641 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64d53f2071b0449ca61caa6c264a3641 already in progress: creating tablet
I20250903 20:05:20.556056 28299 leader_election.cc:290] T c9129a3e773a4fecb28ceee27778a294 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:20.562600 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9129a3e773a4fecb28ceee27778a294" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:20.562717 27244 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9eb24dd5a6243c49a3389320a3f78a0" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:20.562940 27244 raft_consensus.cc:3058] T e9eb24dd5a6243c49a3389320a3f78a0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:20.556430 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9129a3e773a4fecb28ceee27778a294" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:20.564554 27522 raft_consensus.cc:2466] T c9129a3e773a4fecb28ceee27778a294 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:20.557309 27684 tablet_replica.cc:331] T 4c543ac4df33429b9787c6d32bf1ebeb P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:20.565168 27684 raft_consensus.cc:2241] T 4c543ac4df33429b9787c6d32bf1ebeb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:20.565795 27244 raft_consensus.cc:2466] T e9eb24dd5a6243c49a3389320a3f78a0 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:20.562896 27248 raft_consensus.cc:2466] T c9129a3e773a4fecb28ceee27778a294 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:20.567095 27695 tablet_bootstrap.cc:654] T 4be9bdc390de4a74a396003c9c3c2b71 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:20.568588 27059 master_service.cc:948] ReplaceTablet: received request to replace tablet a3103e1595ec4c38ba65dca26fe1167c from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:20.569525 27462 leader_election.cc:304] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:20.569892 27684 raft_consensus.cc:2270] T 4c543ac4df33429b9787c6d32bf1ebeb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:20.574618 28242 raft_consensus.cc:2802] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:20.574883 28242 raft_consensus.cc:695] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:20.575237 28242 consensus_queue.cc:237] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:20.582453 28184 webserver.cc:480] Webserver started at http://127.26.65.125:41613/ using document root <none> and password file <none>
I20250903 20:05:20.583029 27692 tablet_bootstrap.cc:492] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:20.583266 27692 ts_tablet_manager.cc:1397] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.213s user 0.007s sys 0.011s
I20250903 20:05:20.583850 28184 fs_manager.cc:362] Metadata directory not provided
I20250903 20:05:20.584087 28184 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 20:05:20.584304 27692 raft_consensus.cc:357] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:20.584503 27692 raft_consensus.cc:383] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:20.584589 27692 raft_consensus.cc:738] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:20.584820 27692 consensus_queue.cc:260] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:20.591027 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 200c06f296b94eab94f35f312c5a443b
I20250903 20:05:20.591230 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet cac38daf002c4ae698d72903a277a587
I20250903 20:05:20.592304 27684 ts_tablet_manager.cc:1905] T 4c543ac4df33429b9787c6d32bf1ebeb P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:20.592458 27210 tablet_service.cc:1515] Processing DeleteTablet for tablet cac38daf002c4ae698d72903a277a587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:20.594530 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5fa7cc0c03234223af382982ca737e27 consensus state has no leader
W20250903 20:05:20.594988 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e6d019ca2fcd41309e74cdd3a1e20daf consensus state has no leader
I20250903 20:05:20.596161 27210 tablet_service.cc:1468] Processing CreateTablet for tablet 931404d9b3844d908baaed3190d57b56 (DEFAULT_TABLE table=default.Table_dbaa47c36c6f43fab8828747862ebbaa [id=523a2be18abd4730951758e0b8b91a2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.596661 27210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 931404d9b3844d908baaed3190d57b56. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.600898 27490 tablet_service.cc:1515] Processing DeleteTablet for tablet cac38daf002c4ae698d72903a277a587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:20.602427 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b88ff2c2e61049d4a11af5676d28327a" } new_table_name: "default.Table_f6ddd620f29a4333a8f71f99316e1a8d" modify_external_catalogs: true
I20250903 20:05:20.604091 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_bae568b17ab54534914716b3111a041d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:20.604664 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet 200c06f296b94eab94f35f312c5a443b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:20.604750 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bae568b17ab54534914716b3111a041d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.606384 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_fb9f90cfd77144cc81d36b52a6fc032c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:20.606930 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb9f90cfd77144cc81d36b52a6fc032c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.608577 27210 tablet_service.cc:1468] Processing CreateTablet for tablet aa31f2ddb9f84a8093aa1c6f64b278c9 (DEFAULT_TABLE table=default.Table_f42f6303480447ddac82d66378c1ce36 [id=30381efd3bc44a3fb1eddb710355b429]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.608990 27210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa31f2ddb9f84a8093aa1c6f64b278c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.609979 27684 ts_tablet_manager.cc:1918] T 4c543ac4df33429b9787c6d32bf1ebeb P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:20.610116 27684 log.cc:1199] T 4c543ac4df33429b9787c6d32bf1ebeb P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/4c543ac4df33429b9787c6d32bf1ebeb
I20250903 20:05:20.611619 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_a534288cb6194102ade0cbb0b87630ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:20.612200 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a534288cb6194102ade0cbb0b87630ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.612624 27213 tablet_service.cc:1468] Processing CreateTablet for tablet f204f3ead373415daf7c3e765dd8f59e (DEFAULT_TABLE table=default.Table_c891a3989f9a42dbb10ac8788775a83a [id=f46ac8414a904fc2b8eb2a8d1f6bfb5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.613049 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f204f3ead373415daf7c3e765dd8f59e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.614768 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet 200c06f296b94eab94f35f312c5a443b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:20.620056 27051 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 249362a0b8304719863bcb82c47273b3 (table default.Table_1dd1b4a75c914e6a88058266d2a7e98a [id=565dc96c38de4efeb4328c01e030eb8c], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 249362a0b8304719863bcb82c47273b3 consensus state has no leader
I20250903 20:05:20.621429 27220 tablet_service.cc:1468] Processing CreateTablet for tablet 32164b4e88b94e1fbca2e714e6bddfc4 (DEFAULT_TABLE table=default.Table_de42081b245243918dff0bc0301b027a [id=e8328693d93044aa97af4294a8fac8dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.622043 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32164b4e88b94e1fbca2e714e6bddfc4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:20.628562 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bf459db3327640f88574fca7b5f2ca98 consensus state has no leader
I20250903 20:05:20.629372 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0d7367affcaf4dee8b0a8a1f524a7035" } modify_external_catalogs: true
I20250903 20:05:20.629592 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0d7367affcaf4dee8b0a8a1f524a7035" } modify_external_catalogs: true
I20250903 20:05:20.622489 27497 tablet_service.cc:1468] Processing CreateTablet for tablet aa31f2ddb9f84a8093aa1c6f64b278c9 (DEFAULT_TABLE table=default.Table_f42f6303480447ddac82d66378c1ce36 [id=30381efd3bc44a3fb1eddb710355b429]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.636158 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa31f2ddb9f84a8093aa1c6f64b278c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.622740 27500 tablet_service.cc:1468] Processing CreateTablet for tablet f204f3ead373415daf7c3e765dd8f59e (DEFAULT_TABLE table=default.Table_c891a3989f9a42dbb10ac8788775a83a [id=f46ac8414a904fc2b8eb2a8d1f6bfb5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.638782 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet bf2545037c214bb1b5060575e3cdf81a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:20.623056 27490 tablet_service.cc:1468] Processing CreateTablet for tablet 32164b4e88b94e1fbca2e714e6bddfc4 (DEFAULT_TABLE table=default.Table_de42081b245243918dff0bc0301b027a [id=e8328693d93044aa97af4294a8fac8dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.626735 27684 ts_tablet_manager.cc:1939] T 4c543ac4df33429b9787c6d32bf1ebeb P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
W20250903 20:05:20.639741 27462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 68693.1us
I20250903 20:05:20.641096 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 4c543ac4df33429b9787c6d32bf1ebeb (table default.Table_26a46b6cf89f459b99526bc640729603 [id=ec0522912bda4887b0c40e98ac12768a]) successfully deleted
I20250903 20:05:20.646082 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c532bb70298547b58a5b88b95d0934a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.646190 27462 leader_election.cc:304] T c9129a3e773a4fecb28ceee27778a294 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:20.639962 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f204f3ead373415daf7c3e765dd8f59e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.646674 27363 tablet_service.cc:1468] Processing CreateTablet for tablet 32164b4e88b94e1fbca2e714e6bddfc4 (DEFAULT_TABLE table=default.Table_de42081b245243918dff0bc0301b027a [id=e8328693d93044aa97af4294a8fac8dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.647315 27365 tablet_service.cc:1468] Processing CreateTablet for tablet aa31f2ddb9f84a8093aa1c6f64b278c9 (DEFAULT_TABLE table=default.Table_f42f6303480447ddac82d66378c1ce36 [id=30381efd3bc44a3fb1eddb710355b429]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.649258 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet f2629f46704e4a0fa7922533928ca2ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:20.649856 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_2dcb231e5186428fa09678c7968ef431"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:20.650617 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2dcb231e5186428fa09678c7968ef431 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.652456 27687 tablet_replica.cc:331] T 264127511a234e9fbf409f3aa2860ccc P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
W20250903 20:05:20.652047 26890 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 99252.5us
I20250903 20:05:20.652659 27490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32164b4e88b94e1fbca2e714e6bddfc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.641173 27684 tablet_replica.cc:331] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:20.653080 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9eb24dd5a6243c49a3389320a3f78a0" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
W20250903 20:05:20.653124 27320 leader_election.cc:343] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: deb84fc3465443d99c11097e53f5f575
I20250903 20:05:20.653505 27354 tablet_service.cc:1468] Processing CreateTablet for tablet f204f3ead373415daf7c3e765dd8f59e (DEFAULT_TABLE table=default.Table_c891a3989f9a42dbb10ac8788775a83a [id=f46ac8414a904fc2b8eb2a8d1f6bfb5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.653559 27684 raft_consensus.cc:2241] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
W20250903 20:05:20.654078 27320 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 85051us
I20250903 20:05:20.654090 27358 tablet_service.cc:1515] Processing DeleteTablet for tablet cac38daf002c4ae698d72903a277a587 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:20.654078 27684 pending_rounds.cc:70] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f: Trying to abort 1 pending ops.
I20250903 20:05:20.654371 27687 raft_consensus.cc:2241] T 264127511a234e9fbf409f3aa2860ccc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:20.654409 27684 pending_rounds.cc:77] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384954679836672 op_type: NO_OP noop_request { }
I20250903 20:05:20.655560 28301 raft_consensus.cc:2802] T c9129a3e773a4fecb28ceee27778a294 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:20.655750 28301 raft_consensus.cc:491] T c9129a3e773a4fecb28ceee27778a294 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:20.656009 28301 raft_consensus.cc:3058] T c9129a3e773a4fecb28ceee27778a294 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:20.655052 27684 raft_consensus.cc:2887] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:20.656896 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa31f2ddb9f84a8093aa1c6f64b278c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.657357 27687 raft_consensus.cc:2270] T 264127511a234e9fbf409f3aa2860ccc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:20.658118 27687 ts_tablet_manager.cc:1905] T 264127511a234e9fbf409f3aa2860ccc P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:20.658977 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32164b4e88b94e1fbca2e714e6bddfc4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:20.662187 27320 leader_election.cc:343] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 3f78a951bb024cf290116b0fe438ddb4
I20250903 20:05:20.662375 27354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f204f3ead373415daf7c3e765dd8f59e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.662505 27320 leader_election.cc:304] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:20.663414 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9129a3e773a4fecb28ceee27778a294" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:20.664427 27381 raft_consensus.cc:2466] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:20.665853 28298 raft_consensus.cc:2802] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:20.666287 28298 raft_consensus.cc:491] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:20.666846 28298 raft_consensus.cc:3058] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:20.665387 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 200c06f296b94eab94f35f312c5a443b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:20.670706 27693 tablet_bootstrap.cc:492] T e9eb24dd5a6243c49a3389320a3f78a0 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:20.671389 28234 raft_consensus.cc:695] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:20.672173 28234 consensus_queue.cc:237] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:20.678244 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 615f42e95bb24c8480f73d371fd63b9c
I20250903 20:05:20.656716 27684 raft_consensus.cc:2270] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:20.671587 28305 consensus_queue.cc:1162] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20250903 20:05:20.686708 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet 615f42e95bb24c8480f73d371fd63b9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:20.687168 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet 615f42e95bb24c8480f73d371fd63b9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:20.688251 27693 ts_tablet_manager.cc:1397] T e9eb24dd5a6243c49a3389320a3f78a0 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.240s user 0.010s sys 0.012s
I20250903 20:05:20.689370 27693 raft_consensus.cc:357] T e9eb24dd5a6243c49a3389320a3f78a0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:20.689584 28237 raft_consensus.cc:491] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:20.690398 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 615f42e95bb24c8480f73d371fd63b9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:20.694079 28237 raft_consensus.cc:513] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:20.696195 28237 leader_election.cc:290] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:20.696290 27687 ts_tablet_manager.cc:1918] T 264127511a234e9fbf409f3aa2860ccc P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:20.696596 27693 raft_consensus.cc:383] T e9eb24dd5a6243c49a3389320a3f78a0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:20.697860 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "deb84fc3465443d99c11097e53f5f575" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:20.698154 27249 raft_consensus.cc:2466] T deb84fc3465443d99c11097e53f5f575 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:20.699968 27687 log.cc:1199] T 264127511a234e9fbf409f3aa2860ccc P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/264127511a234e9fbf409f3aa2860ccc
I20250903 20:05:20.700228 28184 fs_manager.cc:714] Time spent opening directory manager: real 0.112s user 0.007s sys 0.001s
I20250903 20:05:20.704480 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_791801575c0e4c98b42111d7a3254236"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:20.705122 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_791801575c0e4c98b42111d7a3254236 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.705543 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e79d289ca0384e109873b8d5f2c78905" } modify_external_catalogs: true
I20250903 20:05:20.705698 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e79d289ca0384e109873b8d5f2c78905" } modify_external_catalogs: true
I20250903 20:05:20.706528 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b2824f7f5f324e6ebcacb1b0645de8ec" } modify_external_catalogs: true
I20250903 20:05:20.706713 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b2824f7f5f324e6ebcacb1b0645de8ec" } modify_external_catalogs: true
I20250903 20:05:20.710440 27693 raft_consensus.cc:738] T e9eb24dd5a6243c49a3389320a3f78a0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:20.710876 27693 consensus_queue.cc:260] T e9eb24dd5a6243c49a3389320a3f78a0 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:20.711234 27061 master_service.cc:948] ReplaceTablet: received request to replace tablet 1bf69315f07a470bb2b897ad7c590729 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:20.712334 27693 ts_tablet_manager.cc:1428] T e9eb24dd5a6243c49a3389320a3f78a0 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20250903 20:05:20.713932 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f78a951bb024cf290116b0fe438ddb4" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:20.714200 27249 raft_consensus.cc:2466] T 3f78a951bb024cf290116b0fe438ddb4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:20.716549 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1fef9400227449ed892daff588999caa" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:20.716768 27249 raft_consensus.cc:2466] T 1fef9400227449ed892daff588999caa P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:20.718683 27695 tablet_bootstrap.cc:492] T 4be9bdc390de4a74a396003c9c3c2b71 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:20.718716 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet c9129a3e773a4fecb28ceee27778a294
I20250903 20:05:20.718894 27695 ts_tablet_manager.cc:1397] T 4be9bdc390de4a74a396003c9c3c2b71 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.166s user 0.004s sys 0.000s
I20250903 20:05:20.719513 27363 tablet_service.cc:1515] Processing DeleteTablet for tablet c9129a3e773a4fecb28ceee27778a294 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:20.719744 27695 raft_consensus.cc:357] T 4be9bdc390de4a74a396003c9c3c2b71 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:20.719929 27695 raft_consensus.cc:383] T 4be9bdc390de4a74a396003c9c3c2b71 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:20.720160 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet c9129a3e773a4fecb28ceee27778a294 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:20.720207 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_cd761ea7907f44a6bcf538648450b701" } new_table_name: "default.Table_7b1039b36f6940dd95220a8f25849711" modify_external_catalogs: true
I20250903 20:05:20.720465 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet ebf0a9aaaea1408c8c9caf28f00dbff7
I20250903 20:05:20.721836 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet ebf0a9aaaea1408c8c9caf28f00dbff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:20.722194 27685 ts_tablet_manager.cc:1918] T 8b3121fce1c44d16a0b083313dc67b7a P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:20.722445 27685 log.cc:1199] T 8b3121fce1c44d16a0b083313dc67b7a P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/8b3121fce1c44d16a0b083313dc67b7a
I20250903 20:05:20.722893 27685 ts_tablet_manager.cc:1939] T 8b3121fce1c44d16a0b083313dc67b7a P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:20.723410 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_4bb1596fb7204c0894e7a5d694b16943"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:20.723882 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 365) took 4975 ms. Trace:
W20250903 20:05:20.724056 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4bb1596fb7204c0894e7a5d694b16943 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.724655 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_affbe84913504de69e8d84b2e2d14cdc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:20.725318 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c4a34f406b12427ea406d72fc1be9db9" } new_table_name: "default.Table_ed096bbc1cae4d4bb851ef7a59affc78" modify_external_catalogs: true
W20250903 20:05:20.727990 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_affbe84913504de69e8d84b2e2d14cdc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.730749 27066 master_service.cc:948] ReplaceTablet: received request to replace tablet 39c2ebe4ada5426590ec3769c8662644 from {username='slave'} at 127.0.0.1:45936
W20250903 20:05:20.734951 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 55b2cdee20264ae6b64ebe3607fe1ca6 consensus state has no leader
I20250903 20:05:20.735905 28312 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20250903 20:05:20.736660 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9a71290c27ac4a22bd9aa8e14e5834f5 consensus state has no leader
I20250903 20:05:20.742239 27685 rpcz_store.cc:276] 0903 20:05:15.748123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.748178 (+ 55us) service_pool.cc:225] Handling call
0903 20:05:20.447012 (+4698834us) ts_tablet_manager.cc:1145] Deleting tablet 8b3121fce1c44d16a0b083313dc67b7a
0903 20:05:20.447041 (+ 29us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:20.572759 (+125718us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:20.722065 (+149306us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:20.723827 (+ 1762us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4697767}
I20250903 20:05:20.744702 28184 fs_manager.cc:730] Time spent opening block manager: real 0.040s user 0.004s sys 0.003s
I20250903 20:05:20.745338 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 8b3121fce1c44d16a0b083313dc67b7a (table default.Table_41538df113834b45ae2213b105fb9706 [id=158a2fc69fce4c54917277cc90439126]) successfully deleted
I20250903 20:05:20.738797 27695 raft_consensus.cc:738] T 4be9bdc390de4a74a396003c9c3c2b71 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:20.746009 27695 consensus_queue.cc:260] T 4be9bdc390de4a74a396003c9c3c2b71 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:20.746794 27695 ts_tablet_manager.cc:1428] T 4be9bdc390de4a74a396003c9c3c2b71 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20250903 20:05:20.747195 27695 tablet_bootstrap.cc:492] T 912eee5b48ef467bbd6bbe34f8aa390c P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:20.749838 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 368) took 4849 ms. Trace:
I20250903 20:05:20.750012 27685 rpcz_store.cc:276] 0903 20:05:15.899935 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:15.900701 (+ 766us) service_pool.cc:225] Handling call
0903 20:05:20.749636 (+4848935us) ts_tablet_manager.cc:1145] Deleting tablet a4f3230f50ec4c95b528fead8c191224
0903 20:05:20.749646 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:20.749776 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4825521}
W20250903 20:05:20.750764 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet a4f3230f50ec4c95b528fead8c191224 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4f3230f50ec4c95b528fead8c191224 already in progress: creating tablet
I20250903 20:05:20.752211 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet 4be9bdc390de4a74a396003c9c3c2b71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:20.752667 27685 tablet_replica.cc:331] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:20.752940 27685 raft_consensus.cc:2241] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:20.753479 27685 pending_rounds.cc:70] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a: Trying to abort 1 pending ops.
I20250903 20:05:20.756017 27685 pending_rounds.cc:77] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7196384943975100416 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "68cff0e416a54a5eb37071f5a12f07ab" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 3 new_table_name: "default.Table_1e3f794ae5ec4ad6a9b4351b4bda7b15" dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" new_extra_config { } }
I20250903 20:05:20.753961 27220 tablet_service.cc:1468] Processing CreateTablet for tablet 5253d82a41494ce59bb96928179dfdd9 (DEFAULT_TABLE table=default.Table_a7492190504c4a388eea5ab3ebfe7b83 [id=0b54dabacc024398b526e985536bd605]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.756750 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5253d82a41494ce59bb96928179dfdd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.756799 27065 master_service.cc:948] ReplaceTablet: received request to replace tablet 90baf83357494a0aae325908c118b471 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:20.759128 28298 raft_consensus.cc:513] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:20.759994 27693 tablet_bootstrap.cc:492] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:20.761101 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 1bf69315f07a470bb2b897ad7c590729
I20250903 20:05:20.761634 27061 catalog_manager.cc:6652] ReplaceTablet: tablet 1bf69315f07a470bb2b897ad7c590729 deleted and replaced by tablet 17514a2af39b402abbe8263d7c1abc7f
I20250903 20:05:20.762238 27210 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bf69315f07a470bb2b897ad7c590729 with delete_type TABLET_DATA_DELETED (replaced by tablet 17514a2af39b402abbe8263d7c1abc7f) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:20.746002 28184 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/data,/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/wal
uuid: "35dd69f3fa7a4fd8843b97f8a5d68296"
format_stamp: "Formatted at 2025-09-03 20:05:05 on dist-test-slave-fs2w"
I20250903 20:05:20.763377 28184 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 20:05:20.764681 28298 leader_election.cc:290] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:20.753067 27695 tablet_bootstrap.cc:654] T 912eee5b48ef467bbd6bbe34f8aa390c P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:20.765937 27687 ts_tablet_manager.cc:1939] T 264127511a234e9fbf409f3aa2860ccc P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
W20250903 20:05:20.766040 27685 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:56534: Aborted: Op aborted
I20250903 20:05:20.766500 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:56534 (request call id 425) took 3259 ms. Trace:
I20250903 20:05:20.767174 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 39c2ebe4ada5426590ec3769c8662644
I20250903 20:05:20.766624 27685 rpcz_store.cc:276] 0903 20:05:17.507453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.808037 (+300584us) service_pool.cc:225] Handling call
0903 20:05:20.766450 (+2958413us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:17.962419 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0903 20:05:17.962656 (+ 237us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0903 20:05:17.962667 (+ 11us) alter_schema_op.cc:66] Acquired schema lock
0903 20:05:17.962675 (+ 8us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0903 20:05:17.962722 (+ 47us) alter_schema_op.cc:128] START. Timestamp: 1756929917962671
0903 20:05:17.962725 (+ 3us) op_driver.cc:348] REPLICATION: starting
0903 20:05:18.044920 (+ 82195us) log.cc:844] Serialized 173 byte log entry
0903 20:05:18.080889 (+ 35969us) spinlock_profiling.cc:243] Waited 17 ms on lock 0x60c0003eec40. stack: 00007f589abe0dc6 00007f589abe0b80 00007f5899bc83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f58ae22dc35 00007f589ac26fde 00007f589ac2a41b 00007f589ac2a298 0000000000a774dc 00007f589ac062b8 00007f589e02a6da 00007f5896d4e71e
0903 20:05:20.765925 (+2685036us) op_driver.cc:464] REPLICATION: finished
0903 20:05:20.765960 (+ 35us) op_driver.cc:389] HandleFailure(Aborted: Op aborted)
0903 20:05:20.765968 (+ 8us) alter_schema_op.cc:171] AlterSchemaCommitCallback: op aborted
Metrics: {"child_traces":[["op",{"prepare.queue_time_us":119354,"prepare.run_cpu_time_us":1011,"prepare.run_wall_time_us":82654,"raft.queue_time_us":1098,"raft.run_cpu_time_us":22,"raft.run_wall_time_us":25,"replication_time_us":2803136,"spinlock_wait_cycles":38992256}]]}
I20250903 20:05:20.767134 27321 leader_election.cc:304] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:20.767722 27066 catalog_manager.cc:6652] ReplaceTablet: tablet 39c2ebe4ada5426590ec3769c8662644 deleted and replaced by tablet 2c772ab7c50c4994aacde5a05bb0f08e
I20250903 20:05:20.768158 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet 39c2ebe4ada5426590ec3769c8662644 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c772ab7c50c4994aacde5a05bb0f08e) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:20.768432 27363 tablet_service.cc:1515] Processing DeleteTablet for tablet ebf0a9aaaea1408c8c9caf28f00dbff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:20.769338 27220 tablet_service.cc:1468] Processing CreateTablet for tablet db3033fc47d946cc9f3e806d223d44cd (DEFAULT_TABLE table=default.Table_3453a2165fe44a3d93deee7b39853d3e [id=fd5ce7d7e1cf47eda896ceb491e931d6]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:20.771855 27321 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 54072.8us
I20250903 20:05:20.771955 27363 tablet_service.cc:1468] Processing CreateTablet for tablet db3033fc47d946cc9f3e806d223d44cd (DEFAULT_TABLE table=default.Table_3453a2165fe44a3d93deee7b39853d3e [id=fd5ce7d7e1cf47eda896ceb491e931d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.776373 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db3033fc47d946cc9f3e806d223d44cd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:20.780102 27462 leader_election.cc:343] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:20.780568 28234 raft_consensus.cc:2802] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:20.780768 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 500) took 2212 ms. Trace:
I20250903 20:05:20.781183 27684 ts_tablet_manager.cc:1905] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:20.782493 27692 ts_tablet_manager.cc:1428] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.130s user 0.001s sys 0.001s
I20250903 20:05:20.782608 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet c9129a3e773a4fecb28ceee27778a294 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:20.782800 27505 tablet_service.cc:1515] Processing DeleteTablet for tablet 39c2ebe4ada5426590ec3769c8662644 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c772ab7c50c4994aacde5a05bb0f08e) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:20.784248 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c543ac4df33429b9787c6d32bf1ebeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:20.784391 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_0c13f21b7b574e63abd419e44ea9b157"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:20.785076 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c13f21b7b574e63abd419e44ea9b157 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.786430 27213 tablet_service.cc:1468] Processing CreateTablet for tablet 65d07ebe1ec94baabc6ce011e761b4d5 (DEFAULT_TABLE table=default.Table_026e7093a60a495e95489df73d689967 [id=772809879a0048a29129ed25c5e178ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.786875 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65d07ebe1ec94baabc6ce011e761b4d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.786906 27224 tablet_service.cc:1468] Processing CreateTablet for tablet ec8598ef27cc470fa48938de8d5196fc (DEFAULT_TABLE table=default.Table_e019b2e0ac534620ad809b330da7038b [id=698ad7ffb5f14b31a8808c63e60edd2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.787297 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec8598ef27cc470fa48938de8d5196fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.786144 27210 tablet_service.cc:1468] Processing CreateTablet for tablet 88a6d4c9d9024e718650263608f04be3 (DEFAULT_TABLE table=default.Table_6bb2706114f147fa85c1389ab0b3e752 [id=ae602d681a3a4d1d9aecda22af00161b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.787820 27210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88a6d4c9d9024e718650263608f04be3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.786121 27212 tablet_service.cc:1468] Processing CreateTablet for tablet c059dda502ea4bb28aa8ad7c7baa60e7 (DEFAULT_TABLE table=default.Table_39c530a33b00492fbae7db9de5ea26c5 [id=0d2198edd9954eff86e3527fa9ff58b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.789654 27212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c059dda502ea4bb28aa8ad7c7baa60e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.790527 27227 tablet_service.cc:1468] Processing CreateTablet for tablet 13137053365a428e9d1e8d8852a53b6a (DEFAULT_TABLE table=default.Table_fb9f90cfd77144cc81d36b52a6fc032c [id=3fb365e8860542dfab5936bd1c0756ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.790969 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13137053365a428e9d1e8d8852a53b6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.791610 27695 tablet_bootstrap.cc:492] T 912eee5b48ef467bbd6bbe34f8aa390c P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:20.791796 27695 ts_tablet_manager.cc:1397] T 912eee5b48ef467bbd6bbe34f8aa390c P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.001s
I20250903 20:05:20.792693 27695 raft_consensus.cc:357] T 912eee5b48ef467bbd6bbe34f8aa390c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:20.792906 27695 raft_consensus.cc:383] T 912eee5b48ef467bbd6bbe34f8aa390c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:20.792997 27695 raft_consensus.cc:738] T 912eee5b48ef467bbd6bbe34f8aa390c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:20.793303 27695 consensus_queue.cc:260] T 912eee5b48ef467bbd6bbe34f8aa390c P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:20.793884 27695 ts_tablet_manager.cc:1428] T 912eee5b48ef467bbd6bbe34f8aa390c P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:20.794862 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bf69315f07a470bb2b897ad7c590729 with delete_type TABLET_DATA_DELETED (replaced by tablet 17514a2af39b402abbe8263d7c1abc7f) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:20.795312 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 88a6d4c9d9024e718650263608f04be3 (DEFAULT_TABLE table=default.Table_6bb2706114f147fa85c1389ab0b3e752 [id=ae602d681a3a4d1d9aecda22af00161b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.795558 27491 tablet_service.cc:1468] Processing CreateTablet for tablet db3033fc47d946cc9f3e806d223d44cd (DEFAULT_TABLE table=default.Table_3453a2165fe44a3d93deee7b39853d3e [id=fd5ce7d7e1cf47eda896ceb491e931d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.796002 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet ebf0a9aaaea1408c8c9caf28f00dbff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:20.797122 28234 raft_consensus.cc:491] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:20.797606 27321 leader_election.cc:304] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:20.784861 27364 tablet_service.cc:1468] Processing CreateTablet for tablet 5253d82a41494ce59bb96928179dfdd9 (DEFAULT_TABLE table=default.Table_a7492190504c4a388eea5ab3ebfe7b83 [id=0b54dabacc024398b526e985536bd605]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.801683 27685 raft_consensus.cc:2270] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:20.799404 27687 rpcz_store.cc:276] 0903 20:05:18.561684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.562169 (+ 485us) service_pool.cc:225] Handling call
0903 20:05:20.530362 (+1968193us) ts_tablet_manager.cc:1145] Deleting tablet 264127511a234e9fbf409f3aa2860ccc
0903 20:05:20.530374 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:20.661568 (+131194us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:20.675582 (+ 14014us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:20.773691 (+ 98109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1967594}
I20250903 20:05:20.800827 27354 tablet_service.cc:1515] Processing DeleteTablet for tablet 39c2ebe4ada5426590ec3769c8662644 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c772ab7c50c4994aacde5a05bb0f08e) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:20.802870 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db3033fc47d946cc9f3e806d223d44cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.803128 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_20274d6f2af744d485064ee2ce5d22d6" } modify_external_catalogs: true
I20250903 20:05:20.800858 27358 tablet_service.cc:1468] Processing CreateTablet for tablet 88a6d4c9d9024e718650263608f04be3 (DEFAULT_TABLE table=default.Table_6bb2706114f147fa85c1389ab0b3e752 [id=ae602d681a3a4d1d9aecda22af00161b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.803359 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_20274d6f2af744d485064ee2ce5d22d6" } modify_external_catalogs: true
I20250903 20:05:20.801281 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bf69315f07a470bb2b897ad7c590729 with delete_type TABLET_DATA_DELETED (replaced by tablet 17514a2af39b402abbe8263d7c1abc7f) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:20.803663 27065 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_4258f924ddb3449bb5f49fb7a88a5073" } modify_external_catalogs: true
I20250903 20:05:20.803859 27065 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_4258f924ddb3449bb5f49fb7a88a5073" } modify_external_catalogs: true
I20250903 20:05:20.801807 28298 raft_consensus.cc:2802] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:20.801729 27349 tablet_service.cc:1468] Processing CreateTablet for tablet ec8598ef27cc470fa48938de8d5196fc (DEFAULT_TABLE table=default.Table_e019b2e0ac534620ad809b330da7038b [id=698ad7ffb5f14b31a8808c63e60edd2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.805449 28234 raft_consensus.cc:3058] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:20.807420 27687 tablet_replica.cc:331] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:20.807502 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_004a021331544f1bbdd8a1ede39f7590"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:20.807756 27360 tablet_service.cc:1468] Processing CreateTablet for tablet c059dda502ea4bb28aa8ad7c7baa60e7 (DEFAULT_TABLE table=default.Table_39c530a33b00492fbae7db9de5ea26c5 [id=0d2198edd9954eff86e3527fa9ff58b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.807993 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_bd23bbbddf114a4286a815ee4468be81" } new_table_name: "default.Table_d3945d5a36d84041b35394333dc704f9" modify_external_catalogs: true
I20250903 20:05:20.808334 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_be949d6637d94ea4bab23c54e50fdb1d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:20.808876 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be949d6637d94ea4bab23c54e50fdb1d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:20.809562 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_004a021331544f1bbdd8a1ede39f7590 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.809900 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5253d82a41494ce59bb96928179dfdd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.810786 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6d201f66980a4a7ab430058673f8d513" } new_table_name: "default.Table_9eac2fd82a524aae943ebd76428f95eb" modify_external_catalogs: true
I20250903 20:05:20.812004 27347 tablet_service.cc:1468] Processing CreateTablet for tablet 13137053365a428e9d1e8d8852a53b6a (DEFAULT_TABLE table=default.Table_fb9f90cfd77144cc81d36b52a6fc032c [id=3fb365e8860542dfab5936bd1c0756ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.812394 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet e9eb24dd5a6243c49a3389320a3f78a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:20.812613 28298 raft_consensus.cc:491] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:20.812742 28298 raft_consensus.cc:3058] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:20.813175 27348 tablet_service.cc:1468] Processing CreateTablet for tablet 65d07ebe1ec94baabc6ce011e761b4d5 (DEFAULT_TABLE table=default.Table_026e7093a60a495e95489df73d689967 [id=772809879a0048a29129ed25c5e178ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.813877 27354 tablet_service.cc:1468] Processing CreateTablet for tablet ca17ddf02b514f87857bb98bbe45d04e (DEFAULT_TABLE table=default.Table_bae568b17ab54534914716b3111a041d [id=b14e9e91bad44ad980440df1dd5c384c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.814325 27687 raft_consensus.cc:2241] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:20.814550 27687 pending_rounds.cc:70] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528: Trying to abort 1 pending ops.
I20250903 20:05:20.814641 27687 pending_rounds.cc:77] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384951108550656 op_type: NO_OP noop_request { }
I20250903 20:05:20.814764 27687 raft_consensus.cc:2887] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:20.814888 27687 raft_consensus.cc:2270] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:20.815172 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88a6d4c9d9024e718650263608f04be3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.816279 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13137053365a428e9d1e8d8852a53b6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.817167 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c059dda502ea4bb28aa8ad7c7baa60e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.818183 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65d07ebe1ec94baabc6ce011e761b4d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.818950 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec8598ef27cc470fa48938de8d5196fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.819919 27354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca17ddf02b514f87857bb98bbe45d04e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.822906 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 37a0ea3651684b65a44f4b75e866b4a6 (DEFAULT_TABLE table=default.Table_acf2f9a7259946b3bfe30be00f5f0a66 [id=ed2ffbeef4744e968eb94fe0b54a8e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.823760 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 264127511a234e9fbf409f3aa2860ccc (table default.Table_1ade5bba3a18432c808ee7c4b910d247 [id=d20347a8196e44459448223a3af2dead]) successfully deleted
I20250903 20:05:20.827083 27687 ts_tablet_manager.cc:1905] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:20.827338 27229 tablet_service.cc:1468] Processing CreateTablet for tablet 37a0ea3651684b65a44f4b75e866b4a6 (DEFAULT_TABLE table=default.Table_acf2f9a7259946b3bfe30be00f5f0a66 [id=ed2ffbeef4744e968eb94fe0b54a8e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.828534 27695 tablet_bootstrap.cc:492] T f71d3dd9af2543299296f21381d1d563 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:20.828588 27229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37a0ea3651684b65a44f4b75e866b4a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.828748 27495 tablet_service.cc:1468] Processing CreateTablet for tablet ec8598ef27cc470fa48938de8d5196fc (DEFAULT_TABLE table=default.Table_e019b2e0ac534620ad809b330da7038b [id=698ad7ffb5f14b31a8808c63e60edd2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.829173 27496 tablet_service.cc:1468] Processing CreateTablet for tablet 5253d82a41494ce59bb96928179dfdd9 (DEFAULT_TABLE table=default.Table_a7492190504c4a388eea5ab3ebfe7b83 [id=0b54dabacc024398b526e985536bd605]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.829524 27502 tablet_service.cc:1468] Processing CreateTablet for tablet 65d07ebe1ec94baabc6ce011e761b4d5 (DEFAULT_TABLE table=default.Table_026e7093a60a495e95489df73d689967 [id=772809879a0048a29129ed25c5e178ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.830547 27493 tablet_service.cc:1468] Processing CreateTablet for tablet c059dda502ea4bb28aa8ad7c7baa60e7 (DEFAULT_TABLE table=default.Table_39c530a33b00492fbae7db9de5ea26c5 [id=0d2198edd9954eff86e3527fa9ff58b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.830860 27491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db3033fc47d946cc9f3e806d223d44cd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:20.832379 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4af8c03c9d834405b4e25d35441db961 consensus state has no leader
I20250903 20:05:20.832767 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 46e3a65118b744298b5536f88c22f1db
I20250903 20:05:20.833009 27692 tablet_bootstrap.cc:492] T fbfabb6ff1714d2e921f45256b543759 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:20.833101 27215 tablet_service.cc:1468] Processing CreateTablet for tablet ca17ddf02b514f87857bb98bbe45d04e (DEFAULT_TABLE table=default.Table_bae568b17ab54534914716b3111a041d [id=b14e9e91bad44ad980440df1dd5c384c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:20.833801 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 67ca1dcf6fc7435db555392235f9900f consensus state has no leader
I20250903 20:05:20.834836 27216 tablet_service.cc:1468] Processing CreateTablet for tablet 6db34f69b6344f90b268369c8e8f10a0 (DEFAULT_TABLE table=default.Table_3c841526bf5745998631d5afd01f5147 [id=a26de94c5e914218a167e7a6c1a2c0cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.835830 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 7a62d4e7c4994941b8f1a5cf9041c81d (DEFAULT_TABLE table=default.Table_d608d44f1a1447488bb7c52ee1456fdb [id=1c66aa18242346fa9114a0782a8e479f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:20.836768 28238 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:56534: Illegal state: RaftConsensus is not running: State = Stopped
I20250903 20:05:20.837016 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 0906165a50b047c8a776b30832149964
I20250903 20:05:20.838682 27225 tablet_service.cc:1468] Processing CreateTablet for tablet c432d5a0363f4d9b9cb23bfb6348382b (DEFAULT_TABLE table=default.Table_58f40fbfe6a946eab6b621584922f4a6 [id=36f6fbc890244f72a31dbb169392615d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.840086 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca17ddf02b514f87857bb98bbe45d04e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.841642 27211 tablet_service.cc:1468] Processing CreateTablet for tablet e4347c764cf54f9faaa13a2001ff638e (DEFAULT_TABLE table=default.Table_34e4bbb060e8432e89a9dfb67a1af00c [id=7dfada90dc6e433e8f8c9dfa85c87a75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.842574 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_50be0ce79d0944c0b752412411a5981a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:20.842978 28245 raft_consensus.cc:491] T 912eee5b48ef467bbd6bbe34f8aa390c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250903 20:05:20.843254 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50be0ce79d0944c0b752412411a5981a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.843240 28238 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:56534 (request call id 532) took 1801 ms. Trace:
I20250903 20:05:20.843608 27228 tablet_service.cc:1515] Processing DeleteTablet for tablet 46e3a65118b744298b5536f88c22f1db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:20.843906 28245 raft_consensus.cc:513] T 912eee5b48ef467bbd6bbe34f8aa390c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:20.844702 28245 leader_election.cc:290] T 912eee5b48ef467bbd6bbe34f8aa390c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:20.847247 27211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4347c764cf54f9faaa13a2001ff638e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.848569 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet 0906165a50b047c8a776b30832149964 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:20.850462 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7b75f7edb8c84d6c85a4cbc3faac617d" } new_table_name: "default.Table_76a88a96cd544cda882c413357a7f8c3" modify_external_catalogs: true
I20250903 20:05:20.854334 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c432d5a0363f4d9b9cb23bfb6348382b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.854265 27355 tablet_service.cc:1468] Processing CreateTablet for tablet 6db34f69b6344f90b268369c8e8f10a0 (DEFAULT_TABLE table=default.Table_3c841526bf5745998631d5afd01f5147 [id=a26de94c5e914218a167e7a6c1a2c0cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.855755 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_796a25442570476884197d82618f9ac6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:20.856415 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_796a25442570476884197d82618f9ac6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.861112 27216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6db34f69b6344f90b268369c8e8f10a0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:20.862617 27321 consensus_peers.cc:489] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Couldn't send request to peer 89f8f0ee68b940db9ce250c0604f934a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a10b079e7d5a440580ac00195fe88c65. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:20.863003 27361 tablet_service.cc:1468] Processing CreateTablet for tablet c432d5a0363f4d9b9cb23bfb6348382b (DEFAULT_TABLE table=default.Table_58f40fbfe6a946eab6b621584922f4a6 [id=36f6fbc890244f72a31dbb169392615d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.810896 28297 consensus_queue.cc:1035] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.091s
I20250903 20:05:20.854013 28238 rpcz_store.cc:276] 0903 20:05:19.041575 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:19.042660 (+ 1085us) service_pool.cc:225] Handling call
0903 20:05:20.843185 (+1800525us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:19.174493 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0903 20:05:19.199384 (+ 24891us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0903 20:05:20.781941 (+1582557us) alter_schema_op.cc:66] Acquired schema lock
0903 20:05:20.781954 (+ 13us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0903 20:05:20.782027 (+ 73us) alter_schema_op.cc:128] START. Timestamp: 1756929920781955
0903 20:05:20.782030 (+ 3us) op_driver.cc:348] REPLICATION: starting
0903 20:05:20.827966 (+ 45936us) spinlock_profiling.cc:243] Waited 19.8 ms on lock 0x6160000d3354. stack: 00007f589abe0dc6 00007f589abe0b80 00007f5899bc83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f58a663fedb 00007f58a79e82d5 00007f58a79e6df9 00007f58a79ef220 00007f58a79ef02d 0000000000a774dc 00007f589ac2714f 00007f589ac2a41b 00007f589ac2a298 0000000000a774dc
0903 20:05:20.827995 (+ 29us) op_driver.cc:389] HandleFailure(Illegal state: RaftConsensus is not running: State = Stopped)
0903 20:05:20.828006 (+ 11us) alter_schema_op.cc:171] AlterSchemaCommitCallback: op aborted
Metrics: {"child_traces":[["op",{"prepare.queue_time_us":118921,"spinlock_wait_cycles":45625600,"thread_start_us":92551,"threads_started":1}]]}
I20250903 20:05:20.873267 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37a0ea3651684b65a44f4b75e866b4a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.874331 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a62d4e7c4994941b8f1a5cf9041c81d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.876827 27366 tablet_service.cc:1468] Processing CreateTablet for tablet 7a62d4e7c4994941b8f1a5cf9041c81d (DEFAULT_TABLE table=default.Table_d608d44f1a1447488bb7c52ee1456fdb [id=1c66aa18242346fa9114a0782a8e479f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:20.877242 27321 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 62592.6us
I20250903 20:05:20.877627 27321 leader_election.cc:304] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:20.878511 28208 consensus_queue.cc:1035] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:20.878957 28208 raft_consensus.cc:2802] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:20.879070 28208 raft_consensus.cc:491] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:20.879148 28208 raft_consensus.cc:3058] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:20.879678 27350 tablet_service.cc:1468] Processing CreateTablet for tablet e4347c764cf54f9faaa13a2001ff638e (DEFAULT_TABLE table=default.Table_34e4bbb060e8432e89a9dfb67a1af00c [id=7dfada90dc6e433e8f8c9dfa85c87a75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.880184 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c432d5a0363f4d9b9cb23bfb6348382b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.881242 27355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6db34f69b6344f90b268369c8e8f10a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.881404 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88a6d4c9d9024e718650263608f04be3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.882628 27065 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 912eee5b48ef467bbd6bbe34f8aa390c (table default.Table_f1f4f57d96f04f0caff06d9a642f5c46 [id=fdc7899c8b334d309b834ae1daa14fa0]). Expected version 4 got 0
I20250903 20:05:20.883291 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a62d4e7c4994941b8f1a5cf9041c81d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:20.884146 27066 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24eadc70e10d4d02bcb643d45c9e0e36 consensus state has no leader
I20250903 20:05:20.886656 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_7b5eb542da2845c79bd273f704aa9159"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:20.887301 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b5eb542da2845c79bd273f704aa9159 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.887777 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9129a3e773a4fecb28ceee27778a294" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:20.887952 27249 raft_consensus.cc:3058] T c9129a3e773a4fecb28ceee27778a294 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:20.888816 27065 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_9eac2fd82a524aae943ebd76428f95eb" } modify_external_catalogs: true
I20250903 20:05:20.889074 27065 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_9eac2fd82a524aae943ebd76428f95eb" } modify_external_catalogs: true
W20250903 20:05:20.889278 27462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 91286.4us
I20250903 20:05:20.891785 27497 tablet_service.cc:1468] Processing CreateTablet for tablet 37a0ea3651684b65a44f4b75e866b4a6 (DEFAULT_TABLE table=default.Table_acf2f9a7259946b3bfe30be00f5f0a66 [id=ed2ffbeef4744e968eb94fe0b54a8e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.891119 27500 tablet_service.cc:1468] Processing CreateTablet for tablet 13137053365a428e9d1e8d8852a53b6a (DEFAULT_TABLE table=default.Table_fb9f90cfd77144cc81d36b52a6fc032c [id=3fb365e8860542dfab5936bd1c0756ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.892277 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec8598ef27cc470fa48938de8d5196fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.893373 27210 tablet_service.cc:1468] Processing CreateTablet for tablet b6663ba96faa44b88fe8b6def4ff6951 (DEFAULT_TABLE table=default.Table_affbe84913504de69e8d84b2e2d14cdc [id=a43439ca07504fe4a5849c20eec0cc61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.893615 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "912eee5b48ef467bbd6bbe34f8aa390c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:20.894346 28237 raft_consensus.cc:491] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:20.895906 27693 tablet_bootstrap.cc:654] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:20.900522 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "912eee5b48ef467bbd6bbe34f8aa390c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:20.901331 27183 leader_election.cc:343] T 912eee5b48ef467bbd6bbe34f8aa390c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:20.901404 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_5bf171fc5f3d4fe7b00fb86fd3e60d1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:20.901700 27499 tablet_service.cc:1468] Processing CreateTablet for tablet e4347c764cf54f9faaa13a2001ff638e (DEFAULT_TABLE table=default.Table_34e4bbb060e8432e89a9dfb67a1af00c [id=7dfada90dc6e433e8f8c9dfa85c87a75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.901932 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13137053365a428e9d1e8d8852a53b6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.906572 27685 ts_tablet_manager.cc:1905] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:20.897502 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4347c764cf54f9faaa13a2001ff638e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.908954 28237 raft_consensus.cc:513] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:20.909617 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65d07ebe1ec94baabc6ce011e761b4d5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:20.902194 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bf171fc5f3d4fe7b00fb86fd3e60d1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.916914 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 4af8c03c9d834405b4e25d35441db961
I20250903 20:05:20.917260 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_9b23868e884f40f0b6b2ffdb5e0c99d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:20.919268 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5253d82a41494ce59bb96928179dfdd9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:20.919831 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b23868e884f40f0b6b2ffdb5e0c99d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.922428 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c059dda502ea4bb28aa8ad7c7baa60e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.926398 27492 tablet_service.cc:1468] Processing CreateTablet for tablet c432d5a0363f4d9b9cb23bfb6348382b (DEFAULT_TABLE table=default.Table_58f40fbfe6a946eab6b621584922f4a6 [id=36f6fbc890244f72a31dbb169392615d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.926894 27489 tablet_service.cc:1468] Processing CreateTablet for tablet 6db34f69b6344f90b268369c8e8f10a0 (DEFAULT_TABLE table=default.Table_3c841526bf5745998631d5afd01f5147 [id=a26de94c5e914218a167e7a6c1a2c0cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.926986 27210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6663ba96faa44b88fe8b6def4ff6951. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.927058 27503 tablet_service.cc:1468] Processing CreateTablet for tablet 7a62d4e7c4994941b8f1a5cf9041c81d (DEFAULT_TABLE table=default.Table_d608d44f1a1447488bb7c52ee1456fdb [id=1c66aa18242346fa9114a0782a8e479f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.927188 27488 tablet_service.cc:1515] Processing DeleteTablet for tablet 46e3a65118b744298b5536f88c22f1db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:20.930274 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet 46e3a65118b744298b5536f88c22f1db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:20.935328 28297 raft_consensus.cc:491] T 8b3121fce1c44d16a0b083313dc67b7a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:20.936527 27501 tablet_service.cc:1515] Processing DeleteTablet for tablet 0906165a50b047c8a776b30832149964 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:20.937199 28301 raft_consensus.cc:513] T c9129a3e773a4fecb28ceee27778a294 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:20.938665 28297 raft_consensus.cc:513] T 8b3121fce1c44d16a0b083313dc67b7a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:20.939569 28297 leader_election.cc:290] T 8b3121fce1c44d16a0b083313dc67b7a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:20.939782 28297 raft_consensus.cc:491] T e9eb24dd5a6243c49a3389320a3f78a0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:20.939939 28297 raft_consensus.cc:513] T e9eb24dd5a6243c49a3389320a3f78a0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:20.940572 28297 leader_election.cc:290] T e9eb24dd5a6243c49a3389320a3f78a0 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:20.940979 27064 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ccda245b73064dc2b5634f9593b1888c" } modify_external_catalogs: true
I20250903 20:05:20.941167 27064 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ccda245b73064dc2b5634f9593b1888c" } modify_external_catalogs: true
I20250903 20:05:20.942498 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet 0906165a50b047c8a776b30832149964 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:20.944304 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0253a17daef94a8fb2be91d233de40cd" } new_table_name: "default.Table_8373ad89ca4a41c0ab4d117a017b543c" modify_external_catalogs: true
I20250903 20:05:20.944566 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_016e6db78b02447b8777dba09ae06057" } new_table_name: "default.Table_c82f96b0a1524799b48098e7a76446c9" modify_external_catalogs: true
I20250903 20:05:20.946067 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_fc74bbf4720b42328ed8366e277815e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:20.946777 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc74bbf4720b42328ed8366e277815e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.946887 27065 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2fa1d4a13c1840cd88c36e5fa2c6475f" } modify_external_catalogs: true
I20250903 20:05:20.947129 27065 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2fa1d4a13c1840cd88c36e5fa2c6475f" } modify_external_catalogs: true
I20250903 20:05:20.947602 27213 tablet_service.cc:1468] Processing CreateTablet for tablet 0592818fe1214f9a9432222162799af4 (DEFAULT_TABLE table=default.Table_2dcb231e5186428fa09678c7968ef431 [id=9c6b30d1c88842ca88899c445eec6890]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.947991 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0592818fe1214f9a9432222162799af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.949643 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_afd931c89f1b4dcab359c5c7fa189952"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:20.950189 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_afd931c89f1b4dcab359c5c7fa189952 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:20.952775 27359 tablet_service.cc:1468] Processing CreateTablet for tablet b6663ba96faa44b88fe8b6def4ff6951 (DEFAULT_TABLE table=default.Table_affbe84913504de69e8d84b2e2d14cdc [id=a43439ca07504fe4a5849c20eec0cc61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.961423 27490 tablet_service.cc:1468] Processing CreateTablet for tablet ca17ddf02b514f87857bb98bbe45d04e (DEFAULT_TABLE table=default.Table_bae568b17ab54534914716b3111a041d [id=b14e9e91bad44ad980440df1dd5c384c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:20.968497 27059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f4d3e1cfdfab4be1a9143b78bf3bc46d consensus state has no leader
I20250903 20:05:20.968434 27357 tablet_service.cc:1468] Processing CreateTablet for tablet 2c772ab7c50c4994aacde5a05bb0f08e (DEFAULT_TABLE table=default.Table_d84f067d5a924a23964078d360422878 [id=5cb314181f104d63b1aed98c7198d5c3]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:20.969388 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b4b15d77d07c4fa7856d9c198cf15450 consensus state has no leader
I20250903 20:05:20.973649 27060 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 912eee5b48ef467bbd6bbe34f8aa390c (table default.Table_f1f4f57d96f04f0caff06d9a642f5c46 [id=fdc7899c8b334d309b834ae1daa14fa0]). Expected version 4 got 0
I20250903 20:05:20.975358 28301 leader_election.cc:290] T c9129a3e773a4fecb28ceee27778a294 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:20.982632 28237 leader_election.cc:290] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:20.983965 27352 tablet_service.cc:1468] Processing CreateTablet for tablet 96c9875830b246b3a9117536254cbc4e (DEFAULT_TABLE table=default.Table_a534288cb6194102ade0cbb0b87630ee [id=d365989f821c4f1690bbaedb66fa7716]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.984678 27249 raft_consensus.cc:2466] T c9129a3e773a4fecb28ceee27778a294 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:20.985401 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c772ab7c50c4994aacde5a05bb0f08e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.987589 27356 tablet_service.cc:1468] Processing CreateTablet for tablet 0592818fe1214f9a9432222162799af4 (DEFAULT_TABLE table=default.Table_2dcb231e5186428fa09678c7968ef431 [id=9c6b30d1c88842ca88899c445eec6890]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.988262 27362 tablet_service.cc:1468] Processing CreateTablet for tablet b03708700ae94f27a9d4bf5634f9ab0e (DEFAULT_TABLE table=default.Table_4bb1596fb7204c0894e7a5d694b16943 [id=f0b5aab83f714a66b533dca925219326]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.989892 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9129a3e773a4fecb28ceee27778a294" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:20.992779 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96c9875830b246b3a9117536254cbc4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:20.996385 28298 raft_consensus.cc:513] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:20.997555 27245 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9129a3e773a4fecb28ceee27778a294" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:20.997793 27245 raft_consensus.cc:2391] T c9129a3e773a4fecb28ceee27778a294 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
I20250903 20:05:20.999804 27351 tablet_service.cc:1468] Processing CreateTablet for tablet 17514a2af39b402abbe8263d7c1abc7f (DEFAULT_TABLE table=default.Table_5fb98b65f51b4d628de6bed5d36b53d1 [id=b82c8f8f0f6b415580c2b590f05c0f7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.000000 27380 raft_consensus.cc:2391] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
I20250903 20:05:21.000559 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet d4c703dafbca4e669dbb18946ffac699
I20250903 20:05:21.001888 27211 tablet_service.cc:1468] Processing CreateTablet for tablet 2c772ab7c50c4994aacde5a05bb0f08e (DEFAULT_TABLE table=default.Table_d84f067d5a924a23964078d360422878 [id=5cb314181f104d63b1aed98c7198d5c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.002333 27211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c772ab7c50c4994aacde5a05bb0f08e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.001609 27224 tablet_service.cc:1468] Processing CreateTablet for tablet 96c9875830b246b3a9117536254cbc4e (DEFAULT_TABLE table=default.Table_a534288cb6194102ade0cbb0b87630ee [id=d365989f821c4f1690bbaedb66fa7716]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.004334 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0592818fe1214f9a9432222162799af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.007534 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96c9875830b246b3a9117536254cbc4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.010006 28234 raft_consensus.cc:513] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:21.014663 28298 leader_election.cc:290] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:21.015820 27220 tablet_service.cc:1468] Processing CreateTablet for tablet 17514a2af39b402abbe8263d7c1abc7f (DEFAULT_TABLE table=default.Table_5fb98b65f51b4d628de6bed5d36b53d1 [id=b82c8f8f0f6b415580c2b590f05c0f7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.017570 27353 tablet_service.cc:1468] Processing CreateTablet for tablet 811c22a650c747a6a8db710e3cdd847c (DEFAULT_TABLE table=default.Table_791801575c0e4c98b42111d7a3254236 [id=a26aa913002a401fb50365a0256a5fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.017742 27215 tablet_service.cc:1468] Processing CreateTablet for tablet b03708700ae94f27a9d4bf5634f9ab0e (DEFAULT_TABLE table=default.Table_4bb1596fb7204c0894e7a5d694b16943 [id=f0b5aab83f714a66b533dca925219326]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.021907 27227 tablet_service.cc:1468] Processing CreateTablet for tablet 811c22a650c747a6a8db710e3cdd847c (DEFAULT_TABLE table=default.Table_791801575c0e4c98b42111d7a3254236 [id=a26aa913002a401fb50365a0256a5fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.022292 27214 tablet_service.cc:1468] Processing CreateTablet for tablet c49767d618be4fd18f30e5124ce0cb84 (DEFAULT_TABLE table=default.Table_be949d6637d94ea4bab23c54e50fdb1d [id=e73f9a974e454b049a8176d31275c9c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.022584 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 0d0b3804ced547f3bd482399cbaa9270 (DEFAULT_TABLE table=default.Table_796a25442570476884197d82618f9ac6 [id=83ac2957d35e48e79b3f75522a5e75a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.022857 27223 tablet_service.cc:1468] Processing CreateTablet for tablet 5b26ed39f2144610b786f49e62e6f0a3 (DEFAULT_TABLE table=default.Table_50be0ce79d0944c0b752412411a5981a [id=7302318a3c6a428a933d973ce0e0ca55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.023046 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet d4c703dafbca4e669dbb18946ffac699 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:21.023176 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17514a2af39b402abbe8263d7c1abc7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.023901 27223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b26ed39f2144610b786f49e62e6f0a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.026419 27219 tablet_service.cc:1468] Processing CreateTablet for tablet 441bb228c5c54aebb6aba2bc5761bd00 (DEFAULT_TABLE table=default.Table_004a021331544f1bbdd8a1ede39f7590 [id=bbbde9e4eff54241b58349126f7a4dd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.026700 27219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 441bb228c5c54aebb6aba2bc5761bd00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.029626 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b03708700ae94f27a9d4bf5634f9ab0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.029762 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b03708700ae94f27a9d4bf5634f9ab0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.030427 27353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 811c22a650c747a6a8db710e3cdd847c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.030865 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d0b3804ced547f3bd482399cbaa9270. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.030932 27695 tablet_bootstrap.cc:654] T f71d3dd9af2543299296f21381d1d563 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:21.031117 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c49767d618be4fd18f30e5124ce0cb84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.031352 27351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17514a2af39b402abbe8263d7c1abc7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.031978 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 811c22a650c747a6a8db710e3cdd847c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.035131 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6663ba96faa44b88fe8b6def4ff6951. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.038251 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet 4af8c03c9d834405b4e25d35441db961 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:21.038797 27228 tablet_service.cc:1468] Processing CreateTablet for tablet 8093e36d79894ed3aa8e92c0bca4f93d (DEFAULT_TABLE table=default.Table_0c13f21b7b574e63abd419e44ea9b157 [id=9c8f82f1a60b46c88e51b6bbf6a42986]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.039167 27228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8093e36d79894ed3aa8e92c0bca4f93d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.043202 28234 leader_election.cc:290] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:21.045964 27685 ts_tablet_manager.cc:1918] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20250903 20:05:21.049015 27184 leader_election.cc:343] T 912eee5b48ef467bbd6bbe34f8aa390c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:21.050714 27184 leader_election.cc:304] T 912eee5b48ef467bbd6bbe34f8aa390c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:21.050963 28245 raft_consensus.cc:2747] T 912eee5b48ef467bbd6bbe34f8aa390c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:21.050480 27685 log.cc:1199] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/68cff0e416a54a5eb37071f5a12f07ab
I20250903 20:05:21.056238 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d84f067d5a924a23964078d360422878" } modify_external_catalogs: true
I20250903 20:05:21.056464 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d84f067d5a924a23964078d360422878" } modify_external_catalogs: true
I20250903 20:05:21.056643 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_f6658302f6a546d0a9a2348e78572b99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:21.056981 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ca85cb69be1e4e709c000e5b77bcd3cf" } modify_external_catalogs: true
I20250903 20:05:21.057147 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ca85cb69be1e4e709c000e5b77bcd3cf" } modify_external_catalogs: true
W20250903 20:05:21.057464 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6658302f6a546d0a9a2348e78572b99 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:21.056134 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_14a46b39c912447db13c3d3d84cad475"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:21.058044 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14a46b39c912447db13c3d3d84cad475 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:21.059330 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c5256f043b61433b8debe9e13625bd0d" } new_table_name: "default.Table_fd3053c3354a4359b2742e9350d19d23" modify_external_catalogs: true
I20250903 20:05:21.061441 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_9502dacb275544f5abe3e80a9487371a" } new_table_name: "default.Table_d2625760f0884f809e7aa5e436cddec1" modify_external_catalogs: true
I20250903 20:05:21.063669 28208 raft_consensus.cc:513] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:21.064617 28208 leader_election.cc:290] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:21.069494 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 2c772ab7c50c4994aacde5a05bb0f08e (DEFAULT_TABLE table=default.Table_d84f067d5a924a23964078d360422878 [id=5cb314181f104d63b1aed98c7198d5c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.069764 27488 tablet_service.cc:1468] Processing CreateTablet for tablet 0592818fe1214f9a9432222162799af4 (DEFAULT_TABLE table=default.Table_2dcb231e5186428fa09678c7968ef431 [id=9c6b30d1c88842ca88899c445eec6890]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:20.984922 27501 tablet_service.cc:1468] Processing CreateTablet for tablet b6663ba96faa44b88fe8b6def4ff6951 (DEFAULT_TABLE table=default.Table_affbe84913504de69e8d84b2e2d14cdc [id=a43439ca07504fe4a5849c20eec0cc61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.074373 27462 leader_election.cc:304] T c9129a3e773a4fecb28ceee27778a294 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:21.078518 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_624f5e99e4d344d3ae431fd805971ac7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:21.079144 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_624f5e99e4d344d3ae431fd805971ac7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:21.082610 27685 ts_tablet_manager.cc:1939] T 68cff0e416a54a5eb37071f5a12f07ab P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:21.083039 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 381) took 4083 ms. Trace:
I20250903 20:05:21.073856 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 17514a2af39b402abbe8263d7c1abc7f (DEFAULT_TABLE table=default.Table_5fb98b65f51b4d628de6bed5d36b53d1 [id=b82c8f8f0f6b415580c2b590f05c0f7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.083122 27685 rpcz_store.cc:276] 0903 20:05:17.008011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.008478 (+ 467us) service_pool.cc:225] Handling call
0903 20:05:20.752580 (+3744102us) ts_tablet_manager.cc:1145] Deleting tablet 68cff0e416a54a5eb37071f5a12f07ab
0903 20:05:20.752591 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.043051 (+290460us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.045876 (+ 2825us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.083023 (+ 37147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3740460}
I20250903 20:05:21.083513 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 382) took 4073 ms. Trace:
I20250903 20:05:21.083585 27685 rpcz_store.cc:276] 0903 20:05:17.014273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.014605 (+ 332us) service_pool.cc:225] Handling call
0903 20:05:21.083409 (+4068804us) ts_tablet_manager.cc:1145] Deleting tablet 47c1287bff1145a5b2ab6d14dbd608c7
0903 20:05:21.083417 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.083498 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4031167}
I20250903 20:05:21.083704 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 68cff0e416a54a5eb37071f5a12f07ab (table default.Table_1e3f794ae5ec4ad6a9b4351b4bda7b15 [id=71d850a0cea14b0ba337a822ad99d513]) successfully deleted
I20250903 20:05:21.083920 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 383) took 4031 ms. Trace:
I20250903 20:05:21.083997 27685 rpcz_store.cc:276] 0903 20:05:17.054067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.054423 (+ 356us) service_pool.cc:225] Handling call
0903 20:05:21.083821 (+4029398us) ts_tablet_manager.cc:1145] Deleting tablet 238f2f30377b42c2852b016018aa3245
0903 20:05:21.083828 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.083903 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4027574}
I20250903 20:05:21.084270 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 385) took 4014 ms. Trace:
I20250903 20:05:21.084353 27685 rpcz_store.cc:276] 0903 20:05:17.069523 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.069580 (+ 57us) service_pool.cc:225] Handling call
0903 20:05:21.084181 (+4014601us) ts_tablet_manager.cc:1145] Deleting tablet e78e6d212daf4589a6b4b65711e356cb
0903 20:05:21.084188 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.084257 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16309,"tablet-delete.queue_time_us":4014381}
I20250903 20:05:21.084641 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 384) took 4028 ms. Trace:
I20250903 20:05:21.084707 27685 rpcz_store.cc:276] 0903 20:05:17.069214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.077498 (+ 8284us) service_pool.cc:225] Handling call
0903 20:05:21.084546 (+4007048us) ts_tablet_manager.cc:1145] Deleting tablet 120ca61b3da742dc9a2efa483ead7e97
0903 20:05:21.084553 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.084626 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8485,"tablet-delete.queue_time_us":4006843}
I20250903 20:05:21.084980 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 412) took 3606 ms. Trace:
I20250903 20:05:21.085052 27685 rpcz_store.cc:276] 0903 20:05:17.478010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.608955 (+130945us) service_pool.cc:225] Handling call
0903 20:05:21.084892 (+3475937us) ts_tablet_manager.cc:1145] Deleting tablet 886b8783aa9d4618aad498d4a1115843
0903 20:05:21.084899 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.084966 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3458722}
I20250903 20:05:21.085350 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 420) took 3597 ms. Trace:
I20250903 20:05:21.085414 27685 rpcz_store.cc:276] 0903 20:05:17.487581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.753127 (+265546us) service_pool.cc:225] Handling call
0903 20:05:21.085243 (+3332116us) ts_tablet_manager.cc:1145] Deleting tablet f326ec776ba446be8d5dda5c1dc5b6ec
0903 20:05:21.085252 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.085334 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3331860}
I20250903 20:05:21.085703 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 421) took 3598 ms. Trace:
I20250903 20:05:21.085763 27685 rpcz_store.cc:276] 0903 20:05:17.487672 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.753402 (+265730us) service_pool.cc:225] Handling call
0903 20:05:21.085618 (+3332216us) ts_tablet_manager.cc:1145] Deleting tablet 9c77bc0e8bb34712bf8109cadbc0fc06
0903 20:05:21.085626 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.085690 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3332084}
I20250903 20:05:21.086022 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 422) took 3598 ms. Trace:
I20250903 20:05:21.086440 27494 tablet_service.cc:1468] Processing CreateTablet for tablet 811c22a650c747a6a8db710e3cdd847c (DEFAULT_TABLE table=default.Table_791801575c0e4c98b42111d7a3254236 [id=a26aa913002a401fb50365a0256a5fa0]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:21.087196 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 47c1287bff1145a5b2ab6d14dbd608c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47c1287bff1145a5b2ab6d14dbd608c7 already in progress: creating tablet
W20250903 20:05:21.087692 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 238f2f30377b42c2852b016018aa3245 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 238f2f30377b42c2852b016018aa3245 already in progress: creating tablet
W20250903 20:05:21.088085 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet e78e6d212daf4589a6b4b65711e356cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e78e6d212daf4589a6b4b65711e356cb already in progress: creating tablet
I20250903 20:05:21.088060 27498 tablet_service.cc:1468] Processing CreateTablet for tablet 96c9875830b246b3a9117536254cbc4e (DEFAULT_TABLE table=default.Table_a534288cb6194102ade0cbb0b87630ee [id=d365989f821c4f1690bbaedb66fa7716]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.088222 28301 raft_consensus.cc:2747] T c9129a3e773a4fecb28ceee27778a294 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20250903 20:05:21.088482 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 120ca61b3da742dc9a2efa483ead7e97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 120ca61b3da742dc9a2efa483ead7e97 already in progress: creating tablet
W20250903 20:05:21.088913 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 886b8783aa9d4618aad498d4a1115843 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 886b8783aa9d4618aad498d4a1115843 already in progress: creating tablet
I20250903 20:05:21.091476 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_d3c128a2d7454429bbe0c058c808a9f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:21.092103 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3c128a2d7454429bbe0c058c808a9f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:21.093640 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c891a3989f9a42dbb10ac8788775a83a" } new_table_name: "default.Table_39c7691206fc4227adff3a4a904a2ccb" modify_external_catalogs: true
I20250903 20:05:21.095115 27060 master_service.cc:948] ReplaceTablet: received request to replace tablet ed4902c002ef4198802375ced39ef2ad from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:21.096467 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_c047ef78f17b427ab6e3e998938c5fe1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:21.097417 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c047ef78f17b427ab6e3e998938c5fe1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:21.098734 27504 tablet_service.cc:1468] Processing CreateTablet for tablet b03708700ae94f27a9d4bf5634f9ab0e (DEFAULT_TABLE table=default.Table_4bb1596fb7204c0894e7a5d694b16943 [id=f0b5aab83f714a66b533dca925219326]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.099776 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet a72b85dfb34047528b2b28ff0d40f828
W20250903 20:05:21.116344 27462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 57283.4us
I20250903 20:05:21.123967 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet 4af8c03c9d834405b4e25d35441db961 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:21.124819 27364 tablet_service.cc:1468] Processing CreateTablet for tablet 8093e36d79894ed3aa8e92c0bca4f93d (DEFAULT_TABLE table=default.Table_0c13f21b7b574e63abd419e44ea9b157 [id=9c8f82f1a60b46c88e51b6bbf6a42986]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.133158 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_56af93fd7df844a99722a1b88e0c54c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:21.133886 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56af93fd7df844a99722a1b88e0c54c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:21.142827 27065 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_8bbb5561118c4a68af214f8d319d2caf" } modify_external_catalogs: true
I20250903 20:05:21.143124 27065 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_8bbb5561118c4a68af214f8d319d2caf" } modify_external_catalogs: true
I20250903 20:05:21.144325 27066 master_service.cc:948] ReplaceTablet: received request to replace tablet b7a6d5d8a97a4769a81ec7224d591f8c from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:21.151214 27500 tablet_service.cc:1515] Processing DeleteTablet for tablet 4af8c03c9d834405b4e25d35441db961 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:21.086444 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9eb24dd5a6243c49a3389320a3f78a0" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:21.154385 27492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c432d5a0363f4d9b9cb23bfb6348382b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.155560 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8093e36d79894ed3aa8e92c0bca4f93d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.159900 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_7a8637d7aab94246a883a6e29273e755"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:21.160581 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a8637d7aab94246a883a6e29273e755 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:21.161510 28184 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 20:05:21.162372 27358 tablet_service.cc:1468] Processing CreateTablet for tablet 441bb228c5c54aebb6aba2bc5761bd00 (DEFAULT_TABLE table=default.Table_004a021331544f1bbdd8a1ede39f7590 [id=bbbde9e4eff54241b58349126f7a4dd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.163726 28184 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 20:05:21.164126 28184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 20:05:21.158516 27354 tablet_service.cc:1468] Processing CreateTablet for tablet 0d0b3804ced547f3bd482399cbaa9270 (DEFAULT_TABLE table=default.Table_796a25442570476884197d82618f9ac6 [id=83ac2957d35e48e79b3f75522a5e75a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.165282 27348 tablet_service.cc:1468] Processing CreateTablet for tablet 5b26ed39f2144610b786f49e62e6f0a3 (DEFAULT_TABLE table=default.Table_50be0ce79d0944c0b752412411a5981a [id=7302318a3c6a428a933d973ce0e0ca55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.168063 27502 tablet_service.cc:1468] Processing CreateTablet for tablet 8093e36d79894ed3aa8e92c0bca4f93d (DEFAULT_TABLE table=default.Table_0c13f21b7b574e63abd419e44ea9b157 [id=9c8f82f1a60b46c88e51b6bbf6a42986]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.168504 27692 tablet_bootstrap.cc:654] T fbfabb6ff1714d2e921f45256b543759 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:21.169648 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9129a3e773a4fecb28ceee27778a294" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:21.171684 27354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d0b3804ced547f3bd482399cbaa9270. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.172520 27366 tablet_service.cc:1468] Processing CreateTablet for tablet c49767d618be4fd18f30e5124ce0cb84 (DEFAULT_TABLE table=default.Table_be949d6637d94ea4bab23c54e50fdb1d [id=e73f9a974e454b049a8176d31275c9c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.173015 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet a72b85dfb34047528b2b28ff0d40f828 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:21.177434 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_db5fb46856fa43dc9e96685bc4d691c3" } new_table_name: "default.Table_dc24dd98fc4146eda8660576fd53a29d" modify_external_catalogs: true
I20250903 20:05:21.179764 27347 tablet_service.cc:1515] Processing DeleteTablet for tablet d4c703dafbca4e669dbb18946ffac699 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:21.180258 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b26ed39f2144610b786f49e62e6f0a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.185565 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 441bb228c5c54aebb6aba2bc5761bd00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.185899 27525 raft_consensus.cc:2391] T c9129a3e773a4fecb28ceee27778a294 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 51578fb7bceb4e119b3ab1e58c4f5528 in current term 1: Already voted for candidate e72ef1dbcc5143cf9f04a808f101c80f in this term.
I20250903 20:05:21.186057 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c49767d618be4fd18f30e5124ce0cb84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.192116 27491 tablet_service.cc:1468] Processing CreateTablet for tablet 441bb228c5c54aebb6aba2bc5761bd00 (DEFAULT_TABLE table=default.Table_004a021331544f1bbdd8a1ede39f7590 [id=bbbde9e4eff54241b58349126f7a4dd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.196658 27494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 811c22a650c747a6a8db710e3cdd847c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.206445 27687 ts_tablet_manager.cc:1918] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250903 20:05:21.210300 27320 leader_election.cc:343] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 1fef9400227449ed892daff588999caa
W20250903 20:05:21.210794 27320 leader_election.cc:343] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 4c543ac4df33429b9787c6d32bf1ebeb
I20250903 20:05:21.210551 27687 log.cc:1199] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/fdc77810f27a40758e063b8b1150ab71
W20250903 20:05:21.214303 27320 leader_election.cc:343] T 8b3121fce1c44d16a0b083313dc67b7a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 8b3121fce1c44d16a0b083313dc67b7a
W20250903 20:05:21.214855 27320 consensus_peers.cc:489] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ecd339ec12e84e88acec427630cc8668. This is attempt 1: this message will repeat every 5th retry.
W20250903 20:05:21.216710 27320 consensus_peers.cc:489] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a10b079e7d5a440580ac00195fe88c65. This is attempt 1: this message will repeat every 5th retry.
W20250903 20:05:21.216975 27320 leader_election.cc:343] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 3f78a951bb024cf290116b0fe438ddb4
I20250903 20:05:21.209770 27505 tablet_service.cc:1468] Processing CreateTablet for tablet c49767d618be4fd18f30e5124ce0cb84 (DEFAULT_TABLE table=default.Table_be949d6637d94ea4bab23c54e50fdb1d [id=e73f9a974e454b049a8176d31275c9c5]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:21.219293 27320 leader_election.cc:343] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: deb84fc3465443d99c11097e53f5f575
W20250903 20:05:21.219633 27320 leader_election.cc:343] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 1fef9400227449ed892daff588999caa
I20250903 20:05:21.219867 28245 raft_consensus.cc:491] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51578fb7bceb4e119b3ab1e58c4f5528)
I20250903 20:05:21.220919 28315 consensus_queue.cc:1035] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20250903 20:05:21.230796 27687 ts_tablet_manager.cc:1939] T fdc77810f27a40758e063b8b1150ab71 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:21.234725 27500 tablet_service.cc:1468] Processing CreateTablet for tablet 0d0b3804ced547f3bd482399cbaa9270 (DEFAULT_TABLE table=default.Table_796a25442570476884197d82618f9ac6 [id=83ac2957d35e48e79b3f75522a5e75a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.240311 27491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 441bb228c5c54aebb6aba2bc5761bd00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.240878 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet d4c703dafbca4e669dbb18946ffac699 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:21.241303 27496 tablet_service.cc:1468] Processing CreateTablet for tablet 5b26ed39f2144610b786f49e62e6f0a3 (DEFAULT_TABLE table=default.Table_50be0ce79d0944c0b752412411a5981a [id=7302318a3c6a428a933d973ce0e0ca55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.244179 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0592818fe1214f9a9432222162799af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.244531 27489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6db34f69b6344f90b268369c8e8f10a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.246033 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37a0ea3651684b65a44f4b75e866b4a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.246300 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b26ed39f2144610b786f49e62e6f0a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.248754 27501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6663ba96faa44b88fe8b6def4ff6951. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.249493 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8093e36d79894ed3aa8e92c0bca4f93d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.250533 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet a72b85dfb34047528b2b28ff0d40f828 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:21.268193 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 501) took 2693 ms. Trace:
I20250903 20:05:21.274006 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0bbccf098ebf46ae8d825952b89a2357" } new_table_name: "default.Table_ea5cb66a574f4466bf7e7f635f22b426" modify_external_catalogs: true
W20250903 20:05:21.275739 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f204f3ead373415daf7c3e765dd8f59e consensus state has no leader
I20250903 20:05:21.276286 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_d7d1a268897341aeb586cae942a487b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:21.276978 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7d1a268897341aeb586cae942a487b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:21.283686 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a62d4e7c4994941b8f1a5cf9041c81d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.284907 27687 rpcz_store.cc:276] 0903 20:05:18.574654 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.606036 (+ 31382us) service_pool.cc:225] Handling call
0903 20:05:20.802481 (+2196445us) ts_tablet_manager.cc:1145] Deleting tablet fdc77810f27a40758e063b8b1150ab71
0903 20:05:20.802494 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.093565 (+291071us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.198032 (+104467us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.268122 (+ 70090us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2196083}
I20250903 20:05:21.285799 27687 tablet_replica.cc:331] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:21.285795 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet fdc77810f27a40758e063b8b1150ab71 (table default.Table_9c5c22a18c1d4913abdfab7d6a589058 [id=84da55ea3b4946f0b87d8d6389c63403]) successfully deleted
I20250903 20:05:21.280767 27693 tablet_bootstrap.cc:492] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:21.286648 27687 raft_consensus.cc:2241] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:21.287796 27687 pending_rounds.cc:70] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528: Trying to abort 1 pending ops.
I20250903 20:05:21.287921 27687 pending_rounds.cc:77] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384951836143616 op_type: NO_OP noop_request { }
I20250903 20:05:21.288089 27687 raft_consensus.cc:2887] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:21.288239 27687 raft_consensus.cc:2270] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:21.289033 27687 ts_tablet_manager.cc:1905] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:21.294781 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b03708700ae94f27a9d4bf5634f9ab0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.295224 27498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96c9875830b246b3a9117536254cbc4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.296046 27490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca17ddf02b514f87857bb98bbe45d04e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.086091 27685 rpcz_store.cc:276] 0903 20:05:17.487770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.753545 (+265775us) service_pool.cc:225] Handling call
0903 20:05:21.085935 (+3332390us) ts_tablet_manager.cc:1145] Deleting tablet 950a6fcd35f64f2d889831fe9cdf503a
0903 20:05:21.085943 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.086008 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3332187}
I20250903 20:05:21.300835 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 423) took 3805 ms. Trace:
I20250903 20:05:21.296641 27693 ts_tablet_manager.cc:1397] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.577s user 0.013s sys 0.008s
I20250903 20:05:21.300942 27685 rpcz_store.cc:276] 0903 20:05:17.494929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.753762 (+258833us) service_pool.cc:225] Handling call
0903 20:05:21.300678 (+3546916us) ts_tablet_manager.cc:1145] Deleting tablet f71d3dd9af2543299296f21381d1d563
0903 20:05:21.300689 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.300809 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3546759}
I20250903 20:05:21.303740 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 434) took 3786 ms. Trace:
I20250903 20:05:21.303826 27685 rpcz_store.cc:276] 0903 20:05:17.517063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.869667 (+352604us) service_pool.cc:225] Handling call
0903 20:05:21.303625 (+3433958us) ts_tablet_manager.cc:1145] Deleting tablet 5fb2426aa01e47598f268d0a90475271
0903 20:05:21.303634 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.303723 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3414465}
I20250903 20:05:21.304152 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 436) took 3786 ms. Trace:
I20250903 20:05:21.304253 27685 rpcz_store.cc:276] 0903 20:05:17.517342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.889206 (+371864us) service_pool.cc:225] Handling call
0903 20:05:21.304043 (+3414837us) ts_tablet_manager.cc:1145] Deleting tablet ee40566e497740789575275d0cdca0db
0903 20:05:21.304051 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.304135 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3391753}
I20250903 20:05:21.304620 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 437) took 3787 ms. Trace:
I20250903 20:05:21.304687 27685 rpcz_store.cc:276] 0903 20:05:17.517480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.912334 (+394854us) service_pool.cc:225] Handling call
0903 20:05:21.304502 (+3392168us) ts_tablet_manager.cc:1145] Deleting tablet 33fb6b357bda43e7abeaed5634183ef6
0903 20:05:21.304513 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.304606 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3351262}
I20250903 20:05:21.305017 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 438) took 3787 ms. Trace:
I20250903 20:05:21.305075 27685 rpcz_store.cc:276] 0903 20:05:17.517679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.953269 (+435590us) service_pool.cc:225] Handling call
0903 20:05:21.304928 (+3351659us) ts_tablet_manager.cc:1145] Deleting tablet 60c32370a5274347afafc6a7e1e94270
0903 20:05:21.304938 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.305005 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3350295}
I20250903 20:05:21.305373 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 435) took 3788 ms. Trace:
I20250903 20:05:21.305428 27685 rpcz_store.cc:276] 0903 20:05:17.517219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.890204 (+372985us) service_pool.cc:225] Handling call
0903 20:05:21.305293 (+3415089us) ts_tablet_manager.cc:1145] Deleting tablet e0347eab6c8642beabdc2d7e7bbfdc9b
0903 20:05:21.305300 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.305361 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3350104}
I20250903 20:05:21.305661 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 442) took 3787 ms. Trace:
I20250903 20:05:21.305711 27685 rpcz_store.cc:276] 0903 20:05:17.518264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:17.962717 (+444453us) service_pool.cc:225] Handling call
0903 20:05:21.305590 (+3342873us) ts_tablet_manager.cc:1145] Deleting tablet 8a223f75045345ae833f52bcc8f239f9
0903 20:05:21.305597 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.305651 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3265658}
I20250903 20:05:21.305941 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 444) took 3772 ms. Trace:
I20250903 20:05:21.305991 27685 rpcz_store.cc:276] 0903 20:05:17.533512 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.039975 (+506463us) service_pool.cc:225] Handling call
0903 20:05:21.305860 (+3265885us) ts_tablet_manager.cc:1145] Deleting tablet a3103e1595ec4c38ba65dca26fe1167c
0903 20:05:21.305866 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.305931 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3257119}
W20250903 20:05:21.307704 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet f326ec776ba446be8d5dda5c1dc5b6ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f326ec776ba446be8d5dda5c1dc5b6ec already in progress: creating tablet
W20250903 20:05:21.308029 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 9c77bc0e8bb34712bf8109cadbc0fc06 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c77bc0e8bb34712bf8109cadbc0fc06 already in progress: creating tablet
W20250903 20:05:21.308508 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 950a6fcd35f64f2d889831fe9cdf503a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 950a6fcd35f64f2d889831fe9cdf503a already in progress: creating tablet
W20250903 20:05:21.308787 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet f71d3dd9af2543299296f21381d1d563 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f71d3dd9af2543299296f21381d1d563 already in progress: creating tablet
W20250903 20:05:21.308984 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 5fb2426aa01e47598f268d0a90475271 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5fb2426aa01e47598f268d0a90475271 already in progress: creating tablet
W20250903 20:05:21.309204 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet ee40566e497740789575275d0cdca0db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee40566e497740789575275d0cdca0db already in progress: creating tablet
W20250903 20:05:21.309422 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 33fb6b357bda43e7abeaed5634183ef6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33fb6b357bda43e7abeaed5634183ef6 already in progress: creating tablet
W20250903 20:05:21.309597 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 60c32370a5274347afafc6a7e1e94270 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60c32370a5274347afafc6a7e1e94270 already in progress: creating tablet
W20250903 20:05:21.309772 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet e0347eab6c8642beabdc2d7e7bbfdc9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0347eab6c8642beabdc2d7e7bbfdc9b already in progress: creating tablet
W20250903 20:05:21.309938 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 8a223f75045345ae833f52bcc8f239f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a223f75045345ae833f52bcc8f239f9 already in progress: creating tablet
W20250903 20:05:21.310138 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet a3103e1595ec4c38ba65dca26fe1167c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3103e1595ec4c38ba65dca26fe1167c already in progress: creating tablet
I20250903 20:05:21.310636 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet a72b85dfb34047528b2b28ff0d40f828 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:21.310997 28245 raft_consensus.cc:513] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:21.312533 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 443) took 3783 ms. Trace:
I20250903 20:05:21.312676 27685 rpcz_store.cc:276] 0903 20:05:17.526964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.018699 (+491735us) service_pool.cc:225] Handling call
0903 20:05:21.306146 (+3287447us) ts_tablet_manager.cc:1145] Deleting tablet 6f8cfe53b91b41d7ba7f9074e1005af3
0903 20:05:21.306152 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.306208 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3224576}
W20250903 20:05:21.315214 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 6f8cfe53b91b41d7ba7f9074e1005af3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f8cfe53b91b41d7ba7f9074e1005af3 already in progress: creating tablet
I20250903 20:05:21.317857 28245 leader_election.cc:290] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:21.317996 28245 raft_consensus.cc:491] T ac1b0dc95fca49b4af6697c0a70e9730 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:21.318101 28245 raft_consensus.cc:513] T ac1b0dc95fca49b4af6697c0a70e9730 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:21.324092 27184 leader_election.cc:343] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 2a5c74a281b44019872a6442d41bc373
I20250903 20:05:21.324916 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 445) took 3775 ms. Trace:
I20250903 20:05:21.325034 27685 rpcz_store.cc:276] 0903 20:05:17.549123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.043298 (+494175us) service_pool.cc:225] Handling call
0903 20:05:21.314818 (+3271520us) ts_tablet_manager.cc:1145] Deleting tablet ed4902c002ef4198802375ced39ef2ad
0903 20:05:21.324776 (+ 9958us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.324894 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3216971}
I20250903 20:05:21.326573 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 446) took 3770 ms. Trace:
I20250903 20:05:21.326660 27685 rpcz_store.cc:276] 0903 20:05:17.555872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.048778 (+492906us) service_pool.cc:225] Handling call
0903 20:05:21.326461 (+3277683us) ts_tablet_manager.cc:1145] Deleting tablet e877d56bbbfa4c5a92d025d0311ce5c6
0903 20:05:21.326472 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.326556 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3221058}
I20250903 20:05:21.327936 27685 tablet_replica.cc:331] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:21.328348 27685 raft_consensus.cc:2241] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:21.325963 28245 leader_election.cc:290] T ac1b0dc95fca49b4af6697c0a70e9730 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:21.329107 27685 raft_consensus.cc:2270] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:21.330075 27685 ts_tablet_manager.cc:1905] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:21.335167 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet ed4902c002ef4198802375ced39ef2ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed4902c002ef4198802375ced39ef2ad already in progress: creating tablet
I20250903 20:05:21.337673 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d0b3804ced547f3bd482399cbaa9270. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:21.339712 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet e877d56bbbfa4c5a92d025d0311ce5c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e877d56bbbfa4c5a92d025d0311ce5c6 already in progress: creating tablet
I20250903 20:05:21.350178 27695 tablet_bootstrap.cc:492] T f71d3dd9af2543299296f21381d1d563 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:21.351047 27695 ts_tablet_manager.cc:1397] T f71d3dd9af2543299296f21381d1d563 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.556s user 0.027s sys 0.001s
I20250903 20:05:21.351418 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c49767d618be4fd18f30e5124ce0cb84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.361436 27695 raft_consensus.cc:357] T f71d3dd9af2543299296f21381d1d563 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:21.368005 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac1b0dc95fca49b4af6697c0a70e9730" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:21.368420 27362 tablet_service.cc:1468] Processing CreateTablet for tablet 9767293447214a7eb68ed03d7373db7f (DEFAULT_TABLE table=default.Table_7b5eb542da2845c79bd273f704aa9159 [id=2289e81b474c4b0190dea612c7d58de0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.369063 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c772ab7c50c4994aacde5a05bb0f08e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.370182 27058 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet e8bc99de4ffe43d58351d170b7c61d8a (table default.Table_fd3053c3354a4359b2742e9350d19d23 [id=416ff29f66d14564826a6734dde1c604]). Expected version 3 got 0
I20250903 20:05:21.371096 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_50be0ce79d0944c0b752412411a5981a" } modify_external_catalogs: true
I20250903 20:05:21.371306 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_50be0ce79d0944c0b752412411a5981a" } modify_external_catalogs: true
I20250903 20:05:21.373603 27693 raft_consensus.cc:357] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:21.374064 27693 raft_consensus.cc:383] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:21.375099 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4347c764cf54f9faaa13a2001ff638e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.374078 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac1b0dc95fca49b4af6697c0a70e9730" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:21.376477 27693 raft_consensus.cc:738] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:21.377213 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9767293447214a7eb68ed03d7373db7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.377632 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_455b30dcbf624a178417e64410fbb531"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:21.378305 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_455b30dcbf624a178417e64410fbb531 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:21.376892 27693 consensus_queue.cc:260] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:21.363631 27183 leader_election.cc:343] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 2a5c74a281b44019872a6442d41bc373
I20250903 20:05:21.383105 27695 raft_consensus.cc:383] T f71d3dd9af2543299296f21381d1d563 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:21.383186 27695 raft_consensus.cc:738] T f71d3dd9af2543299296f21381d1d563 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:21.383155 27183 leader_election.cc:304] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:21.383373 27183 raft_consensus.cc:2684] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250903 20:05:21.383394 27695 consensus_queue.cc:260] T f71d3dd9af2543299296f21381d1d563 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:21.376821 27184 leader_election.cc:343] T ac1b0dc95fca49b4af6697c0a70e9730 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:21.379676 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9eb24dd5a6243c49a3389320a3f78a0" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:21.384404 27249 raft_consensus.cc:2391] T e9eb24dd5a6243c49a3389320a3f78a0 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 51578fb7bceb4e119b3ab1e58c4f5528 in current term 1: Already voted for candidate e72ef1dbcc5143cf9f04a808f101c80f in this term.
I20250903 20:05:21.384740 27227 tablet_service.cc:1468] Processing CreateTablet for tablet b32c82c3d8c4420cb6e3a95129550076 (DEFAULT_TABLE table=default.Table_5bf171fc5f3d4fe7b00fb86fd3e60d1e [id=9ac920c53445437da386f79e6d0f87e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.385141 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b32c82c3d8c4420cb6e3a95129550076. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.385215 27693 ts_tablet_manager.cc:1428] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.077s user 0.001s sys 0.005s
I20250903 20:05:21.385660 27693 tablet_bootstrap.cc:492] T 5fa7cc0c03234223af382982ca737e27 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
W20250903 20:05:21.386129 27320 leader_election.cc:343] T e9eb24dd5a6243c49a3389320a3f78a0 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: Tablet not RUNNING: STOPPED
I20250903 20:05:21.386210 27695 ts_tablet_manager.cc:1428] T f71d3dd9af2543299296f21381d1d563 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.025s user 0.000s sys 0.001s
W20250903 20:05:21.387893 27320 consensus_peers.cc:489] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5bc3b9771a0042fd87e706204dcbb712. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:21.388187 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_8c666d5ff1be4e47b55808f90d025198"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:21.388836 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c666d5ff1be4e47b55808f90d025198 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:21.389220 27183 leader_election.cc:343] T ac1b0dc95fca49b4af6697c0a70e9730 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:21.389640 27183 leader_election.cc:304] T ac1b0dc95fca49b4af6697c0a70e9730 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:21.390179 27695 tablet_bootstrap.cc:492] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:21.390913 28332 raft_consensus.cc:2747] T ac1b0dc95fca49b4af6697c0a70e9730 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:21.377012 27321 leader_election.cc:304] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:21.391609 27695 tablet_bootstrap.cc:654] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:21.380350 27245 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c543ac4df33429b9787c6d32bf1ebeb" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:21.391798 27245 raft_consensus.cc:2466] T 4c543ac4df33429b9787c6d32bf1ebeb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:21.380409 27244 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f78a951bb024cf290116b0fe438ddb4" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:21.392035 27244 raft_consensus.cc:3058] T 3f78a951bb024cf290116b0fe438ddb4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:21.392858 27059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d859c0222c3647dab3dddd105e4cf0fc consensus state has no leader
W20250903 20:05:21.393671 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2059ccbd6d884085886d7d564afe03aa consensus state has no leader
I20250903 20:05:21.394662 27244 raft_consensus.cc:2466] T 3f78a951bb024cf290116b0fe438ddb4 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:21.380756 27246 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "deb84fc3465443d99c11097e53f5f575" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:21.394995 27246 raft_consensus.cc:3058] T deb84fc3465443d99c11097e53f5f575 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:21.395030 27685 ts_tablet_manager.cc:1918] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:21.395140 27685 log.cc:1199] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/2a5c74a281b44019872a6442d41bc373
I20250903 20:05:21.395512 27685 ts_tablet_manager.cc:1939] T 2a5c74a281b44019872a6442d41bc373 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:21.395864 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e73660c6b8c7464795b0571cfbd8b801"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:21.396418 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e73660c6b8c7464795b0571cfbd8b801 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:21.397702 27246 raft_consensus.cc:2466] T deb84fc3465443d99c11097e53f5f575 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:21.397719 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6b2c13e18ac747b4bb616391101d313b" } new_table_name: "default.Table_2b6469f261a04e5d9cb69bf836d97734" modify_external_catalogs: true
I20250903 20:05:21.380829 27247 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1fef9400227449ed892daff588999caa" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:21.397977 27247 raft_consensus.cc:3058] T 1fef9400227449ed892daff588999caa P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:21.400445 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f7800a262084435690aa7507a259155c" } new_table_name: "default.Table_6e9cf5ead5a24831a6a14d131dd8397e" modify_external_catalogs: true
I20250903 20:05:21.400856 27247 raft_consensus.cc:2466] T 1fef9400227449ed892daff588999caa P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:21.407150 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 9767293447214a7eb68ed03d7373db7f (DEFAULT_TABLE table=default.Table_7b5eb542da2845c79bd273f704aa9159 [id=2289e81b474c4b0190dea612c7d58de0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.409394 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17514a2af39b402abbe8263d7c1abc7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.410115 27687 ts_tablet_manager.cc:1918] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:21.410292 27687 log.cc:1199] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/ecd339ec12e84e88acec427630cc8668
I20250903 20:05:21.412906 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 453) took 3204 ms. Trace:
I20250903 20:05:21.412997 27685 rpcz_store.cc:276] 0903 20:05:18.208518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.208578 (+ 60us) service_pool.cc:225] Handling call
0903 20:05:21.327890 (+3119312us) ts_tablet_manager.cc:1145] Deleting tablet 2a5c74a281b44019872a6442d41bc373
0903 20:05:21.327900 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.333679 (+ 5779us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.362193 (+ 28514us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.412888 (+ 50695us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3105607}
I20250903 20:05:21.413317 27685 tablet_replica.cc:331] T deb84fc3465443d99c11097e53f5f575 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:21.413504 27685 raft_consensus.cc:2241] T deb84fc3465443d99c11097e53f5f575 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:21.413640 27685 raft_consensus.cc:2270] T deb84fc3465443d99c11097e53f5f575 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:21.414120 27685 ts_tablet_manager.cc:1905] T deb84fc3465443d99c11097e53f5f575 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:21.414469 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 2a5c74a281b44019872a6442d41bc373 (table default.Table_5c08431ef8e04a6aab9b96ab78615d2f [id=a9ddfed19dab46a7bb4e049483e3996f]) successfully deleted
I20250903 20:05:21.415344 28315 raft_consensus.cc:2802] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
W20250903 20:05:21.415762 27321 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 87529.1us
I20250903 20:05:21.416199 27687 ts_tablet_manager.cc:1939] T ecd339ec12e84e88acec427630cc8668 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:21.417021 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 502) took 2763 ms. Trace:
I20250903 20:05:21.390718 27364 tablet_service.cc:1468] Processing CreateTablet for tablet b32c82c3d8c4420cb6e3a95129550076 (DEFAULT_TABLE table=default.Table_5bf171fc5f3d4fe7b00fb86fd3e60d1e [id=9ac920c53445437da386f79e6d0f87e4]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:21.421365 27321 leader_election.cc:343] T 8b3121fce1c44d16a0b083313dc67b7a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 8b3121fce1c44d16a0b083313dc67b7a
I20250903 20:05:21.421515 27321 leader_election.cc:304] T 8b3121fce1c44d16a0b083313dc67b7a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:21.422032 27321 consensus_peers.cc:489] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Couldn't send request to peer 89f8f0ee68b940db9ce250c0604f934a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5bc3b9771a0042fd87e706204dcbb712. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:21.422303 27321 leader_election.cc:304] T e9eb24dd5a6243c49a3389320a3f78a0 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:21.422713 27321 leader_election.cc:304] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:21.423207 27321 leader_election.cc:304] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:21.423561 27321 leader_election.cc:304] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:21.423857 28237 raft_consensus.cc:2747] T 8b3121fce1c44d16a0b083313dc67b7a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:21.424037 28237 raft_consensus.cc:3058] T e9eb24dd5a6243c49a3389320a3f78a0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:21.426343 28297 raft_consensus.cc:2802] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:21.426465 28297 raft_consensus.cc:491] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:21.426553 28297 raft_consensus.cc:3058] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:21.429383 28297 raft_consensus.cc:513] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:21.430078 28297 leader_election.cc:290] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:21.430349 28298 raft_consensus.cc:2802] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:21.430480 28298 raft_consensus.cc:695] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:21.430773 28298 consensus_queue.cc:237] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:21.432031 28315 raft_consensus.cc:695] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:21.432276 28326 raft_consensus.cc:491] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:21.432379 28315 consensus_queue.cc:237] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:21.433404 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b32c82c3d8c4420cb6e3a95129550076. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.441484 27692 tablet_bootstrap.cc:492] T fbfabb6ff1714d2e921f45256b543759 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:21.445104 27693 tablet_bootstrap.cc:654] T 5fa7cc0c03234223af382982ca737e27 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:21.444864 27687 rpcz_store.cc:276] 0903 20:05:18.653310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.655579 (+ 2269us) service_pool.cc:225] Handling call
0903 20:05:21.285550 (+2629971us) ts_tablet_manager.cc:1145] Deleting tablet ecd339ec12e84e88acec427630cc8668
0903 20:05:21.285710 (+ 160us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.297182 (+ 11472us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.395972 (+ 98790us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.416971 (+ 20999us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2628857}
I20250903 20:05:21.432516 28326 raft_consensus.cc:513] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:21.447116 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet ecd339ec12e84e88acec427630cc8668 (table default.Table_54bf0e4b392545b08001021008e93985 [id=a803b1a5d2404f74aa4671eebd23e8ae]) successfully deleted
I20250903 20:05:21.448378 27247 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf2545037c214bb1b5060575e3cdf81a" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:21.448673 27247 raft_consensus.cc:2466] T bf2545037c214bb1b5060575e3cdf81a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:21.452373 28237 raft_consensus.cc:2747] T e9eb24dd5a6243c49a3389320a3f78a0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:21.454236 27462 leader_election.cc:304] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:21.454952 27687 tablet_replica.cc:331] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
W20250903 20:05:21.455307 27320 leader_election.cc:343] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 4c543ac4df33429b9787c6d32bf1ebeb
I20250903 20:05:21.455480 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf2545037c214bb1b5060575e3cdf81a" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:21.458498 27493 consensus_meta.cc:287] T c059dda502ea4bb28aa8ad7c7baa60e7 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent flushing consensus metadata: real 0.515s user 0.003s sys 0.003s
I20250903 20:05:21.459776 28299 raft_consensus.cc:2802] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:21.460013 27687 raft_consensus.cc:2241] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:21.460386 27687 pending_rounds.cc:70] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528: Trying to abort 1 pending ops.
W20250903 20:05:21.460419 27462 leader_election.cc:343] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:21.461205 28299 raft_consensus.cc:491] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:21.461337 28299 raft_consensus.cc:3058] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:21.461378 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9767293447214a7eb68ed03d7373db7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.462520 27692 ts_tablet_manager.cc:1397] T fbfabb6ff1714d2e921f45256b543759 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.617s user 0.008s sys 0.011s
I20250903 20:05:21.466578 28299 raft_consensus.cc:513] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:21.467293 28299 leader_election.cc:290] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:21.472101 27687 pending_rounds.cc:77] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384951019421696 op_type: NO_OP noop_request { }
I20250903 20:05:21.472283 27687 raft_consensus.cc:2887] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:21.471309 28326 leader_election.cc:290] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:21.480223 27247 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf2545037c214bb1b5060575e3cdf81a" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:21.480479 27247 raft_consensus.cc:3058] T bf2545037c214bb1b5060575e3cdf81a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:21.484278 27247 raft_consensus.cc:2466] T bf2545037c214bb1b5060575e3cdf81a P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:21.484839 28331 raft_consensus.cc:491] T 4be9bdc390de4a74a396003c9c3c2b71 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:21.485008 28331 raft_consensus.cc:513] T 4be9bdc390de4a74a396003c9c3c2b71 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:21.485816 27685 ts_tablet_manager.cc:1918] T deb84fc3465443d99c11097e53f5f575 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:21.485917 27685 log.cc:1199] T deb84fc3465443d99c11097e53f5f575 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/deb84fc3465443d99c11097e53f5f575
I20250903 20:05:21.492259 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9bdc390de4a74a396003c9c3c2b71" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:21.493330 27687 raft_consensus.cc:2270] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:21.494057 27687 ts_tablet_manager.cc:1905] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:21.496596 27462 leader_election.cc:304] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:21.497191 28326 raft_consensus.cc:2802] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:21.497509 28297 raft_consensus.cc:2802] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:21.497967 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf2545037c214bb1b5060575e3cdf81a" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:21.498332 28326 raft_consensus.cc:695] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:21.498644 28326 consensus_queue.cc:237] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:21.500010 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9bdc390de4a74a396003c9c3c2b71" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:21.501608 28297 raft_consensus.cc:695] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:21.503396 28297 consensus_queue.cc:237] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:21.504699 27462 leader_election.cc:343] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:21.486328 28331 leader_election.cc:290] T 4be9bdc390de4a74a396003c9c3c2b71 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
W20250903 20:05:21.492909 27183 leader_election.cc:343] T 4be9bdc390de4a74a396003c9c3c2b71 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:21.528772 27184 leader_election.cc:343] T 4be9bdc390de4a74a396003c9c3c2b71 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:21.529011 27184 leader_election.cc:304] T 4be9bdc390de4a74a396003c9c3c2b71 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:21.529556 28331 raft_consensus.cc:2747] T 4be9bdc390de4a74a396003c9c3c2b71 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:21.530459 27685 ts_tablet_manager.cc:1939] T deb84fc3465443d99c11097e53f5f575 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:21.530815 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 455) took 3307 ms. Trace:
I20250903 20:05:21.530948 27685 rpcz_store.cc:276] 0903 20:05:18.222923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.222994 (+ 71us) service_pool.cc:225] Handling call
0903 20:05:21.413276 (+3190282us) ts_tablet_manager.cc:1145] Deleting tablet deb84fc3465443d99c11097e53f5f575
0903 20:05:21.413284 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.468883 (+ 55599us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.485693 (+ 16810us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.530801 (+ 45108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3181660}
I20250903 20:05:21.531541 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet deb84fc3465443d99c11097e53f5f575 (table default.Table_a642c53ef0da461ba2f33dd4010e7129 [id=95919dc5590942bd8e47047598b6d6bc]) successfully deleted
I20250903 20:05:21.531913 27685 tablet_replica.cc:331] T 1fef9400227449ed892daff588999caa P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:21.532137 27685 raft_consensus.cc:2241] T 1fef9400227449ed892daff588999caa P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:21.532459 27685 raft_consensus.cc:2270] T 1fef9400227449ed892daff588999caa P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:21.533013 27685 ts_tablet_manager.cc:1905] T 1fef9400227449ed892daff588999caa P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:21.538892 27692 raft_consensus.cc:357] T fbfabb6ff1714d2e921f45256b543759 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:21.539134 27692 raft_consensus.cc:383] T fbfabb6ff1714d2e921f45256b543759 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:21.539212 27692 raft_consensus.cc:738] T fbfabb6ff1714d2e921f45256b543759 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:21.540773 27500 tablet_service.cc:1468] Processing CreateTablet for tablet 9767293447214a7eb68ed03d7373db7f (DEFAULT_TABLE table=default.Table_7b5eb542da2845c79bd273f704aa9159 [id=2289e81b474c4b0190dea612c7d58de0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.541309 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9767293447214a7eb68ed03d7373db7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.543274 27503 tablet_service.cc:1468] Processing CreateTablet for tablet b32c82c3d8c4420cb6e3a95129550076 (DEFAULT_TABLE table=default.Table_5bf171fc5f3d4fe7b00fb86fd3e60d1e [id=9ac920c53445437da386f79e6d0f87e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.543766 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b32c82c3d8c4420cb6e3a95129550076. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.549261 27692 consensus_queue.cc:260] T fbfabb6ff1714d2e921f45256b543759 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:21.562641 27692 ts_tablet_manager.cc:1428] T fbfabb6ff1714d2e921f45256b543759 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.100s user 0.002s sys 0.000s
I20250903 20:05:21.572338 27692 tablet_bootstrap.cc:492] T 369c750ff5bf449e989d95ffd89497d4 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
W20250903 20:05:21.574679 27320 consensus_peers.cc:489] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57a1167457b84ea8a493be232b1d088a. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:21.587289 27684 ts_tablet_manager.cc:1918] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:21.589336 27684 log.cc:1199] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/e9eb24dd5a6243c49a3389320a3f78a0
I20250903 20:05:21.589771 27684 ts_tablet_manager.cc:1939] T e9eb24dd5a6243c49a3389320a3f78a0 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:21.592160 27692 tablet_bootstrap.cc:654] T 369c750ff5bf449e989d95ffd89497d4 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:21.596679 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 581) took 1342 ms. Trace:
I20250903 20:05:21.596833 27684 rpcz_store.cc:276] 0903 20:05:20.254169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.254260 (+ 91us) service_pool.cc:225] Handling call
0903 20:05:20.641099 (+386839us) ts_tablet_manager.cc:1145] Deleting tablet e9eb24dd5a6243c49a3389320a3f78a0
0903 20:05:20.641111 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.118315 (+477204us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.587100 (+468785us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.596632 (+ 9532us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3762,"tablet-delete.queue_time_us":386543}
I20250903 20:05:21.597471 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 585) took 1335 ms. Trace:
I20250903 20:05:21.597512 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet e9eb24dd5a6243c49a3389320a3f78a0 (table default.Table_091b5cecb0b04277a78dc078b4f800a6 [id=1b9cb57cf2b445098305391cc7935b69]) successfully deleted
I20250903 20:05:21.597573 27684 rpcz_store.cc:276] 0903 20:05:20.262267 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.262315 (+ 48us) service_pool.cc:225] Handling call
0903 20:05:21.597298 (+1334983us) ts_tablet_manager.cc:1145] Deleting tablet 1bf570855b204b96acd070587c434463
0903 20:05:21.597311 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.597440 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1334705}
I20250903 20:05:21.597913 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 589) took 1298 ms. Trace:
W20250903 20:05:21.598114 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 1bf570855b204b96acd070587c434463 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1bf570855b204b96acd070587c434463 already in progress: creating tablet
I20250903 20:05:21.602139 27687 ts_tablet_manager.cc:1918] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:21.602387 27687 log.cc:1199] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/a10b079e7d5a440580ac00195fe88c65
I20250903 20:05:21.598021 27684 rpcz_store.cc:276] 0903 20:05:20.299219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.302678 (+ 3459us) service_pool.cc:225] Handling call
0903 20:05:21.597783 (+1295105us) ts_tablet_manager.cc:1145] Deleting tablet 259bc73ca9df47e9874692725a60bb1c
0903 20:05:21.597792 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.597873 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1294874}
I20250903 20:05:21.602963 27687 ts_tablet_manager.cc:1939] T a10b079e7d5a440580ac00195fe88c65 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
W20250903 20:05:21.603061 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 259bc73ca9df47e9874692725a60bb1c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 259bc73ca9df47e9874692725a60bb1c already in progress: creating tablet
I20250903 20:05:21.603736 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 597) took 1165 ms. Trace:
I20250903 20:05:21.603902 27684 rpcz_store.cc:276] 0903 20:05:20.456100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.456148 (+ 48us) service_pool.cc:225] Handling call
0903 20:05:21.603563 (+1147415us) ts_tablet_manager.cc:1145] Deleting tablet 847d0a0a3d2b45938cec9058c77f6231
0903 20:05:21.603577 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.603688 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1125364}
I20250903 20:05:21.604254 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 503) took 2928 ms. Trace:
W20250903 20:05:21.604467 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 847d0a0a3d2b45938cec9058c77f6231 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 847d0a0a3d2b45938cec9058c77f6231 already in progress: creating tablet
I20250903 20:05:21.604408 27687 rpcz_store.cc:276] 0903 20:05:18.676162 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.679462 (+ 3300us) service_pool.cc:225] Handling call
0903 20:05:21.446715 (+2767253us) ts_tablet_manager.cc:1145] Deleting tablet a10b079e7d5a440580ac00195fe88c65
0903 20:05:21.446732 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.522074 (+ 75342us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.551715 (+ 29641us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.604196 (+ 52481us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3758,"tablet-delete.queue_time_us":2766877}
I20250903 20:05:21.605049 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 598) took 1099 ms. Trace:
I20250903 20:05:21.605093 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet a10b079e7d5a440580ac00195fe88c65 (table default.Table_9b20469406c94986ab496ea3e678b586 [id=6d6f5e8cadf9445d8695c075cfabb2ea]) successfully deleted
I20250903 20:05:21.605229 27684 rpcz_store.cc:276] 0903 20:05:20.505161 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.505393 (+ 232us) service_pool.cc:225] Handling call
0903 20:05:21.604873 (+1099480us) ts_tablet_manager.cc:1145] Deleting tablet f2629f46704e4a0fa7922533928ca2ed
0903 20:05:21.604886 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.604999 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1096600}
I20250903 20:05:21.605649 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 504) took 2901 ms. Trace:
W20250903 20:05:21.605796 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet f2629f46704e4a0fa7922533928ca2ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2629f46704e4a0fa7922533928ca2ed already in progress: creating tablet
I20250903 20:05:21.605798 27687 rpcz_store.cc:276] 0903 20:05:18.703994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.705584 (+ 1590us) service_pool.cc:225] Handling call
0903 20:05:21.605446 (+2899862us) ts_tablet_manager.cc:1145] Deleting tablet 20c25c84c6fb4609a1964a4ff3c526da
0903 20:05:21.605462 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.605593 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2899568}
W20250903 20:05:21.606310 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 20c25c84c6fb4609a1964a4ff3c526da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20c25c84c6fb4609a1964a4ff3c526da already in progress: creating tablet
I20250903 20:05:21.606377 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 599) took 1100 ms. Trace:
I20250903 20:05:21.606510 27684 rpcz_store.cc:276] 0903 20:05:20.505753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.511534 (+ 5781us) service_pool.cc:225] Handling call
0903 20:05:21.606176 (+1094642us) ts_tablet_manager.cc:1145] Deleting tablet 4e9a6a6e05464b7b97ae624f51402cf0
0903 20:05:21.606187 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.606345 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1087560}
I20250903 20:05:21.606817 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 505) took 2897 ms. Trace:
I20250903 20:05:21.606956 27687 rpcz_store.cc:276] 0903 20:05:18.708980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.710066 (+ 1086us) service_pool.cc:225] Handling call
0903 20:05:21.606648 (+2896582us) ts_tablet_manager.cc:1145] Deleting tablet ac9b5d4f39db4f05a15316585040d142
0903 20:05:21.606661 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.606770 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2896340}
W20250903 20:05:21.607075 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 4e9a6a6e05464b7b97ae624f51402cf0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e9a6a6e05464b7b97ae624f51402cf0 already in progress: creating tablet
W20250903 20:05:21.607411 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet ac9b5d4f39db4f05a15316585040d142 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac9b5d4f39db4f05a15316585040d142 already in progress: creating tablet
I20250903 20:05:21.580159 28339 tablet.cc:1722] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:21.607554 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 601) took 1006 ms. Trace:
I20250903 20:05:21.607714 27684 rpcz_store.cc:276] 0903 20:05:20.600711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.600815 (+ 104us) service_pool.cc:225] Handling call
0903 20:05:21.607414 (+1006599us) ts_tablet_manager.cc:1145] Deleting tablet cac38daf002c4ae698d72903a277a587
0903 20:05:21.607424 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.607517 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1005101}
W20250903 20:05:21.608322 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet cac38daf002c4ae698d72903a277a587 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cac38daf002c4ae698d72903a277a587 already in progress: creating tablet
I20250903 20:05:21.608440 27687 tablet_replica.cc:331] T 1396d4f8a93b412ca8397368fb98834e P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:21.608696 27687 raft_consensus.cc:2241] T 1396d4f8a93b412ca8397368fb98834e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:21.608939 27687 raft_consensus.cc:2270] T 1396d4f8a93b412ca8397368fb98834e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:21.609084 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 600) took 1008 ms. Trace:
I20250903 20:05:21.609248 27684 rpcz_store.cc:276] 0903 20:05:20.600354 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.614627 (+ 14273us) service_pool.cc:225] Handling call
0903 20:05:21.608875 (+994248us) ts_tablet_manager.cc:1145] Deleting tablet 200c06f296b94eab94f35f312c5a443b
0903 20:05:21.608885 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.608976 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":973202}
I20250903 20:05:21.609738 27687 ts_tablet_manager.cc:1905] T 1396d4f8a93b412ca8397368fb98834e P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:21.609782 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 200c06f296b94eab94f35f312c5a443b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 200c06f296b94eab94f35f312c5a443b already in progress: creating tablet
I20250903 20:05:21.610186 27684 tablet_replica.cc:331] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:21.617311 27687 ts_tablet_manager.cc:1918] T 1396d4f8a93b412ca8397368fb98834e P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:21.617509 27687 log.cc:1199] T 1396d4f8a93b412ca8397368fb98834e P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/1396d4f8a93b412ca8397368fb98834e
I20250903 20:05:21.617969 27687 ts_tablet_manager.cc:1939] T 1396d4f8a93b412ca8397368fb98834e P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
W20250903 20:05:21.620218 27053 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet f204f3ead373415daf7c3e765dd8f59e (table default.Table_39c7691206fc4227adff3a4a904a2ccb [id=f46ac8414a904fc2b8eb2a8d1f6bfb5b], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f204f3ead373415daf7c3e765dd8f59e consensus state has no leader
I20250903 20:05:21.626863 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 508) took 2876 ms. Trace:
I20250903 20:05:21.628945 27687 rpcz_store.cc:276] 0903 20:05:18.749873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.749928 (+ 55us) service_pool.cc:225] Handling call
0903 20:05:21.608350 (+2858422us) ts_tablet_manager.cc:1145] Deleting tablet 1396d4f8a93b412ca8397368fb98834e
0903 20:05:21.608363 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.613472 (+ 5109us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.617168 (+ 3696us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.626781 (+ 9613us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2858179}
I20250903 20:05:21.630155 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 1396d4f8a93b412ca8397368fb98834e (table default.Table_35a3167eda1a45949f8cd8a281a88733 [id=da438a4eb783471ea0b09fe1117ec822]) successfully deleted
I20250903 20:05:21.630852 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 509) took 2855 ms. Trace:
I20250903 20:05:21.631754 27687 rpcz_store.cc:276] 0903 20:05:18.775776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.776324 (+ 548us) service_pool.cc:225] Handling call
0903 20:05:21.630648 (+2854324us) ts_tablet_manager.cc:1145] Deleting tablet 0fef9aa7153a4902abce522e3ff0df3f
0903 20:05:21.630659 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.630791 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2854044}
W20250903 20:05:21.633132 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 0fef9aa7153a4902abce522e3ff0df3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0fef9aa7153a4902abce522e3ff0df3f already in progress: creating tablet
I20250903 20:05:21.634022 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 510) took 2856 ms. Trace:
I20250903 20:05:21.636145 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 41056c5c5003415993bee74d7621bd14
I20250903 20:05:21.636417 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 2c772ab7c50c4994aacde5a05bb0f08e
I20250903 20:05:21.637709 27684 raft_consensus.cc:2241] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:21.638110 27062 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 6238740ce1894e899be9a40321f0ab73 (table default.Table_0fef1d019eff4323b7ba56368614aadc [id=53074cd97dec40c7b0e1444183d6707a]). Expected version 2 got 0
I20250903 20:05:21.638185 27684 pending_rounds.cc:70] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f: Trying to abort 1 pending ops.
I20250903 20:05:21.638290 27684 pending_rounds.cc:77] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384958460719104 op_type: NO_OP noop_request { }
I20250903 20:05:21.638425 27684 raft_consensus.cc:2887] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:21.638458 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet a253e3399622409e92299069b19d3698
I20250903 20:05:21.638556 27684 raft_consensus.cc:2270] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:21.639098 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e036e82874544ec9bad0f3df65db6cab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:21.639694 27498 tablet_service.cc:1515] Processing DeleteTablet for tablet a253e3399622409e92299069b19d3698 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:21.639724 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e036e82874544ec9bad0f3df65db6cab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:21.641286 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet b7a6d5d8a97a4769a81ec7224d591f8c
I20250903 20:05:21.641852 27066 catalog_manager.cc:6652] ReplaceTablet: tablet b7a6d5d8a97a4769a81ec7224d591f8c deleted and replaced by tablet 79c0cc0c30ab42eab0be415e3db9589f
I20250903 20:05:21.642297 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7e582ad4ca9446c68da4850af02126ea" } modify_external_catalogs: true
I20250903 20:05:21.642488 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7e582ad4ca9446c68da4850af02126ea" } modify_external_catalogs: true
I20250903 20:05:21.642545 27498 tablet_service.cc:1515] Processing DeleteTablet for tablet b7a6d5d8a97a4769a81ec7224d591f8c with delete_type TABLET_DATA_DELETED (replaced by tablet 79c0cc0c30ab42eab0be415e3db9589f) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:21.644320 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_072fbb13945d42d88062f1b63efb5b6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:21.644860 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_072fbb13945d42d88062f1b63efb5b6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:21.645921 27062 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet e8bc99de4ffe43d58351d170b7c61d8a (table default.Table_fd3053c3354a4359b2742e9350d19d23 [id=416ff29f66d14564826a6734dde1c604]). Expected version 3 got 0
I20250903 20:05:21.646073 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet 41056c5c5003415993bee74d7621bd14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:21.646183 27685 ts_tablet_manager.cc:1918] T 1fef9400227449ed892daff588999caa P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:21.646313 27685 log.cc:1199] T 1fef9400227449ed892daff588999caa P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/1fef9400227449ed892daff588999caa
I20250903 20:05:21.646579 27685 ts_tablet_manager.cc:1939] T 1fef9400227449ed892daff588999caa P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:21.646871 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 454) took 3428 ms. Trace:
I20250903 20:05:21.646790 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ad72e251621b4a718e423879c1f34203"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:21.646944 27685 rpcz_store.cc:276] 0903 20:05:18.218040 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.218107 (+ 67us) service_pool.cc:225] Handling call
0903 20:05:21.531866 (+3313759us) ts_tablet_manager.cc:1145] Deleting tablet 1fef9400227449ed892daff588999caa
0903 20:05:21.531875 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.561117 (+ 29242us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.646066 (+ 84949us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.646854 (+ 788us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3293086}
I20250903 20:05:21.647114 27496 tablet_service.cc:1515] Processing DeleteTablet for tablet 2c772ab7c50c4994aacde5a05bb0f08e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:21.647331 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad72e251621b4a718e423879c1f34203 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:21.647418 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 1fef9400227449ed892daff588999caa (table default.Table_d3945d5a36d84041b35394333dc704f9 [id=5568f788d1414c83b4359b5adf28632a]) successfully deleted
I20250903 20:05:21.647928 28352 tablet.cc:1722] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250903 20:05:21.650280 27227 tablet_service.cc:1515] Processing DeleteTablet for tablet 41056c5c5003415993bee74d7621bd14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:21.650487 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet a253e3399622409e92299069b19d3698 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:21.650678 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet b7a6d5d8a97a4769a81ec7224d591f8c with delete_type TABLET_DATA_DELETED (replaced by tablet 79c0cc0c30ab42eab0be415e3db9589f) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:21.650708 28358 tablet.cc:1722] T e8bc99de4ffe43d58351d170b7c61d8a P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:21.639750 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 5b26ed39f2144610b786f49e62e6f0a3
I20250903 20:05:21.652725 27496 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b26ed39f2144610b786f49e62e6f0a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:21.652913 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b26ed39f2144610b786f49e62e6f0a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:21.654906 27058 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet e8bc99de4ffe43d58351d170b7c61d8a (table default.Table_fd3053c3354a4359b2742e9350d19d23 [id=416ff29f66d14564826a6734dde1c604]). Expected version 3 got 2
I20250903 20:05:21.656569 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6cc84d6d6ac64e75ab04f276ffbd6c79" } new_table_name: "default.Table_051d82784c114e9f96c5717143bbc65d" modify_external_catalogs: true
W20250903 20:05:21.636132 27321 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 212590us
I20250903 20:05:21.658445 27687 rpcz_store.cc:276] 0903 20:05:18.777261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.777318 (+ 57us) service_pool.cc:225] Handling call
0903 20:05:21.633838 (+2856520us) ts_tablet_manager.cc:1145] Deleting tablet 1223743918814d88b401782ca4b521dd
0903 20:05:21.633849 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.633947 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2856288}
I20250903 20:05:21.658890 27321 leader_election.cc:304] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:21.658986 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 1223743918814d88b401782ca4b521dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1223743918814d88b401782ca4b521dd already in progress: creating tablet
I20250903 20:05:21.659507 27693 tablet_bootstrap.cc:492] T 5fa7cc0c03234223af382982ca737e27 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:21.659684 27693 ts_tablet_manager.cc:1397] T 5fa7cc0c03234223af382982ca737e27 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.274s user 0.005s sys 0.009s
I20250903 20:05:21.660169 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 511) took 2872 ms. Trace:
I20250903 20:05:21.660281 27687 rpcz_store.cc:276] 0903 20:05:18.787974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.788034 (+ 60us) service_pool.cc:225] Handling call
0903 20:05:21.659936 (+2871902us) ts_tablet_manager.cc:1145] Deleting tablet 6f0cba70240f4f38a4fa5ccbcd99b3b1
0903 20:05:21.659947 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.660112 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2871680}
I20250903 20:05:21.660364 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c543ac4df33429b9787c6d32bf1ebeb" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:21.660554 27248 raft_consensus.cc:3058] T 4c543ac4df33429b9787c6d32bf1ebeb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:21.660670 27693 raft_consensus.cc:357] T 5fa7cc0c03234223af382982ca737e27 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:21.660763 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 6f0cba70240f4f38a4fa5ccbcd99b3b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f0cba70240f4f38a4fa5ccbcd99b3b1 already in progress: creating tablet
I20250903 20:05:21.660866 27693 raft_consensus.cc:383] T 5fa7cc0c03234223af382982ca737e27 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:21.660933 27693 raft_consensus.cc:738] T 5fa7cc0c03234223af382982ca737e27 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:21.661231 27693 consensus_queue.cc:260] T 5fa7cc0c03234223af382982ca737e27 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:21.661945 27693 ts_tablet_manager.cc:1428] T 5fa7cc0c03234223af382982ca737e27 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:21.662307 27693 tablet_bootstrap.cc:492] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:21.662488 27687 tablet_replica.cc:331] T 6ea37668d64d4feeb92a162711609c13 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:21.662688 27687 raft_consensus.cc:2241] T 6ea37668d64d4feeb92a162711609c13 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:21.662868 27687 raft_consensus.cc:2270] T 6ea37668d64d4feeb92a162711609c13 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:21.663407 27687 ts_tablet_manager.cc:1905] T 6ea37668d64d4feeb92a162711609c13 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:21.663436 27248 raft_consensus.cc:2466] T 4c543ac4df33429b9787c6d32bf1ebeb P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:21.665480 27693 tablet_bootstrap.cc:654] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:21.666389 27321 leader_election.cc:304] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:21.667001 28298 raft_consensus.cc:2802] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:21.667152 28298 raft_consensus.cc:695] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:21.667400 28298 consensus_queue.cc:237] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:21.668674 28237 raft_consensus.cc:2802] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:21.668812 28237 raft_consensus.cc:695] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:21.669126 28237 consensus_queue.cc:237] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:21.670009 28208 consensus_queue.cc:1035] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:21.670583 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 2c772ab7c50c4994aacde5a05bb0f08e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:21.671380 27695 tablet_bootstrap.cc:492] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:21.671545 27695 ts_tablet_manager.cc:1397] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.282s user 0.005s sys 0.000s
I20250903 20:05:21.672421 27695 raft_consensus.cc:357] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:21.672622 27695 raft_consensus.cc:383] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:21.672688 27695 raft_consensus.cc:738] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:21.672907 27695 consensus_queue.cc:260] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:21.673789 27695 ts_tablet_manager.cc:1428] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:21.674121 27695 tablet_bootstrap.cc:492] T 516830fc42e84af693b3af52caee30ee P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:21.642668 28339 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:54902 (request call id 579) took 1210 ms. Trace:
I20250903 20:05:21.644492 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet 41056c5c5003415993bee74d7621bd14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:21.644904 27358 tablet_service.cc:1515] Processing DeleteTablet for tablet 2c772ab7c50c4994aacde5a05bb0f08e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:21.645058 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet a253e3399622409e92299069b19d3698 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:21.649550 27348 tablet_service.cc:1515] Processing DeleteTablet for tablet b7a6d5d8a97a4769a81ec7224d591f8c with delete_type TABLET_DATA_DELETED (replaced by tablet 79c0cc0c30ab42eab0be415e3db9589f) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:21.652853 27351 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b26ed39f2144610b786f49e62e6f0a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:21.680537 28360 tablet.cc:1722] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250903 20:05:21.680071 28339 rpcz_store.cc:276] 0903 20:05:20.431998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.432760 (+ 762us) service_pool.cc:225] Handling call
0903 20:05:21.642632 (+1209872us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:20.650463 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0903 20:05:20.650678 (+ 215us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0903 20:05:20.650687 (+ 9us) alter_schema_op.cc:66] Acquired schema lock
0903 20:05:20.650695 (+ 8us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0903 20:05:20.650845 (+ 150us) alter_schema_op.cc:128] START. Timestamp: 1756929920650078
0903 20:05:20.650848 (+ 3us) op_driver.cc:348] REPLICATION: starting
0903 20:05:20.651197 (+ 349us) log.cc:844] Serialized 173 byte log entry
0903 20:05:21.418493 (+767296us) op_driver.cc:464] REPLICATION: finished
0903 20:05:21.577243 (+158750us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0903 20:05:21.608217 (+ 30974us) log.cc:844] Serialized 5 byte log entry
0903 20:05:21.627233 (+ 19016us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.642334 (+ 15101us) log.cc:844] Serialized 14 byte log entry
0903 20:05:21.642589 (+ 255us) alter_schema_op.cc:73] Released schema lock
0903 20:05:21.642595 (+ 6us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":158557,"prepare.queue_time_us":217237,"prepare.run_cpu_time_us":3860,"prepare.run_wall_time_us":60270,"replication_time_us":767610,"thread_start_us":431385,"threads_started":3,"wal-append.queue_time_us":58770}]]}
I20250903 20:05:21.681404 28339 tablet.cc:1722] T c8a293c6fa034e9fb29682439cb3bcc2 P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250903 20:05:21.687800 28364 tablet.cc:1722] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250903 20:05:21.691893 27692 tablet_bootstrap.cc:492] T 369c750ff5bf449e989d95ffd89497d4 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:21.692080 27692 ts_tablet_manager.cc:1397] T 369c750ff5bf449e989d95ffd89497d4 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.120s user 0.002s sys 0.002s
I20250903 20:05:21.692836 27692 raft_consensus.cc:357] T 369c750ff5bf449e989d95ffd89497d4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:21.692957 27060 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet e8bc99de4ffe43d58351d170b7c61d8a (table default.Table_fd3053c3354a4359b2742e9350d19d23 [id=416ff29f66d14564826a6734dde1c604]). Expected version 3 got 0
I20250903 20:05:21.693034 27692 raft_consensus.cc:383] T 369c750ff5bf449e989d95ffd89497d4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:21.693122 27692 raft_consensus.cc:738] T 369c750ff5bf449e989d95ffd89497d4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:21.693370 27692 consensus_queue.cc:260] T 369c750ff5bf449e989d95ffd89497d4 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:21.693827 27692 ts_tablet_manager.cc:1428] T 369c750ff5bf449e989d95ffd89497d4 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:21.694963 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_17ef4b4f7c1845088dac85dd6d8f0c0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:21.695763 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17ef4b4f7c1845088dac85dd6d8f0c0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:21.695832 27496 tablet_service.cc:1515] Processing DeleteTablet for tablet fbfabb6ff1714d2e921f45256b543759 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:21.696260 27692 tablet_bootstrap.cc:492] T 07565ea2b5f34dcfb407ff3bfde17243 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
W20250903 20:05:21.697543 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 53881ad30bc843839008ace4012a3f70 consensus state has no leader
I20250903 20:05:21.698024 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_60a3bd90021349e694df58967ede5cf5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:21.698815 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60a3bd90021349e694df58967ede5cf5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:21.699513 27684 ts_tablet_manager.cc:1905] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:21.700539 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0f5b87c1c4a24f8abdbfd6194f3b2c4a consensus state has no leader
I20250903 20:05:21.701009 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_35f8f4785b9f4c459653cff84dfcfed8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:21.701675 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35f8f4785b9f4c459653cff84dfcfed8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:21.715629 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_affbe84913504de69e8d84b2e2d14cdc" } new_table_name: "default.Table_ce7f1e2809ba4ae097811cd75676be66" modify_external_catalogs: true
I20250903 20:05:21.725188 27692 tablet_bootstrap.cc:654] T 07565ea2b5f34dcfb407ff3bfde17243 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:21.708092 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 460) took 3449 ms. Trace:
I20250903 20:05:21.741020 27685 rpcz_store.cc:276] 0903 20:05:18.259049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.259457 (+ 408us) service_pool.cc:225] Handling call
0903 20:05:21.706976 (+3447519us) ts_tablet_manager.cc:1145] Deleting tablet 6d13884fff8c43b690836c6a341a2d92
0903 20:05:21.706989 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.708066 (+ 1077us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3446418}
I20250903 20:05:21.741163 28358 tablet.cc:1722] T e8bc99de4ffe43d58351d170b7c61d8a P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250903 20:05:21.741757 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 6d13884fff8c43b690836c6a341a2d92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d13884fff8c43b690836c6a341a2d92 already in progress: creating tablet
I20250903 20:05:21.742697 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 464) took 3481 ms. Trace:
I20250903 20:05:21.742851 27685 rpcz_store.cc:276] 0903 20:05:18.261621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.261691 (+ 70us) service_pool.cc:225] Handling call
0903 20:05:21.742541 (+3480850us) ts_tablet_manager.cc:1145] Deleting tablet f9709dfd18654c5bb071e39da7c8889f
0903 20:05:21.742561 (+ 20us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.742676 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3477550}
W20250903 20:05:21.743449 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet f9709dfd18654c5bb071e39da7c8889f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9709dfd18654c5bb071e39da7c8889f already in progress: creating tablet
I20250903 20:05:21.743880 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 462) took 3484 ms. Trace:
I20250903 20:05:21.742624 28356 tablet.cc:1722] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:21.743999 27685 rpcz_store.cc:276] 0903 20:05:18.259365 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.259580 (+ 215us) service_pool.cc:225] Handling call
0903 20:05:21.743759 (+3484179us) ts_tablet_manager.cc:1145] Deleting tablet 541278ca9ffb4e41b341139c2faa337f
0903 20:05:21.743769 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.743861 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3473176}
W20250903 20:05:21.744515 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 541278ca9ffb4e41b341139c2faa337f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 541278ca9ffb4e41b341139c2faa337f already in progress: creating tablet
I20250903 20:05:21.744889 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 463) took 3485 ms. Trace:
I20250903 20:05:21.745000 27685 rpcz_store.cc:276] 0903 20:05:18.259465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.259755 (+ 290us) service_pool.cc:225] Handling call
0903 20:05:21.744788 (+3485033us) ts_tablet_manager.cc:1145] Deleting tablet b84f487773c24735a61d0fd0898b2bcb
0903 20:05:21.744797 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.744876 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3472812}
W20250903 20:05:21.745496 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet b84f487773c24735a61d0fd0898b2bcb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b84f487773c24735a61d0fd0898b2bcb already in progress: creating tablet
I20250903 20:05:21.745787 27685 tablet_replica.cc:331] T ecd339ec12e84e88acec427630cc8668 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:21.746001 27685 raft_consensus.cc:2241] T ecd339ec12e84e88acec427630cc8668 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:21.746429 27685 pending_rounds.cc:70] T ecd339ec12e84e88acec427630cc8668 P 89f8f0ee68b940db9ce250c0604f934a: Trying to abort 1 pending ops.
I20250903 20:05:21.746526 27685 pending_rounds.cc:77] T ecd339ec12e84e88acec427630cc8668 P 89f8f0ee68b940db9ce250c0604f934a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384951836143616 op_type: NO_OP noop_request { }
I20250903 20:05:21.746685 27685 raft_consensus.cc:2887] T ecd339ec12e84e88acec427630cc8668 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:21.746825 27685 raft_consensus.cc:2270] T ecd339ec12e84e88acec427630cc8668 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:21.747877 27685 ts_tablet_manager.cc:1905] T ecd339ec12e84e88acec427630cc8668 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:21.751287 27051 consensus_peers.cc:489] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 -> Peer 35dd69f3fa7a4fd8843b97f8a5d68296 (127.26.65.125:44059): Couldn't send request to peer 35dd69f3fa7a4fd8843b97f8a5d68296. Status: Network error: Client connection negotiation failed: client connection to 127.26.65.125:44059: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250903 20:05:21.760871 28360 tablet.cc:1722] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250903 20:05:21.777455 27687 ts_tablet_manager.cc:1918] T 6ea37668d64d4feeb92a162711609c13 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:21.777617 27687 log.cc:1199] T 6ea37668d64d4feeb92a162711609c13 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/6ea37668d64d4feeb92a162711609c13
I20250903 20:05:21.779196 27687 ts_tablet_manager.cc:1939] T 6ea37668d64d4feeb92a162711609c13 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:21.780202 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 507) took 3035 ms. Trace:
I20250903 20:05:21.780509 27687 rpcz_store.cc:276] 0903 20:05:18.744982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.794990 (+ 50008us) service_pool.cc:225] Handling call
0903 20:05:21.662390 (+2867400us) ts_tablet_manager.cc:1145] Deleting tablet 6ea37668d64d4feeb92a162711609c13
0903 20:05:21.662399 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.678271 (+ 15872us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.776165 (+ 97894us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.780161 (+ 3996us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2867171}
I20250903 20:05:21.781031 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 512) took 2977 ms. Trace:
I20250903 20:05:21.781378 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 6ea37668d64d4feeb92a162711609c13 (table default.Table_cc799e62ff6b47e1af28b7006551bbf5 [id=d97016a400e34272830add1e220c65ae]) successfully deleted
I20250903 20:05:21.781365 27687 rpcz_store.cc:276] 0903 20:05:18.803678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.804069 (+ 391us) service_pool.cc:225] Handling call
0903 20:05:21.780844 (+2976775us) ts_tablet_manager.cc:1145] Deleting tablet 512908bc4d6941ec81b145ee5795405f
0903 20:05:21.780857 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.780983 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2976429}
I20250903 20:05:21.781684 27687 tablet_replica.cc:331] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
W20250903 20:05:21.782200 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 512908bc4d6941ec81b145ee5795405f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 512908bc4d6941ec81b145ee5795405f already in progress: creating tablet
I20250903 20:05:21.783339 28245 raft_consensus.cc:491] T f71d3dd9af2543299296f21381d1d563 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:21.783494 28245 raft_consensus.cc:513] T f71d3dd9af2543299296f21381d1d563 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:21.784173 28245 leader_election.cc:290] T f71d3dd9af2543299296f21381d1d563 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:21.785310 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f71d3dd9af2543299296f21381d1d563" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:21.785892 27183 leader_election.cc:343] T f71d3dd9af2543299296f21381d1d563 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:21.786731 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f71d3dd9af2543299296f21381d1d563" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:21.787423 27184 leader_election.cc:343] T f71d3dd9af2543299296f21381d1d563 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:21.787568 27184 leader_election.cc:304] T f71d3dd9af2543299296f21381d1d563 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:21.788234 28245 raft_consensus.cc:2747] T f71d3dd9af2543299296f21381d1d563 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:21.786736 27687 raft_consensus.cc:2241] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:21.796227 27685 ts_tablet_manager.cc:1918] T ecd339ec12e84e88acec427630cc8668 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:21.806473 27685 log.cc:1199] T ecd339ec12e84e88acec427630cc8668 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/ecd339ec12e84e88acec427630cc8668
I20250903 20:05:21.806926 27685 ts_tablet_manager.cc:1939] T ecd339ec12e84e88acec427630cc8668 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:21.808007 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 465) took 3543 ms. Trace:
I20250903 20:05:21.808140 27685 rpcz_store.cc:276] 0903 20:05:18.264779 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.265951 (+ 1172us) service_pool.cc:225] Handling call
0903 20:05:21.745743 (+3479792us) ts_tablet_manager.cc:1145] Deleting tablet ecd339ec12e84e88acec427630cc8668
0903 20:05:21.745751 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.781093 (+ 35342us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.796096 (+ 15003us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.807978 (+ 11882us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3467218}
I20250903 20:05:21.812820 27693 tablet_bootstrap.cc:492] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:21.813076 27693 ts_tablet_manager.cc:1397] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.151s user 0.004s sys 0.000s
I20250903 20:05:21.815939 27684 ts_tablet_manager.cc:1918] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:21.816076 27687 pending_rounds.cc:70] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528: Trying to abort 1 pending ops.
I20250903 20:05:21.816253 27684 log.cc:1199] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/bf2545037c214bb1b5060575e3cdf81a
I20250903 20:05:21.816212 27687 pending_rounds.cc:77] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384951893729280 op_type: NO_OP noop_request { }
I20250903 20:05:21.816351 27687 raft_consensus.cc:2887] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:21.816491 27687 raft_consensus.cc:2270] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:21.817065 27684 ts_tablet_manager.cc:1939] T bf2545037c214bb1b5060575e3cdf81a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:21.817096 27687 ts_tablet_manager.cc:1905] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:21.818684 27693 raft_consensus.cc:357] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:21.819070 27693 raft_consensus.cc:383] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:21.819249 27693 raft_consensus.cc:738] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:21.819672 27693 consensus_queue.cc:260] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:21.822165 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 605) took 1185 ms. Trace:
I20250903 20:05:21.822402 27684 rpcz_store.cc:276] 0903 20:05:20.636753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.638677 (+ 1924us) service_pool.cc:225] Handling call
0903 20:05:21.610115 (+971438us) ts_tablet_manager.cc:1145] Deleting tablet bf2545037c214bb1b5060575e3cdf81a
0903 20:05:21.610126 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.638019 (+ 27893us) spinlock_profiling.cc:243] Waited 27.2 ms on lock 0x6160000a5754. stack: 00007f71bc6f7dc6 00007f71bc6f7b80 00007f71bb6df3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f71c8181600 00007f71c9494b13 00007f71cbf772b7 00007f71cbfa9344 00007f71cbf8ada0 00007f71cbf8ab2d 0000000000a774dc 00007f71bc73e14f 00007f71bc74141b 00007f71bc741298
0903 20:05:21.703777 (+ 65758us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.815773 (+111996us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.822116 (+ 6343us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":62562176,"tablet-delete.queue_time_us":971179}
I20250903 20:05:21.823858 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet bf2545037c214bb1b5060575e3cdf81a (table default.Table_143fea9b5e3d4acbb92adc441aeede47 [id=42c554f2f532414a88b50818e2dbb458]) successfully deleted
I20250903 20:05:21.824476 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 606) took 1141 ms. Trace:
I20250903 20:05:21.828342 27062 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet e8bc99de4ffe43d58351d170b7c61d8a (table default.Table_fd3053c3354a4359b2742e9350d19d23 [id=416ff29f66d14564826a6734dde1c604]). Expected version 3 got 2
I20250903 20:05:21.829006 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet dbfdab4332234b2a93fd3c901a9cb645
I20250903 20:05:21.829573 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_35d35c0c3a6048bc9d7def161d9e16db" } new_table_name: "default.Table_28a829a9988442049d50e6ae254e6274" modify_external_catalogs: true
I20250903 20:05:21.829638 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ddf4574624894772bd7c0aa8c5e177cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:21.830202 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ddf4574624894772bd7c0aa8c5e177cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:21.832177 27057 master_service.cc:948] ReplaceTablet: received request to replace tablet 283a3afed7414a958b2dc4176f00bb01 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:21.832720 27057 master_service.cc:948] ReplaceTablet: received request to replace tablet 8605bc329627438998812ed028327f53 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:21.833209 27057 master_service.cc:948] ReplaceTablet: received request to replace tablet 17e7deb74389448db37289563b86c253 from {username='slave'} at 127.0.0.1:45936
W20250903 20:05:21.833642 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6d21f38759d743af972635f8b5b8d8d2 consensus state has no leader
I20250903 20:05:21.834198 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet ecd339ec12e84e88acec427630cc8668 (table default.Table_54bf0e4b392545b08001021008e93985 [id=a803b1a5d2404f74aa4671eebd23e8ae]) successfully deleted
I20250903 20:05:21.834654 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 461) took 3575 ms. Trace:
I20250903 20:05:21.834803 27685 rpcz_store.cc:276] 0903 20:05:18.259233 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.259550 (+ 317us) service_pool.cc:225] Handling call
0903 20:05:21.834508 (+3574958us) ts_tablet_manager.cc:1145] Deleting tablet 8218868ff40b405387d560f614427c08
0903 20:05:21.834521 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.834633 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3536564}
W20250903 20:05:21.835400 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 8218868ff40b405387d560f614427c08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8218868ff40b405387d560f614427c08 already in progress: creating tablet
I20250903 20:05:21.835812 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 475) took 3570 ms. Trace:
I20250903 20:05:21.835925 27685 rpcz_store.cc:276] 0903 20:05:18.265569 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.305537 (+ 39968us) service_pool.cc:225] Handling call
0903 20:05:21.835697 (+3530160us) ts_tablet_manager.cc:1145] Deleting tablet fa05f54ebc324b40855fa3288760e9e9
0903 20:05:21.835707 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.835795 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3517965}
W20250903 20:05:21.836400 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet fa05f54ebc324b40855fa3288760e9e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fa05f54ebc324b40855fa3288760e9e9 already in progress: creating tablet
I20250903 20:05:21.837298 27065 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet e8bc99de4ffe43d58351d170b7c61d8a (table default.Table_fd3053c3354a4359b2742e9350d19d23 [id=416ff29f66d14564826a6734dde1c604]). Expected version 3 got 0
I20250903 20:05:21.838866 27065 master_service.cc:948] ReplaceTablet: received request to replace tablet 4157633525034a019fd6a80185892cec from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:21.839102 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet f71d3dd9af2543299296f21381d1d563 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:21.839802 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 474) took 3574 ms. Trace:
I20250903 20:05:21.839913 27685 rpcz_store.cc:276] 0903 20:05:18.265493 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.304273 (+ 38780us) service_pool.cc:225] Handling call
0903 20:05:21.839680 (+3535407us) ts_tablet_manager.cc:1145] Deleting tablet e2441aa3c2c748bba5360259fb663c81
0903 20:05:21.839691 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.839784 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3512193}
W20250903 20:05:21.840428 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet e2441aa3c2c748bba5360259fb663c81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2441aa3c2c748bba5360259fb663c81 already in progress: creating tablet
I20250903 20:05:21.841341 27351 tablet_service.cc:1515] Processing DeleteTablet for tablet dbfdab4332234b2a93fd3c901a9cb645 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:21.843596 27685 tablet_replica.cc:331] T 4c543ac4df33429b9787c6d32bf1ebeb P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:21.843854 27685 raft_consensus.cc:2241] T 4c543ac4df33429b9787c6d32bf1ebeb P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:21.844406 27685 raft_consensus.cc:2270] T 4c543ac4df33429b9787c6d32bf1ebeb P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:21.845026 27685 ts_tablet_manager.cc:1905] T 4c543ac4df33429b9787c6d32bf1ebeb P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:21.845293 28298 raft_consensus.cc:491] T 5fa7cc0c03234223af382982ca737e27 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:21.845463 28298 raft_consensus.cc:513] T 5fa7cc0c03234223af382982ca737e27 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:21.846129 28298 leader_election.cc:290] T 5fa7cc0c03234223af382982ca737e27 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:21.846477 27693 ts_tablet_manager.cc:1428] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20250903 20:05:21.846719 27247 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5fa7cc0c03234223af382982ca737e27" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:21.846904 27247 raft_consensus.cc:2466] T 5fa7cc0c03234223af382982ca737e27 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:21.847775 27693 tablet_bootstrap.cc:492] T ac1b0dc95fca49b4af6697c0a70e9730 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:21.849365 27321 leader_election.cc:304] T 5fa7cc0c03234223af382982ca737e27 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:21.849844 28298 raft_consensus.cc:2802] T 5fa7cc0c03234223af382982ca737e27 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:21.850102 28298 raft_consensus.cc:491] T 5fa7cc0c03234223af382982ca737e27 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:21.850206 28298 raft_consensus.cc:3058] T 5fa7cc0c03234223af382982ca737e27 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:21.721735 27695 tablet_bootstrap.cc:654] T 516830fc42e84af693b3af52caee30ee P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:21.829892 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet dbfdab4332234b2a93fd3c901a9cb645 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:21.858630 27685 ts_tablet_manager.cc:1918] T 4c543ac4df33429b9787c6d32bf1ebeb P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:21.858844 27685 log.cc:1199] T 4c543ac4df33429b9787c6d32bf1ebeb P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/4c543ac4df33429b9787c6d32bf1ebeb
I20250903 20:05:21.859241 27685 ts_tablet_manager.cc:1939] T 4c543ac4df33429b9787c6d32bf1ebeb P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:21.859640 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 477) took 3521 ms. Trace:
I20250903 20:05:21.859879 27685 rpcz_store.cc:276] 0903 20:05:18.338485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.338549 (+ 64us) service_pool.cc:225] Handling call
0903 20:05:21.840689 (+3502140us) ts_tablet_manager.cc:1145] Deleting tablet 4c543ac4df33429b9787c6d32bf1ebeb
0903 20:05:21.843547 (+ 2858us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.854934 (+ 11387us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.858520 (+ 3586us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.859620 (+ 1100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3501899}
I20250903 20:05:21.860518 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 4c543ac4df33429b9787c6d32bf1ebeb (table default.Table_26a46b6cf89f459b99526bc640729603 [id=ec0522912bda4887b0c40e98ac12768a]) successfully deleted
I20250903 20:05:21.860729 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 476) took 3595 ms. Trace:
I20250903 20:05:21.860850 27685 rpcz_store.cc:276] 0903 20:05:18.265629 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.317789 (+ 52160us) service_pool.cc:225] Handling call
0903 20:05:21.860520 (+3542731us) ts_tablet_manager.cc:1145] Deleting tablet aa9f5e0964b348b7af184f60cd145666
0903 20:05:21.860529 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.860710 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3517855}
W20250903 20:05:21.861402 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet aa9f5e0964b348b7af184f60cd145666 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa9f5e0964b348b7af184f60cd145666 already in progress: creating tablet
I20250903 20:05:21.861464 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 482) took 3461 ms. Trace:
I20250903 20:05:21.861634 27685 rpcz_store.cc:276] 0903 20:05:18.400409 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.400778 (+ 369us) service_pool.cc:225] Handling call
0903 20:05:21.861348 (+3460570us) ts_tablet_manager.cc:1145] Deleting tablet 0ca6e2165a9d41299c1fe0bfa9475915
0903 20:05:21.861359 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.861449 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3456641}
I20250903 20:05:21.862185 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 483) took 3424 ms. Trace:
W20250903 20:05:21.862356 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 0ca6e2165a9d41299c1fe0bfa9475915 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ca6e2165a9d41299c1fe0bfa9475915 already in progress: creating tablet
I20250903 20:05:21.862342 27685 rpcz_store.cc:276] 0903 20:05:18.437457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.437514 (+ 57us) service_pool.cc:225] Handling call
0903 20:05:21.862080 (+3424566us) ts_tablet_manager.cc:1145] Deleting tablet 63a1f26fac3b4b939ebb79c6a07d8f8e
0903 20:05:21.862088 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.862168 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3413551}
I20250903 20:05:21.862844 27685 tablet_replica.cc:331] T 3f78a951bb024cf290116b0fe438ddb4 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:21.863083 27685 raft_consensus.cc:2241] T 3f78a951bb024cf290116b0fe438ddb4 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:21.863312 27685 raft_consensus.cc:2270] T 3f78a951bb024cf290116b0fe438ddb4 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:21.863952 27685 ts_tablet_manager.cc:1905] T 3f78a951bb024cf290116b0fe438ddb4 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:21.867511 28298 raft_consensus.cc:513] T 5fa7cc0c03234223af382982ca737e27 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:21.868180 27687 ts_tablet_manager.cc:1918] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:21.868336 27687 log.cc:1199] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/5bc3b9771a0042fd87e706204dcbb712
I20250903 20:05:21.868739 27687 ts_tablet_manager.cc:1939] T 5bc3b9771a0042fd87e706204dcbb712 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:21.869558 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 516) took 3055 ms. Trace:
I20250903 20:05:21.869668 27687 rpcz_store.cc:276] 0903 20:05:18.814105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.814298 (+ 193us) service_pool.cc:225] Handling call
0903 20:05:21.781612 (+2967314us) ts_tablet_manager.cc:1145] Deleting tablet 5bc3b9771a0042fd87e706204dcbb712
0903 20:05:21.781624 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.843410 (+ 61786us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.867325 (+ 23915us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.869520 (+ 2195us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25959,"tablet-delete.queue_time_us":2877755}
I20250903 20:05:21.870153 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 538) took 2828 ms. Trace:
I20250903 20:05:21.870287 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 5bc3b9771a0042fd87e706204dcbb712 (table default.Table_459903503b5148d09d3cb8f2f5519fab [id=3c69612aaf164dc899dcc252d85636f6]) successfully deleted
I20250903 20:05:21.870292 27687 rpcz_store.cc:276] 0903 20:05:19.041467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:19.201814 (+160347us) service_pool.cc:225] Handling call
0903 20:05:21.869966 (+2668152us) ts_tablet_manager.cc:1145] Deleting tablet f1c3c8e46a2b40ccb8d5bb61843cecd1
0903 20:05:21.869978 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.870120 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2654577}
I20250903 20:05:21.870663 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 547) took 2770 ms. Trace:
W20250903 20:05:21.870756 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet f1c3c8e46a2b40ccb8d5bb61843cecd1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1c3c8e46a2b40ccb8d5bb61843cecd1 already in progress: creating tablet
I20250903 20:05:21.870769 27687 rpcz_store.cc:276] 0903 20:05:19.100625 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:19.408697 (+308072us) service_pool.cc:225] Handling call
0903 20:05:21.870532 (+2461835us) ts_tablet_manager.cc:1145] Deleting tablet bf2545037c214bb1b5060575e3cdf81a
0903 20:05:21.870542 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.870631 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2461195}
I20250903 20:05:21.871133 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 548) took 2770 ms. Trace:
W20250903 20:05:21.871187 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet bf2545037c214bb1b5060575e3cdf81a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf2545037c214bb1b5060575e3cdf81a already in progress: creating tablet
I20250903 20:05:21.871237 27687 rpcz_store.cc:276] 0903 20:05:19.100717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:19.409404 (+308687us) service_pool.cc:225] Handling call
0903 20:05:21.870996 (+2461592us) ts_tablet_manager.cc:1145] Deleting tablet 8400e23873fd4be18de2998afc64a689
0903 20:05:21.871018 (+ 22us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.871103 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2461236}
I20250903 20:05:21.871572 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 549) took 2770 ms. Trace:
W20250903 20:05:21.871619 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 8400e23873fd4be18de2998afc64a689 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8400e23873fd4be18de2998afc64a689 already in progress: creating tablet
I20250903 20:05:21.871676 27687 rpcz_store.cc:276] 0903 20:05:19.100823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:19.409810 (+308987us) service_pool.cc:225] Handling call
0903 20:05:21.871455 (+2461645us) ts_tablet_manager.cc:1145] Deleting tablet 64c4dd2645034a7eaf243f5474603f10
0903 20:05:21.871464 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.871542 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2460519}
I20250903 20:05:21.872025 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 550) took 2681 ms. Trace:
W20250903 20:05:21.872071 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 64c4dd2645034a7eaf243f5474603f10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64c4dd2645034a7eaf243f5474603f10 already in progress: creating tablet
I20250903 20:05:21.872134 27687 rpcz_store.cc:276] 0903 20:05:19.190974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:19.411186 (+220212us) service_pool.cc:225] Handling call
0903 20:05:21.871883 (+2460697us) ts_tablet_manager.cc:1145] Deleting tablet 90baf83357494a0aae325908c118b471
0903 20:05:21.871891 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.871980 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2460392}
I20250903 20:05:21.872467 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 558) took 1926 ms. Trace:
W20250903 20:05:21.872514 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 90baf83357494a0aae325908c118b471 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90baf83357494a0aae325908c118b471 already in progress: creating tablet
I20250903 20:05:21.872567 27687 rpcz_store.cc:276] 0903 20:05:19.946475 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:19.946566 (+ 91us) service_pool.cc:225] Handling call
0903 20:05:21.872347 (+1925781us) ts_tablet_manager.cc:1145] Deleting tablet e3e82f37c6fa4bebb0115e6087919a27
0903 20:05:21.872356 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.872437 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1924965}
I20250903 20:05:21.872888 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 557) took 1928 ms. Trace:
W20250903 20:05:21.872975 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet e3e82f37c6fa4bebb0115e6087919a27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3e82f37c6fa4bebb0115e6087919a27 already in progress: creating tablet
I20250903 20:05:21.830582 27503 tablet_service.cc:1515] Processing DeleteTablet for tablet 369c750ff5bf449e989d95ffd89497d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:21.830786 27496 tablet_service.cc:1515] Processing DeleteTablet for tablet dbfdab4332234b2a93fd3c901a9cb645 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:21.874913 27247 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5fa7cc0c03234223af382982ca737e27" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:21.875157 27247 raft_consensus.cc:3058] T 5fa7cc0c03234223af382982ca737e27 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:21.878443 27695 tablet_bootstrap.cc:492] T 516830fc42e84af693b3af52caee30ee P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:21.878628 27695 ts_tablet_manager.cc:1397] T 516830fc42e84af693b3af52caee30ee P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.205s user 0.005s sys 0.000s
I20250903 20:05:21.879647 27695 raft_consensus.cc:357] T 516830fc42e84af693b3af52caee30ee P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:21.879877 27695 raft_consensus.cc:383] T 516830fc42e84af693b3af52caee30ee P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:21.879938 27695 raft_consensus.cc:738] T 516830fc42e84af693b3af52caee30ee P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:21.880251 27695 consensus_queue.cc:260] T 516830fc42e84af693b3af52caee30ee P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:21.880872 27695 ts_tablet_manager.cc:1428] T 516830fc42e84af693b3af52caee30ee P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:21.881779 27695 tablet_bootstrap.cc:492] T ac9b5d4f39db4f05a15316585040d142 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:21.883641 27695 tablet_bootstrap.cc:654] T ac9b5d4f39db4f05a15316585040d142 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:21.883793 28298 leader_election.cc:290] T 5fa7cc0c03234223af382982ca737e27 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:21.884487 27247 raft_consensus.cc:2466] T 5fa7cc0c03234223af382982ca737e27 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:21.887259 27321 leader_election.cc:304] T 5fa7cc0c03234223af382982ca737e27 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:21.887928 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_01215977307c453e9236e5bd72bc8427" } modify_external_catalogs: true
I20250903 20:05:21.888134 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_01215977307c453e9236e5bd72bc8427" } modify_external_catalogs: true
I20250903 20:05:21.887730 28298 raft_consensus.cc:2802] T 5fa7cc0c03234223af382982ca737e27 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:21.888581 28298 raft_consensus.cc:695] T 5fa7cc0c03234223af382982ca737e27 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:21.889109 27064 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_07159d0e0aef45218e8698b043568648" } modify_external_catalogs: true
I20250903 20:05:21.889325 27064 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_07159d0e0aef45218e8698b043568648" } modify_external_catalogs: true
W20250903 20:05:21.890275 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b6663ba96faa44b88fe8b6def4ff6951 consensus state has no leader
I20250903 20:05:21.890697 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_be949d6637d94ea4bab23c54e50fdb1d" } modify_external_catalogs: true
I20250903 20:05:21.890873 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_be949d6637d94ea4bab23c54e50fdb1d" } modify_external_catalogs: true
I20250903 20:05:21.890442 28298 consensus_queue.cc:237] T 5fa7cc0c03234223af382982ca737e27 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:21.873020 27687 rpcz_store.cc:276] 0903 20:05:19.944826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:19.944909 (+ 83us) service_pool.cc:225] Handling call
0903 20:05:21.872774 (+1927865us) ts_tablet_manager.cc:1145] Deleting tablet f31e3e5ea72940de953c7dd787515c62
0903 20:05:21.872782 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.872857 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1925162}
W20250903 20:05:21.894589 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet f31e3e5ea72940de953c7dd787515c62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f31e3e5ea72940de953c7dd787515c62 already in progress: creating tablet
I20250903 20:05:21.895143 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 559) took 1945 ms. Trace:
I20250903 20:05:21.895905 27687 rpcz_store.cc:276] 0903 20:05:19.949514 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:19.950338 (+ 824us) service_pool.cc:225] Handling call
0903 20:05:21.894963 (+1944625us) ts_tablet_manager.cc:1145] Deleting tablet 3919f7f95a954f59b68f56d0051c0109
0903 20:05:21.894975 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.895104 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1941205}
W20250903 20:05:21.899762 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9e98cd1d9bd246e182e2969d147e68bf consensus state has no leader
I20250903 20:05:21.900285 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_56360d004737406d91c2fdadbd39cdbe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:21.900902 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 3919f7f95a954f59b68f56d0051c0109 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3919f7f95a954f59b68f56d0051c0109 already in progress: creating tablet
W20250903 20:05:21.900956 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56360d004737406d91c2fdadbd39cdbe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:21.901091 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c8d55d18c6bb4511ab10f35d2db6d2f9" } modify_external_catalogs: true
I20250903 20:05:21.901294 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c8d55d18c6bb4511ab10f35d2db6d2f9" } modify_external_catalogs: true
I20250903 20:05:21.902624 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 560) took 1946 ms. Trace:
I20250903 20:05:21.902747 27687 rpcz_store.cc:276] 0903 20:05:19.955742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:19.956647 (+ 905us) service_pool.cc:225] Handling call
0903 20:05:21.902411 (+1945764us) ts_tablet_manager.cc:1145] Deleting tablet bc98762790f441038972366b1cc2c55b
0903 20:05:21.902423 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.902583 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1945331}
W20250903 20:05:21.903237 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet bc98762790f441038972366b1cc2c55b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc98762790f441038972366b1cc2c55b already in progress: creating tablet
I20250903 20:05:21.903648 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 561) took 1943 ms. Trace:
I20250903 20:05:21.903751 27687 rpcz_store.cc:276] 0903 20:05:19.960176 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:19.961395 (+ 1219us) service_pool.cc:225] Handling call
0903 20:05:21.903505 (+1942110us) ts_tablet_manager.cc:1145] Deleting tablet 314856bf269f4df7aa25f5a30cec0d5b
0903 20:05:21.903515 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.903617 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1940168}
W20250903 20:05:21.904229 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 314856bf269f4df7aa25f5a30cec0d5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 314856bf269f4df7aa25f5a30cec0d5b already in progress: creating tablet
I20250903 20:05:21.904615 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 562) took 1924 ms. Trace:
I20250903 20:05:21.904748 27061 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 5fa7cc0c03234223af382982ca737e27 (table default.Table_30f001eba314494d93e8ecac2f0bc3dc [id=66eb535587dc49e5b1ee72124cca0374]). Expected version 1 got 0
I20250903 20:05:21.904718 27687 rpcz_store.cc:276] 0903 20:05:19.980638 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:19.981487 (+ 849us) service_pool.cc:225] Handling call
0903 20:05:21.904483 (+1922996us) ts_tablet_manager.cc:1145] Deleting tablet 514e382479014fa7bf1f404efff11c86
0903 20:05:21.904492 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.904575 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1922557}
I20250903 20:05:21.905566 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_83e2f4211b724a698cf478ddde974155" } new_table_name: "default.Table_63be5f347200408ba649d6672260ac6e" modify_external_catalogs: true
I20250903 20:05:21.906167 27693 tablet_bootstrap.cc:654] T ac1b0dc95fca49b4af6697c0a70e9730 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:21.906837 27684 rpcz_store.cc:276] 0903 20:05:20.686522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.686592 (+ 70us) service_pool.cc:225] Handling call
0903 20:05:21.824266 (+1137674us) ts_tablet_manager.cc:1145] Deleting tablet 615f42e95bb24c8480f73d371fd63b9c
0903 20:05:21.824279 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.824434 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1137393}
I20250903 20:05:21.907425 27059 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 639ecb9caa4a423899ab8eb35bb716d7 (table default.Table_3da0f03ae8114e348baf44301196b2e5 [id=ad8b3a9bedef4c188b89640ab5e2473a]). Expected version 2 got 0
W20250903 20:05:21.908059 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 615f42e95bb24c8480f73d371fd63b9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 615f42e95bb24c8480f73d371fd63b9c already in progress: creating tablet
I20250903 20:05:21.908406 27692 tablet_bootstrap.cc:492] T 07565ea2b5f34dcfb407ff3bfde17243 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:21.908486 27058 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_fd3053c3354a4359b2742e9350d19d23 [id=416ff29f66d14564826a6734dde1c604] alter complete (version 3)
W20250903 20:05:21.910148 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 514e382479014fa7bf1f404efff11c86 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 514e382479014fa7bf1f404efff11c86 already in progress: creating tablet
I20250903 20:05:21.910744 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 3f43480c4f2042e18772e7f0d31dd676 (DEFAULT_TABLE table=default.Table_7a8637d7aab94246a883a6e29273e755 [id=6e619ed79f3049b2bdcfae1e454da8b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.911235 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f43480c4f2042e18772e7f0d31dd676. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.914577 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 005d7bc2c7534f288caec8326fdf2416 (DEFAULT_TABLE table=default.Table_56af93fd7df844a99722a1b88e0c54c6 [id=8c34ca6735a2428b88780dde53c69553]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.915058 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 005d7bc2c7534f288caec8326fdf2416. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.918195 27220 tablet_service.cc:1468] Processing CreateTablet for tablet b0d176b63e624afa8c90841ca0f82a51 (DEFAULT_TABLE table=default.Table_624f5e99e4d344d3ae431fd805971ac7 [id=a17e600dcc3643fa83ff684614128175]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.918614 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0d176b63e624afa8c90841ca0f82a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.920331 27227 tablet_service.cc:1468] Processing CreateTablet for tablet cc374781309042a394d517734fdd6918 (DEFAULT_TABLE table=default.Table_14a46b39c912447db13c3d3d84cad475 [id=1847418363a14f6aaa79180b9cec9c8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.920789 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc374781309042a394d517734fdd6918. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.922372 27692 ts_tablet_manager.cc:1397] T 07565ea2b5f34dcfb407ff3bfde17243 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.212s user 0.001s sys 0.004s
I20250903 20:05:21.923640 27692 raft_consensus.cc:357] T 07565ea2b5f34dcfb407ff3bfde17243 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:21.924903 27692 raft_consensus.cc:383] T 07565ea2b5f34dcfb407ff3bfde17243 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:21.925016 27692 raft_consensus.cc:738] T 07565ea2b5f34dcfb407ff3bfde17243 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:21.925630 27692 consensus_queue.cc:260] T 07565ea2b5f34dcfb407ff3bfde17243 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:21.926159 27692 ts_tablet_manager.cc:1428] T 07565ea2b5f34dcfb407ff3bfde17243 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20250903 20:05:21.927968 27503 tablet_service.cc:1468] Processing CreateTablet for tablet 3f43480c4f2042e18772e7f0d31dd676 (DEFAULT_TABLE table=default.Table_7a8637d7aab94246a883a6e29273e755 [id=6e619ed79f3049b2bdcfae1e454da8b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.929033 27685 ts_tablet_manager.cc:1918] T 3f78a951bb024cf290116b0fe438ddb4 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:21.929129 27685 log.cc:1199] T 3f78a951bb024cf290116b0fe438ddb4 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/3f78a951bb024cf290116b0fe438ddb4
I20250903 20:05:21.929442 27685 ts_tablet_manager.cc:1939] T 3f78a951bb024cf290116b0fe438ddb4 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:21.929715 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 484) took 3452 ms. Trace:
I20250903 20:05:21.929775 27685 rpcz_store.cc:276] 0903 20:05:18.476990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.477320 (+ 330us) service_pool.cc:225] Handling call
0903 20:05:21.862798 (+3385478us) ts_tablet_manager.cc:1145] Deleting tablet 3f78a951bb024cf290116b0fe438ddb4
0903 20:05:21.862806 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.905595 (+ 42789us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.928925 (+ 23330us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.929702 (+ 777us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3374323}
I20250903 20:05:21.930680 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 3f78a951bb024cf290116b0fe438ddb4 (table default.Table_835ced1f8b194192809c3b6fbf4d1d96 [id=23f448924ec04ef08a541a14bd2ce5df]) successfully deleted
I20250903 20:05:21.930909 27211 tablet_service.cc:1468] Processing CreateTablet for tablet 24ddae9179cd4240914cbbae31d8b851 (DEFAULT_TABLE table=default.Table_9b23868e884f40f0b6b2ffdb5e0c99d2 [id=5aed608e8446408ebb65dab7f8004ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.930928 27229 tablet_service.cc:1468] Processing CreateTablet for tablet cc1bc205bf39447bb9c3c7863a0959cf (DEFAULT_TABLE table=default.Table_f6658302f6a546d0a9a2348e78572b99 [id=4f746a6d5a52400c8bbe6e98e853c7bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.931396 27211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ddae9179cd4240914cbbae31d8b851. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.931597 27229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc1bc205bf39447bb9c3c7863a0959cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.933521 27210 tablet_service.cc:1468] Processing CreateTablet for tablet 3f08c7edbcf54c16bc222f59773f72b1 (DEFAULT_TABLE table=default.Table_afd931c89f1b4dcab359c5c7fa189952 [id=6cf04b0c84ec444683a35c2bb240fcd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.933897 27210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f08c7edbcf54c16bc222f59773f72b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.938884 27497 tablet_service.cc:1468] Processing CreateTablet for tablet b0d176b63e624afa8c90841ca0f82a51 (DEFAULT_TABLE table=default.Table_624f5e99e4d344d3ae431fd805971ac7 [id=a17e600dcc3643fa83ff684614128175]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.938884 27496 tablet_service.cc:1468] Processing CreateTablet for tablet 005d7bc2c7534f288caec8326fdf2416 (DEFAULT_TABLE table=default.Table_56af93fd7df844a99722a1b88e0c54c6 [id=8c34ca6735a2428b88780dde53c69553]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.939806 27211 tablet_service.cc:1468] Processing CreateTablet for tablet de205e12e58e408a85bf40670680a8b2 (DEFAULT_TABLE table=default.Table_c047ef78f17b427ab6e3e998938c5fe1 [id=b58755f767a442309963a180e29fa296]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.939841 27500 tablet_service.cc:1468] Processing CreateTablet for tablet cc374781309042a394d517734fdd6918 (DEFAULT_TABLE table=default.Table_14a46b39c912447db13c3d3d84cad475 [id=1847418363a14f6aaa79180b9cec9c8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.940253 27498 tablet_service.cc:1468] Processing CreateTablet for tablet cc1bc205bf39447bb9c3c7863a0959cf (DEFAULT_TABLE table=default.Table_f6658302f6a546d0a9a2348e78572b99 [id=4f746a6d5a52400c8bbe6e98e853c7bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.940342 27213 tablet_service.cc:1468] Processing CreateTablet for tablet fc641b36ec434434967218436d8ce924 (DEFAULT_TABLE table=default.Table_fc74bbf4720b42328ed8366e277815e8 [id=b1b65ec45c89496b94a84ca3be9512eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.940623 27491 tablet_service.cc:1468] Processing CreateTablet for tablet 3f08c7edbcf54c16bc222f59773f72b1 (DEFAULT_TABLE table=default.Table_afd931c89f1b4dcab359c5c7fa189952 [id=6cf04b0c84ec444683a35c2bb240fcd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.940689 27211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de205e12e58e408a85bf40670680a8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.940645 27229 tablet_service.cc:1468] Processing CreateTablet for tablet f6df379fb1a54e1cbc2aff4852c3f313 (DEFAULT_TABLE table=default.Table_d3c128a2d7454429bbe0c058c808a9f5 [id=89456c90627d4ceba04fd10c5febc12f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.940907 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc641b36ec434434967218436d8ce924. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.940943 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 24ddae9179cd4240914cbbae31d8b851 (DEFAULT_TABLE table=default.Table_9b23868e884f40f0b6b2ffdb5e0c99d2 [id=5aed608e8446408ebb65dab7f8004ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.941115 27229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6df379fb1a54e1cbc2aff4852c3f313. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.941335 27507 tablet_service.cc:1468] Processing CreateTablet for tablet de205e12e58e408a85bf40670680a8b2 (DEFAULT_TABLE table=default.Table_c047ef78f17b427ab6e3e998938c5fe1 [id=b58755f767a442309963a180e29fa296]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.941671 27490 tablet_service.cc:1468] Processing CreateTablet for tablet f6df379fb1a54e1cbc2aff4852c3f313 (DEFAULT_TABLE table=default.Table_d3c128a2d7454429bbe0c058c808a9f5 [id=89456c90627d4ceba04fd10c5febc12f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.942010 27505 tablet_service.cc:1468] Processing CreateTablet for tablet fc641b36ec434434967218436d8ce924 (DEFAULT_TABLE table=default.Table_fc74bbf4720b42328ed8366e277815e8 [id=b1b65ec45c89496b94a84ca3be9512eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:21.943022 27684 tablet_replica.cc:331] T c9129a3e773a4fecb28ceee27778a294 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:21.943270 27684 raft_consensus.cc:2241] T c9129a3e773a4fecb28ceee27778a294 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:21.943452 27684 raft_consensus.cc:2270] T c9129a3e773a4fecb28ceee27778a294 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:21.944020 27684 ts_tablet_manager.cc:1905] T c9129a3e773a4fecb28ceee27778a294 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:21.946288 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5fa7cc0c03234223af382982ca737e27" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:21.947127 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5fa7cc0c03234223af382982ca737e27" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:21.949168 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0d176b63e624afa8c90841ca0f82a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.955377 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 005d7bc2c7534f288caec8326fdf2416. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.958642 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ddae9179cd4240914cbbae31d8b851. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.959553 27490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6df379fb1a54e1cbc2aff4852c3f313. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.962496 27491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f08c7edbcf54c16bc222f59773f72b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.962759 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 485) took 3450 ms. Trace:
I20250903 20:05:21.962850 27685 rpcz_store.cc:276] 0903 20:05:18.512682 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.512750 (+ 68us) service_pool.cc:225] Handling call
0903 20:05:21.962576 (+3449826us) ts_tablet_manager.cc:1145] Deleting tablet f0efbb7eefb84ab6bee5b9625b91e298
0903 20:05:21.962589 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.962731 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3442880}
W20250903 20:05:21.963425 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet f0efbb7eefb84ab6bee5b9625b91e298 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0efbb7eefb84ab6bee5b9625b91e298 already in progress: creating tablet
I20250903 20:05:21.964864 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 487) took 3425 ms. Trace:
I20250903 20:05:21.964946 27685 rpcz_store.cc:276] 0903 20:05:18.538946 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.539481 (+ 535us) service_pool.cc:225] Handling call
0903 20:05:21.964723 (+3425242us) ts_tablet_manager.cc:1145] Deleting tablet a38f65e992884635aae9942681c0dc4f
0903 20:05:21.964733 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.964845 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3424804}
W20250903 20:05:21.965451 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet a38f65e992884635aae9942681c0dc4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a38f65e992884635aae9942681c0dc4f already in progress: creating tablet
I20250903 20:05:21.965724 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc641b36ec434434967218436d8ce924. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.969112 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de205e12e58e408a85bf40670680a8b2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:21.969700 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8330ab3e4b6f4560b9e41a9df280c211 consensus state has no leader
I20250903 20:05:21.970053 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_fb9f90cfd77144cc81d36b52a6fc032c" } new_table_name: "default.Table_abbe893fd70a474eaccad776210d7f2f" modify_external_catalogs: true
I20250903 20:05:21.970088 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ced71ca60b354c2fb16776250f6db517"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:21.970750 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ced71ca60b354c2fb16776250f6db517 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:21.971542 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 4157633525034a019fd6a80185892cec
I20250903 20:05:21.972043 27065 catalog_manager.cc:6652] ReplaceTablet: tablet 4157633525034a019fd6a80185892cec deleted and replaced by tablet ed7c564138cc43e5a3bb4e92fb5060b6
I20250903 20:05:21.972147 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 36e1061915574407b6e7695be8b1e55a
I20250903 20:05:21.972386 27498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc1bc205bf39447bb9c3c7863a0959cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.972473 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_59c192b7b71c41f4b9a349690a5e4ea3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:21.972882 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 17e7deb74389448db37289563b86c253
W20250903 20:05:21.973014 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59c192b7b71c41f4b9a349690a5e4ea3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:21.973371 27057 catalog_manager.cc:6652] ReplaceTablet: tablet 17e7deb74389448db37289563b86c253 deleted and replaced by tablet e4d631bff80747b6b00be65f1325d80d
I20250903 20:05:21.973748 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet cd92e3dafca6419f8eba7a4ef51a911e
I20250903 20:05:21.974561 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet e9126e4a18c546c28927f22c9f16426e
I20250903 20:05:21.975270 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet c49767d618be4fd18f30e5124ce0cb84
I20250903 20:05:21.975740 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc374781309042a394d517734fdd6918. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.978498 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f43480c4f2042e18772e7f0d31dd676. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:21.979382 27695 tablet_bootstrap.cc:492] T ac9b5d4f39db4f05a15316585040d142 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:21.979568 27695 ts_tablet_manager.cc:1397] T ac9b5d4f39db4f05a15316585040d142 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.098s user 0.006s sys 0.000s
I20250903 20:05:21.979394 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 486) took 3440 ms. Trace:
I20250903 20:05:21.980316 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet cd92e3dafca6419f8eba7a4ef51a911e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:21.980541 27224 tablet_service.cc:1515] Processing DeleteTablet for tablet c49767d618be4fd18f30e5124ce0cb84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:21.980552 27227 tablet_service.cc:1515] Processing DeleteTablet for tablet 36e1061915574407b6e7695be8b1e55a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:21.979954 27685 rpcz_store.cc:276] 0903 20:05:18.538710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.540721 (+ 2011us) service_pool.cc:225] Handling call
0903 20:05:21.970363 (+3429642us) ts_tablet_manager.cc:1145] Deleting tablet f4dec89fc6bb436ba6542c267eaf9c15
0903 20:05:21.970375 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.979370 (+ 8995us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3418351}
I20250903 20:05:21.981452 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 488) took 3277 ms. Trace:
I20250903 20:05:21.981670 27228 tablet_service.cc:1515] Processing DeleteTablet for tablet 17e7deb74389448db37289563b86c253 with delete_type TABLET_DATA_DELETED (replaced by tablet e4d631bff80747b6b00be65f1325d80d) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:21.981935 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet f4dec89fc6bb436ba6542c267eaf9c15 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4dec89fc6bb436ba6542c267eaf9c15 already in progress: creating tablet
I20250903 20:05:21.981536 27685 rpcz_store.cc:276] 0903 20:05:18.703961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.704032 (+ 71us) service_pool.cc:225] Handling call
0903 20:05:21.981288 (+3277256us) ts_tablet_manager.cc:1145] Deleting tablet 20c25c84c6fb4609a1964a4ff3c526da
0903 20:05:21.981302 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.981433 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3276959}
I20250903 20:05:21.982709 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 489) took 3273 ms. Trace:
I20250903 20:05:21.982753 27060 master_service.cc:948] ReplaceTablet: received request to replace tablet 441bb228c5c54aebb6aba2bc5761bd00 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:21.982789 27685 rpcz_store.cc:276] 0903 20:05:18.708872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.708928 (+ 56us) service_pool.cc:225] Handling call
0903 20:05:21.982576 (+3273648us) ts_tablet_manager.cc:1145] Deleting tablet ac9b5d4f39db4f05a15316585040d142
0903 20:05:21.982585 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.982694 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3273411}
I20250903 20:05:21.983034 27685 tablet_replica.cc:331] T 57a1167457b84ea8a493be232b1d088a P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:21.983242 27685 raft_consensus.cc:2241] T 57a1167457b84ea8a493be232b1d088a P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:21.980511 27695 raft_consensus.cc:357] T ac9b5d4f39db4f05a15316585040d142 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:21.983606 27695 raft_consensus.cc:383] T ac9b5d4f39db4f05a15316585040d142 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:21.983635 27219 tablet_service.cc:1515] Processing DeleteTablet for tablet e9126e4a18c546c28927f22c9f16426e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:21.983659 27695 raft_consensus.cc:738] T ac9b5d4f39db4f05a15316585040d142 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:21.983863 27695 consensus_queue.cc:260] T ac9b5d4f39db4f05a15316585040d142 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:21.984066 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 20c25c84c6fb4609a1964a4ff3c526da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20c25c84c6fb4609a1964a4ff3c526da already in progress: creating tablet
I20250903 20:05:21.984200 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 4157633525034a019fd6a80185892cec with delete_type TABLET_DATA_DELETED (replaced by tablet ed7c564138cc43e5a3bb4e92fb5060b6) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:21.984335 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet ac9b5d4f39db4f05a15316585040d142 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac9b5d4f39db4f05a15316585040d142 already in progress: creating tablet
I20250903 20:05:21.984387 27695 ts_tablet_manager.cc:1428] T ac9b5d4f39db4f05a15316585040d142 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250903 20:05:21.984481 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet 36e1061915574407b6e7695be8b1e55a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:21.984690 27695 tablet_bootstrap.cc:492] T 7c398634d6304a12b35c0579fba4c813 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:21.984694 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet 17e7deb74389448db37289563b86c253 with delete_type TABLET_DATA_DELETED (replaced by tablet e4d631bff80747b6b00be65f1325d80d) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:21.983436 27685 raft_consensus.cc:2270] T 57a1167457b84ea8a493be232b1d088a P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:21.984889 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet cd92e3dafca6419f8eba7a4ef51a911e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:21.985066 27489 tablet_service.cc:1515] Processing DeleteTablet for tablet e9126e4a18c546c28927f22c9f16426e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:21.985270 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet c49767d618be4fd18f30e5124ce0cb84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:21.985838 27685 ts_tablet_manager.cc:1905] T 57a1167457b84ea8a493be232b1d088a P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:21.986482 27695 tablet_bootstrap.cc:654] T 7c398634d6304a12b35c0579fba4c813 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:21.988193 27692 tablet_bootstrap.cc:492] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:21.988945 27229 tablet_service.cc:1515] Processing DeleteTablet for tablet 4157633525034a019fd6a80185892cec with delete_type TABLET_DATA_DELETED (replaced by tablet ed7c564138cc43e5a3bb4e92fb5060b6) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:21.993140 27692 tablet_bootstrap.cc:654] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:21.995877 27685 ts_tablet_manager.cc:1918] T 57a1167457b84ea8a493be232b1d088a P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:21.998309 27685 log.cc:1199] T 57a1167457b84ea8a493be232b1d088a P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/57a1167457b84ea8a493be232b1d088a
I20250903 20:05:21.998886 27685 ts_tablet_manager.cc:1939] T 57a1167457b84ea8a493be232b1d088a P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:22.000169 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 490) took 3283 ms. Trace:
I20250903 20:05:22.001961 27685 rpcz_store.cc:276] 0903 20:05:18.716831 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.717052 (+ 221us) service_pool.cc:225] Handling call
0903 20:05:21.982993 (+3265941us) ts_tablet_manager.cc:1145] Deleting tablet 57a1167457b84ea8a493be232b1d088a
0903 20:05:21.983001 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:21.989732 (+ 6731us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:21.995752 (+ 6020us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:22.000144 (+ 4392us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3265666}
I20250903 20:05:22.002844 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 57a1167457b84ea8a493be232b1d088a (table default.Table_b2a82a2c310140caaeac5d0dcb4bb61a [id=e2548ad469e54a41af7c0181357d2467]) successfully deleted
I20250903 20:05:22.003161 27685 tablet_replica.cc:331] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:22.003573 27685 raft_consensus.cc:2241] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:22.004206 27685 raft_consensus.cc:2270] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:22.005218 27685 ts_tablet_manager.cc:1905] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:22.007880 27695 tablet_bootstrap.cc:492] T 7c398634d6304a12b35c0579fba4c813 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:22.008100 27695 ts_tablet_manager.cc:1397] T 7c398634d6304a12b35c0579fba4c813 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20250903 20:05:22.009037 27695 raft_consensus.cc:357] T 7c398634d6304a12b35c0579fba4c813 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:22.009342 27695 raft_consensus.cc:383] T 7c398634d6304a12b35c0579fba4c813 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:22.009449 27695 raft_consensus.cc:738] T 7c398634d6304a12b35c0579fba4c813 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:22.009752 27695 consensus_queue.cc:260] T 7c398634d6304a12b35c0579fba4c813 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:22.010363 27695 ts_tablet_manager.cc:1428] T 7c398634d6304a12b35c0579fba4c813 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:22.012390 27695 tablet_bootstrap.cc:492] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:22.014372 27695 tablet_bootstrap.cc:654] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:22.029932 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_cb6d16bb91ba492b9f25371c40efa85e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:22.030823 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb6d16bb91ba492b9f25371c40efa85e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:22.032505 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_4f8ccb14c2844148b738bf1cc6b94ce7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:22.033191 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f8ccb14c2844148b738bf1cc6b94ce7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:22.033679 27695 tablet_bootstrap.cc:492] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:22.033842 27695 ts_tablet_manager.cc:1397] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.005s
I20250903 20:05:22.034723 27695 raft_consensus.cc:357] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:22.034910 27695 raft_consensus.cc:383] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:22.035001 27695 raft_consensus.cc:738] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:22.035218 27695 consensus_queue.cc:260] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:22.036266 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e73660c6b8c7464795b0571cfbd8b801" } modify_external_catalogs: true
I20250903 20:05:22.036473 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e73660c6b8c7464795b0571cfbd8b801" } modify_external_catalogs: true
I20250903 20:05:22.041316 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_7d6fa591f0c44be9af702f89f2a924f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:22.042142 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d6fa591f0c44be9af702f89f2a924f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:22.044895 27695 ts_tablet_manager.cc:1428] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20250903 20:05:22.040190 27685 ts_tablet_manager.cc:1918] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:22.047551 27685 log.cc:1199] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/1396d4f8a93b412ca8397368fb98834e
I20250903 20:05:22.049427 27687 tablet_replica.cc:331] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:22.050436 27351 tablet_service.cc:1468] Processing CreateTablet for tablet 3f43480c4f2042e18772e7f0d31dd676 (DEFAULT_TABLE table=default.Table_7a8637d7aab94246a883a6e29273e755 [id=6e619ed79f3049b2bdcfae1e454da8b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.050930 27351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f43480c4f2042e18772e7f0d31dd676. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.053282 27687 raft_consensus.cc:2241] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:22.053575 27687 pending_rounds.cc:70] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528: Trying to abort 1 pending ops.
W20250903 20:05:22.053989 27320 leader_election.cc:343] T 5fa7cc0c03234223af382982ca737e27 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:22.057785 27320 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 107232us
I20250903 20:05:22.060261 27362 tablet_service.cc:1468] Processing CreateTablet for tablet b0d176b63e624afa8c90841ca0f82a51 (DEFAULT_TABLE table=default.Table_624f5e99e4d344d3ae431fd805971ac7 [id=a17e600dcc3643fa83ff684614128175]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.060757 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0d176b63e624afa8c90841ca0f82a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.059501 27684 ts_tablet_manager.cc:1918] T c9129a3e773a4fecb28ceee27778a294 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:22.060933 27684 log.cc:1199] T c9129a3e773a4fecb28ceee27778a294 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/c9129a3e773a4fecb28ceee27778a294
I20250903 20:05:22.061326 27684 ts_tablet_manager.cc:1939] T c9129a3e773a4fecb28ceee27778a294 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:22.057644 27348 tablet_service.cc:1468] Processing CreateTablet for tablet 005d7bc2c7534f288caec8326fdf2416 (DEFAULT_TABLE table=default.Table_56af93fd7df844a99722a1b88e0c54c6 [id=8c34ca6735a2428b88780dde53c69553]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.062198 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 607) took 1288 ms. Trace:
I20250903 20:05:22.062350 27684 rpcz_store.cc:276] 0903 20:05:20.773625 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.774453 (+ 828us) service_pool.cc:225] Handling call
0903 20:05:21.907139 (+1132686us) ts_tablet_manager.cc:1145] Deleting tablet c9129a3e773a4fecb28ceee27778a294
0903 20:05:21.942903 (+ 35764us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:22.029595 (+ 86692us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:22.059352 (+ 29757us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:22.062136 (+ 2784us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1124414}
I20250903 20:05:22.062837 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 612) took 1288 ms. Trace:
I20250903 20:05:22.062930 27684 rpcz_store.cc:276] 0903 20:05:20.774317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.774520 (+ 203us) service_pool.cc:225] Handling call
0903 20:05:22.062651 (+1288131us) ts_tablet_manager.cc:1145] Deleting tablet 39c2ebe4ada5426590ec3769c8662644
0903 20:05:22.062661 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:22.062773 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1279753}
I20250903 20:05:22.063040 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet c9129a3e773a4fecb28ceee27778a294 (table default.Table_b2824f7f5f324e6ebcacb1b0645de8ec [id=50898d980302442fbd70cb88f8f4faba]) successfully deleted
I20250903 20:05:22.063323 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 611) took 1289 ms. Trace:
I20250903 20:05:22.063409 27684 rpcz_store.cc:276] 0903 20:05:20.774189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.774500 (+ 311us) service_pool.cc:225] Handling call
0903 20:05:22.063169 (+1288669us) ts_tablet_manager.cc:1145] Deleting tablet 1bf69315f07a470bb2b897ad7c590729
0903 20:05:22.063178 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:22.063265 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1268151}
I20250903 20:05:22.063774 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 608) took 1289 ms. Trace:
I20250903 20:05:22.063863 27684 rpcz_store.cc:276] 0903 20:05:20.773908 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.774600 (+ 692us) service_pool.cc:225] Handling call
0903 20:05:22.063624 (+1289024us) ts_tablet_manager.cc:1145] Deleting tablet ebf0a9aaaea1408c8c9caf28f00dbff7
0903 20:05:22.063632 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:22.063718 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1267457}
I20250903 20:05:22.064162 27684 tablet_replica.cc:331] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
W20250903 20:05:22.064167 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 1bf69315f07a470bb2b897ad7c590729 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1bf69315f07a470bb2b897ad7c590729 already in progress: creating tablet
I20250903 20:05:22.064356 27684 raft_consensus.cc:2241] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
W20250903 20:05:22.064699 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet ebf0a9aaaea1408c8c9caf28f00dbff7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ebf0a9aaaea1408c8c9caf28f00dbff7 already in progress: creating tablet
I20250903 20:05:22.064723 27684 raft_consensus.cc:2270] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:22.064981 27320 leader_election.cc:343] T 5fa7cc0c03234223af382982ca737e27 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:22.065369 27684 ts_tablet_manager.cc:1905] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:22.065618 27358 tablet_service.cc:1468] Processing CreateTablet for tablet cc374781309042a394d517734fdd6918 (DEFAULT_TABLE table=default.Table_14a46b39c912447db13c3d3d84cad475 [id=1847418363a14f6aaa79180b9cec9c8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.066270 27364 tablet_service.cc:1468] Processing CreateTablet for tablet cc1bc205bf39447bb9c3c7863a0959cf (DEFAULT_TABLE table=default.Table_f6658302f6a546d0a9a2348e78572b99 [id=4f746a6d5a52400c8bbe6e98e853c7bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.066857 27692 tablet_bootstrap.cc:492] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:22.065376 27687 pending_rounds.cc:77] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384958183276544 op_type: NO_OP noop_request { }
I20250903 20:05:22.067004 27692 ts_tablet_manager.cc:1397] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.079s user 0.003s sys 0.002s
I20250903 20:05:22.067219 27350 tablet_service.cc:1468] Processing CreateTablet for tablet 3f08c7edbcf54c16bc222f59773f72b1 (DEFAULT_TABLE table=default.Table_afd931c89f1b4dcab359c5c7fa189952 [id=6cf04b0c84ec444683a35c2bb240fcd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.067893 27366 tablet_service.cc:1468] Processing CreateTablet for tablet 24ddae9179cd4240914cbbae31d8b851 (DEFAULT_TABLE table=default.Table_9b23868e884f40f0b6b2ffdb5e0c99d2 [id=5aed608e8446408ebb65dab7f8004ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.067862 27692 raft_consensus.cc:357] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:22.068045 27692 raft_consensus.cc:383] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:22.068145 27692 raft_consensus.cc:738] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:22.068212 27695 tablet_bootstrap.cc:492] T 8d2d64f103fc44d7b390559b92cbf4b4 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:22.068395 27692 consensus_queue.cc:260] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:22.069644 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 005d7bc2c7534f288caec8326fdf2416. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.070194 27692 ts_tablet_manager.cc:1428] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20250903 20:05:22.070564 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f08c7edbcf54c16bc222f59773f72b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.071424 27692 tablet_bootstrap.cc:492] T 5fa7cc0c03234223af382982ca737e27 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:22.073061 27692 tablet_bootstrap.cc:654] T 5fa7cc0c03234223af382982ca737e27 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:22.073292 27687 raft_consensus.cc:2887] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:22.074259 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ddae9179cd4240914cbbae31d8b851. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.074913 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc374781309042a394d517734fdd6918. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.075205 27685 ts_tablet_manager.cc:1939] T 1396d4f8a93b412ca8397368fb98834e P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:22.078387 27695 tablet_bootstrap.cc:654] T 8d2d64f103fc44d7b390559b92cbf4b4 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:22.079278 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 492) took 3329 ms. Trace:
I20250903 20:05:22.079413 27685 rpcz_store.cc:276] 0903 20:05:18.750146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.750192 (+ 46us) service_pool.cc:225] Handling call
0903 20:05:22.003106 (+3252914us) ts_tablet_manager.cc:1145] Deleting tablet 1396d4f8a93b412ca8397368fb98834e
0903 20:05:22.003117 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:22.021474 (+ 18357us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:22.040022 (+ 18548us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:22.079221 (+ 39199us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3252689}
I20250903 20:05:22.079854 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 1396d4f8a93b412ca8397368fb98834e (table default.Table_35a3167eda1a45949f8cd8a281a88733 [id=da438a4eb783471ea0b09fe1117ec822]) successfully deleted
I20250903 20:05:22.080341 27058 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_d2625760f0884f809e7aa5e436cddec1 [id=5cfc8827e4ca4f4a94424c48d6f71eb5] alter complete (version 4)
I20250903 20:05:22.080825 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_56360d004737406d91c2fdadbd39cdbe" } new_table_name: "default.Table_639629ff34c84a8e8dc097b924db94c4" modify_external_catalogs: true
I20250903 20:05:22.083184 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 57a1167457b84ea8a493be232b1d088a with delete_type TABLET_DATA_DELETED (replaced by tablet f326ec776ba446be8d5dda5c1dc5b6ec) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:22.089067 27685 tablet_replica.cc:331] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:22.090001 27685 raft_consensus.cc:2241] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:22.090559 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc1bc205bf39447bb9c3c7863a0959cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.092690 27685 raft_consensus.cc:2270] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:22.094007 27685 ts_tablet_manager.cc:1905] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:22.095588 27687 raft_consensus.cc:2270] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:22.095963 28245 raft_consensus.cc:491] T 7c398634d6304a12b35c0579fba4c813 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:22.096134 28245 raft_consensus.cc:513] T 7c398634d6304a12b35c0579fba4c813 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:22.096764 28245 leader_election.cc:290] T 7c398634d6304a12b35c0579fba4c813 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:22.097399 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c398634d6304a12b35c0579fba4c813" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:22.097867 27183 leader_election.cc:343] T 7c398634d6304a12b35c0579fba4c813 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:22.098161 27687 ts_tablet_manager.cc:1905] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:22.099123 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 13137053365a428e9d1e8d8852a53b6a consensus state has no leader
I20250903 20:05:22.101476 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_6f2ad684529148418295c9b463e0a0f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:22.102137 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f2ad684529148418295c9b463e0a0f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:22.102360 27503 tablet_service.cc:1515] Processing DeleteTablet for tablet 07565ea2b5f34dcfb407ff3bfde17243 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:22.102838 27503 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a76a17b64a74fbfbb4f2e92a02ae1cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:22.103546 27354 tablet_service.cc:1468] Processing CreateTablet for tablet de205e12e58e408a85bf40670680a8b2 (DEFAULT_TABLE table=default.Table_c047ef78f17b427ab6e3e998938c5fe1 [id=b58755f767a442309963a180e29fa296]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.103965 27354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de205e12e58e408a85bf40670680a8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.106770 27352 tablet_service.cc:1468] Processing CreateTablet for tablet f6df379fb1a54e1cbc2aff4852c3f313 (DEFAULT_TABLE table=default.Table_d3c128a2d7454429bbe0c058c808a9f5 [id=89456c90627d4ceba04fd10c5febc12f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.107465 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6df379fb1a54e1cbc2aff4852c3f313. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.114912 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 4157633525034a019fd6a80185892cec with delete_type TABLET_DATA_DELETED (replaced by tablet ed7c564138cc43e5a3bb4e92fb5060b6) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:22.115088 27356 tablet_service.cc:1468] Processing CreateTablet for tablet fc641b36ec434434967218436d8ce924 (DEFAULT_TABLE table=default.Table_fc74bbf4720b42328ed8366e277815e8 [id=b1b65ec45c89496b94a84ca3be9512eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.116111 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc641b36ec434434967218436d8ce924. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.119661 27215 tablet_service.cc:1468] Processing CreateTablet for tablet b3bb55085af044ec8bb2368c845e7404 (DEFAULT_TABLE table=default.Table_35f8f4785b9f4c459653cff84dfcfed8 [id=e3f58a81c819465d8eb10c69483bfc65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.120141 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3bb55085af044ec8bb2368c845e7404. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.120584 27684 ts_tablet_manager.cc:1918] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 20:05:22.120712 27684 log.cc:1199] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/46e3a65118b744298b5536f88c22f1db
I20250903 20:05:22.102700 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_375c4a1643654cb09b0e80840c461183"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:22.122889 27684 ts_tablet_manager.cc:1939] T 46e3a65118b744298b5536f88c22f1db P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
W20250903 20:05:22.123378 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_375c4a1643654cb09b0e80840c461183 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:22.123781 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 624) took 1224 ms. Trace:
I20250903 20:05:22.123903 27684 rpcz_store.cc:276] 0903 20:05:20.899075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.899485 (+ 410us) service_pool.cc:225] Handling call
0903 20:05:22.064062 (+1164577us) ts_tablet_manager.cc:1145] Deleting tablet 46e3a65118b744298b5536f88c22f1db
0903 20:05:22.064073 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:22.106817 (+ 42744us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:22.120435 (+ 13618us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:22.123729 (+ 3294us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1136767}
I20250903 20:05:22.103353 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b87f7ee9a2444f9892fcadaa791f93dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:22.124369 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 625) took 1223 ms. Trace:
I20250903 20:05:22.124481 27684 rpcz_store.cc:276] 0903 20:05:20.901115 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.901528 (+ 413us) service_pool.cc:225] Handling call
0903 20:05:22.124213 (+1222685us) ts_tablet_manager.cc:1145] Deleting tablet 0906165a50b047c8a776b30832149964
0903 20:05:22.124225 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:22.124338 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1187529}
W20250903 20:05:22.124744 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b87f7ee9a2444f9892fcadaa791f93dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:22.124868 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 633) took 1049 ms. Trace:
I20250903 20:05:22.124981 27684 rpcz_store.cc:276] 0903 20:05:21.075061 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.084815 (+ 9754us) service_pool.cc:225] Handling call
0903 20:05:22.124740 (+1039925us) ts_tablet_manager.cc:1145] Deleting tablet 4af8c03c9d834405b4e25d35441db961
0903 20:05:22.124750 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:22.124837 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":973225}
I20250903 20:05:22.106719 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7f9b2d482da241399d11bfa5a80b6729" } new_table_name: "default.Table_e27e9a52e38149619ab852649bff1d03" modify_external_catalogs: true
I20250903 20:05:22.107450 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5215f44c06fc49d6a9a740f77e9c21fe" } new_table_name: "default.Table_04c1db2796be4ffd90e43a7e2e29194f" modify_external_catalogs: true
I20250903 20:05:22.126814 27210 tablet_service.cc:1468] Processing CreateTablet for tablet 20d826f1b1d64746bd8e409a12b74eb1 (DEFAULT_TABLE table=default.Table_072fbb13945d42d88062f1b63efb5b6c [id=580947124eb94ef093baa27a96f20513]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.127249 27210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20d826f1b1d64746bd8e409a12b74eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.110718 27059 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 516830fc42e84af693b3af52caee30ee (table default.Table_4d303308b63c4ce09056e226d1a30498 [id=098bc9a9b0a24a4a9edb7859310ac9b3]). Expected version 3 got 0
I20250903 20:05:22.128950 27214 tablet_service.cc:1468] Processing CreateTablet for tablet ebc222df5d1445fea88939e1152142b9 (DEFAULT_TABLE table=default.Table_e036e82874544ec9bad0f3df65db6cab [id=9e11f503c813474590c027cdaf9fa88c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.129160 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 36e1061915574407b6e7695be8b1e55a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:22.129159 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_260dc7423c364e949e0f0a46af6cf67d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:22.129632 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_260dc7423c364e949e0f0a46af6cf67d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:22.130373 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebc222df5d1445fea88939e1152142b9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:22.120961 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 82ee2034cc5f4a57b72ceab3f2dbe53c has no consensus state
I20250903 20:05:22.132696 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 17e7deb74389448db37289563b86c253 with delete_type TABLET_DATA_DELETED (replaced by tablet e4d631bff80747b6b00be65f1325d80d) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:22.133256 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 441bb228c5c54aebb6aba2bc5761bd00
I20250903 20:05:22.133690 27060 catalog_manager.cc:6652] ReplaceTablet: tablet 441bb228c5c54aebb6aba2bc5761bd00 deleted and replaced by tablet 3766a0fa511c4e25bf832b094d26c160
I20250903 20:05:22.133703 28315 raft_consensus.cc:491] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:22.134135 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet cd92e3dafca6419f8eba7a4ef51a911e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:22.135351 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet e9126e4a18c546c28927f22c9f16426e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:22.136106 27220 tablet_service.cc:1468] Processing CreateTablet for tablet 14f97a610f25488eb510ba0ce966470d (DEFAULT_TABLE table=default.Table_455b30dcbf624a178417e64410fbb531 [id=ed3ebf50fc884b10ad29a9949943baa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.136507 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14f97a610f25488eb510ba0ce966470d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.136426 27500 tablet_service.cc:1468] Processing CreateTablet for tablet 20d826f1b1d64746bd8e409a12b74eb1 (DEFAULT_TABLE table=default.Table_072fbb13945d42d88062f1b63efb5b6c [id=580947124eb94ef093baa27a96f20513]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.136807 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20d826f1b1d64746bd8e409a12b74eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.137250 27211 tablet_service.cc:1468] Processing CreateTablet for tablet 15d369d9d483414c899b1eceffbf7a61 (DEFAULT_TABLE table=default.Table_ad72e251621b4a718e423879c1f34203 [id=c29f724038f647f3a3542dc70cf81658]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.137569 27211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15d369d9d483414c899b1eceffbf7a61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.142056 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet c49767d618be4fd18f30e5124ce0cb84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:22.142344 28315 raft_consensus.cc:513] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:22.143755 27247 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6238740ce1894e899be9a40321f0ab73" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:22.143932 27247 raft_consensus.cc:2466] T 6238740ce1894e899be9a40321f0ab73 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:22.144904 28315 leader_election.cc:290] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:22.145210 27321 leader_election.cc:304] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:22.145555 28315 raft_consensus.cc:2802] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:22.145674 27213 tablet_service.cc:1468] Processing CreateTablet for tablet 79c0cc0c30ab42eab0be415e3db9589f (DEFAULT_TABLE table=default.Table_cbe4f0209a3345bc9ae4a50a260346c8 [id=21b4fd8022f6461691f87200e040e264]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.146085 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79c0cc0c30ab42eab0be415e3db9589f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.146189 28315 raft_consensus.cc:491] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:22.146354 28315 raft_consensus.cc:3058] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:22.146965 27684 tablet_replica.cc:331] T fbfabb6ff1714d2e921f45256b543759 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:22.147368 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 46e3a65118b744298b5536f88c22f1db (table default.Table_4258f924ddb3449bb5f49fb7a88a5073 [id=decb9fcf2ee64971ba8bcd8326736852]) successfully deleted
W20250903 20:05:22.147670 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 0906165a50b047c8a776b30832149964 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0906165a50b047c8a776b30832149964 already in progress: creating tablet
W20250903 20:05:22.147897 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 4af8c03c9d834405b4e25d35441db961 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4af8c03c9d834405b4e25d35441db961 already in progress: creating tablet
W20250903 20:05:22.148087 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet d4c703dafbca4e669dbb18946ffac699 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4c703dafbca4e669dbb18946ffac699 already in progress: creating tablet
W20250903 20:05:22.148479 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet a72b85dfb34047528b2b28ff0d40f828 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a72b85dfb34047528b2b28ff0d40f828 already in progress: creating tablet
W20250903 20:05:22.148713 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet a253e3399622409e92299069b19d3698 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a253e3399622409e92299069b19d3698 already in progress: creating tablet
I20250903 20:05:22.148679 27361 tablet_service.cc:1468] Processing CreateTablet for tablet b3bb55085af044ec8bb2368c845e7404 (DEFAULT_TABLE table=default.Table_35f8f4785b9f4c459653cff84dfcfed8 [id=e3f58a81c819465d8eb10c69483bfc65]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:22.148900 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet b7a6d5d8a97a4769a81ec7224d591f8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7a6d5d8a97a4769a81ec7224d591f8c already in progress: creating tablet
W20250903 20:05:22.149089 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 41056c5c5003415993bee74d7621bd14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41056c5c5003415993bee74d7621bd14 already in progress: creating tablet
W20250903 20:05:22.149272 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 2c772ab7c50c4994aacde5a05bb0f08e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c772ab7c50c4994aacde5a05bb0f08e already in progress: creating tablet
I20250903 20:05:22.149508 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3bb55085af044ec8bb2368c845e7404. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:22.149523 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 5b26ed39f2144610b786f49e62e6f0a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b26ed39f2144610b786f49e62e6f0a3 already in progress: creating tablet
I20250903 20:05:22.149938 27349 tablet_service.cc:1468] Processing CreateTablet for tablet 20d826f1b1d64746bd8e409a12b74eb1 (DEFAULT_TABLE table=default.Table_072fbb13945d42d88062f1b63efb5b6c [id=580947124eb94ef093baa27a96f20513]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.150897 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 16601b34e4764a8aa14ab95ef8778202
I20250903 20:05:22.150990 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20d826f1b1d64746bd8e409a12b74eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.153403 27357 tablet_service.cc:1468] Processing CreateTablet for tablet ebc222df5d1445fea88939e1152142b9 (DEFAULT_TABLE table=default.Table_e036e82874544ec9bad0f3df65db6cab [id=9e11f503c813474590c027cdaf9fa88c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.153903 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebc222df5d1445fea88939e1152142b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.156616 27347 tablet_service.cc:1468] Processing CreateTablet for tablet 14f97a610f25488eb510ba0ce966470d (DEFAULT_TABLE table=default.Table_455b30dcbf624a178417e64410fbb531 [id=ed3ebf50fc884b10ad29a9949943baa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.157136 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14f97a610f25488eb510ba0ce966470d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.159125 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 15d369d9d483414c899b1eceffbf7a61 (DEFAULT_TABLE table=default.Table_ad72e251621b4a718e423879c1f34203 [id=c29f724038f647f3a3542dc70cf81658]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.159180 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 79c0cc0c30ab42eab0be415e3db9589f (DEFAULT_TABLE table=default.Table_cbe4f0209a3345bc9ae4a50a260346c8 [id=21b4fd8022f6461691f87200e040e264]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.159706 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15d369d9d483414c899b1eceffbf7a61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.159999 27359 tablet_service.cc:1468] Processing CreateTablet for tablet df5f439f5aca4231af229159811b7a75 (DEFAULT_TABLE table=default.Table_60a3bd90021349e694df58967ede5cf5 [id=e905c7bb5d2f446c951bc38ba88f1945]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.160838 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df5f439f5aca4231af229159811b7a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.166754 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79c0cc0c30ab42eab0be415e3db9589f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.187539 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_62256f24380d4b02949f23a0c3c07d0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:22.188203 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62256f24380d4b02949f23a0c3c07d0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:22.189608 27064 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2bbccfd73e824cdcb9d9d13e7da829c3" } modify_external_catalogs: true
I20250903 20:05:22.189808 27064 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2bbccfd73e824cdcb9d9d13e7da829c3" } modify_external_catalogs: true
I20250903 20:05:22.189361 28315 raft_consensus.cc:513] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:22.190359 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2b6469f261a04e5d9cb69bf836d97734" } modify_external_catalogs: true
I20250903 20:05:22.190529 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2b6469f261a04e5d9cb69bf836d97734" } modify_external_catalogs: true
I20250903 20:05:22.191053 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d15cadbd10314321848d5f0657333ace" } new_table_name: "default.Table_7179a3a012834e34b703156b9c834722" modify_external_catalogs: true
I20250903 20:05:22.196650 27220 tablet_service.cc:1468] Processing CreateTablet for tablet df5f439f5aca4231af229159811b7a75 (DEFAULT_TABLE table=default.Table_60a3bd90021349e694df58967ede5cf5 [id=e905c7bb5d2f446c951bc38ba88f1945]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.202291 27229 tablet_service.cc:1515] Processing DeleteTablet for tablet ac9b5d4f39db4f05a15316585040d142 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:22.204035 27219 tablet_service.cc:1468] Processing CreateTablet for tablet 16601b34e4764a8aa14ab95ef8778202 (DEFAULT_TABLE table=default.Table_e73660c6b8c7464795b0571cfbd8b801 [id=11a727d0949240be86b3f6bb346adcfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.205585 27228 tablet_service.cc:1515] Processing DeleteTablet for tablet 16601b34e4764a8aa14ab95ef8778202 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:22.206095 27227 tablet_service.cc:1468] Processing CreateTablet for tablet 1f2124e9baae4c819aeb51cef79c9ff3 (DEFAULT_TABLE table=default.Table_17ef4b4f7c1845088dac85dd6d8f0c0c [id=9d88f62da4e947e49846cd9e879fd835]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.206735 27218 tablet_service.cc:1468] Processing CreateTablet for tablet a868191630454a1eb3a4301664178a9a (DEFAULT_TABLE table=default.Table_8c666d5ff1be4e47b55808f90d025198 [id=6e20879f3ad948faa7333872b9382536]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.207455 27224 tablet_service.cc:1468] Processing CreateTablet for tablet 838041eaf62e4d40a8cfdc0dab5494ba (DEFAULT_TABLE table=default.Table_d7d1a268897341aeb586cae942a487b3 [id=e2d04c38564b4e83a9ad30cf5c017382]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.207993 27229 tablet_service.cc:1515] Processing DeleteTablet for tablet 441bb228c5c54aebb6aba2bc5761bd00 with delete_type TABLET_DATA_DELETED (replaced by tablet 3766a0fa511c4e25bf832b094d26c160) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:22.210529 27210 tablet_service.cc:1468] Processing CreateTablet for tablet e12222227f7f4000b4601d80888e1b89 (DEFAULT_TABLE table=default.Table_ddf4574624894772bd7c0aa8c5e177cf [id=4b91255b9b124810af86c7d136c8724a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.225345 27211 tablet_service.cc:1515] Processing DeleteTablet for tablet 7c398634d6304a12b35c0579fba4c813 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:22.230624 27247 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6238740ce1894e899be9a40321f0ab73" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:22.230930 27247 raft_consensus.cc:3058] T 6238740ce1894e899be9a40321f0ab73 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:22.235311 27247 raft_consensus.cc:2466] T 6238740ce1894e899be9a40321f0ab73 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:22.236169 27321 leader_election.cc:304] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:22.236578 28340 raft_consensus.cc:2802] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:22.236716 28340 raft_consensus.cc:695] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:22.237000 28340 consensus_queue.cc:237] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:22.240361 28315 leader_election.cc:290] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:22.240998 27353 tablet_service.cc:1468] Processing CreateTablet for tablet e12222227f7f4000b4601d80888e1b89 (DEFAULT_TABLE table=default.Table_ddf4574624894772bd7c0aa8c5e177cf [id=4b91255b9b124810af86c7d136c8724a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.246013 27353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e12222227f7f4000b4601d80888e1b89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.249684 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 838041eaf62e4d40a8cfdc0dab5494ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.254681 27219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16601b34e4764a8aa14ab95ef8778202. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.258992 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a868191630454a1eb3a4301664178a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.259538 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f2124e9baae4c819aeb51cef79c9ff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.271819 27356 tablet_service.cc:1468] Processing CreateTablet for tablet 1f2124e9baae4c819aeb51cef79c9ff3 (DEFAULT_TABLE table=default.Table_17ef4b4f7c1845088dac85dd6d8f0c0c [id=9d88f62da4e947e49846cd9e879fd835]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.273650 27210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e12222227f7f4000b4601d80888e1b89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.274825 27351 tablet_service.cc:1515] Processing DeleteTablet for tablet 16601b34e4764a8aa14ab95ef8778202 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:22.275130 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f2124e9baae4c819aeb51cef79c9ff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.274482 27352 tablet_service.cc:1468] Processing CreateTablet for tablet 16601b34e4764a8aa14ab95ef8778202 (DEFAULT_TABLE table=default.Table_e73660c6b8c7464795b0571cfbd8b801 [id=11a727d0949240be86b3f6bb346adcfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.281067 27358 tablet_service.cc:1515] Processing DeleteTablet for tablet 441bb228c5c54aebb6aba2bc5761bd00 with delete_type TABLET_DATA_DELETED (replaced by tablet 3766a0fa511c4e25bf832b094d26c160) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:22.283121 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16601b34e4764a8aa14ab95ef8778202. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.288110 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df5f439f5aca4231af229159811b7a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.305956 27360 tablet_service.cc:1468] Processing CreateTablet for tablet a868191630454a1eb3a4301664178a9a (DEFAULT_TABLE table=default.Table_8c666d5ff1be4e47b55808f90d025198 [id=6e20879f3ad948faa7333872b9382536]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.310681 27684 raft_consensus.cc:2241] T fbfabb6ff1714d2e921f45256b543759 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:22.312681 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet 441bb228c5c54aebb6aba2bc5761bd00 with delete_type TABLET_DATA_DELETED (replaced by tablet 3766a0fa511c4e25bf832b094d26c160) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:22.310649 27503 tablet_service.cc:1468] Processing CreateTablet for tablet b3bb55085af044ec8bb2368c845e7404 (DEFAULT_TABLE table=default.Table_35f8f4785b9f4c459653cff84dfcfed8 [id=e3f58a81c819465d8eb10c69483bfc65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.317032 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 15d369d9d483414c899b1eceffbf7a61 (DEFAULT_TABLE table=default.Table_ad72e251621b4a718e423879c1f34203 [id=c29f724038f647f3a3542dc70cf81658]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.316195 27684 raft_consensus.cc:2270] T fbfabb6ff1714d2e921f45256b543759 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:22.317495 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3bb55085af044ec8bb2368c845e7404. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.317598 27498 tablet_service.cc:1468] Processing CreateTablet for tablet 79c0cc0c30ab42eab0be415e3db9589f (DEFAULT_TABLE table=default.Table_cbe4f0209a3345bc9ae4a50a260346c8 [id=21b4fd8022f6461691f87200e040e264]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.317770 27507 tablet_service.cc:1468] Processing CreateTablet for tablet ebc222df5d1445fea88939e1152142b9 (DEFAULT_TABLE table=default.Table_e036e82874544ec9bad0f3df65db6cab [id=9e11f503c813474590c027cdaf9fa88c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.317919 27489 tablet_service.cc:1468] Processing CreateTablet for tablet 1f2124e9baae4c819aeb51cef79c9ff3 (DEFAULT_TABLE table=default.Table_17ef4b4f7c1845088dac85dd6d8f0c0c [id=9d88f62da4e947e49846cd9e879fd835]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.318017 27685 ts_tablet_manager.cc:1918] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:22.318065 27490 tablet_service.cc:1468] Processing CreateTablet for tablet df5f439f5aca4231af229159811b7a75 (DEFAULT_TABLE table=default.Table_60a3bd90021349e694df58967ede5cf5 [id=e905c7bb5d2f446c951bc38ba88f1945]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.318171 27685 log.cc:1199] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/9a76a17b64a74fbfbb4f2e92a02ae1cd
I20250903 20:05:22.318274 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6238740ce1894e899be9a40321f0ab73" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:22.318624 27685 ts_tablet_manager.cc:1939] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:22.318863 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet 16601b34e4764a8aa14ab95ef8778202 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:22.318948 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 493) took 3560 ms. Trace:
I20250903 20:05:22.318926 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ed096bbc1cae4d4bb851ef7a59affc78" } modify_external_catalogs: true
I20250903 20:05:22.319017 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6238740ce1894e899be9a40321f0ab73" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:22.319033 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c398634d6304a12b35c0579fba4c813" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:22.319017 27685 rpcz_store.cc:276] 0903 20:05:18.758025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.758081 (+ 56us) service_pool.cc:225] Handling call
0903 20:05:22.088987 (+3330906us) ts_tablet_manager.cc:1145] Deleting tablet 9a76a17b64a74fbfbb4f2e92a02ae1cd
0903 20:05:22.089000 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:22.105788 (+ 16788us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:22.119446 (+ 13658us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:22.318931 (+199485us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3330639}
I20250903 20:05:22.319170 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ed096bbc1cae4d4bb851ef7a59affc78" } modify_external_catalogs: true
I20250903 20:05:22.319262 27506 tablet_service.cc:1468] Processing CreateTablet for tablet e12222227f7f4000b4601d80888e1b89 (DEFAULT_TABLE table=default.Table_ddf4574624894772bd7c0aa8c5e177cf [id=4b91255b9b124810af86c7d136c8724a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.319429 27491 tablet_service.cc:1468] Processing CreateTablet for tablet 14f97a610f25488eb510ba0ce966470d (DEFAULT_TABLE table=default.Table_455b30dcbf624a178417e64410fbb531 [id=ed3ebf50fc884b10ad29a9949943baa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.319532 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 9a76a17b64a74fbfbb4f2e92a02ae1cd (table default.Table_e822637ebd11495cbb3c65907c6a7c78 [id=297ca0b9100044d783d2c610c137f77b]) successfully deleted
I20250903 20:05:22.320010 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0c13f21b7b574e63abd419e44ea9b157" } modify_external_catalogs: true
I20250903 20:05:22.320062 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 494) took 3545 ms. Trace:
I20250903 20:05:22.320195 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0c13f21b7b574e63abd419e44ea9b157" } modify_external_catalogs: true
I20250903 20:05:22.320133 27685 rpcz_store.cc:276] 0903 20:05:18.774957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.775014 (+ 57us) service_pool.cc:225] Handling call
0903 20:05:22.319916 (+3544902us) ts_tablet_manager.cc:1145] Deleting tablet 0fef9aa7153a4902abce522e3ff0df3f
0903 20:05:22.319926 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:22.320044 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3544642}
W20250903 20:05:22.320633 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 0fef9aa7153a4902abce522e3ff0df3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0fef9aa7153a4902abce522e3ff0df3f already in progress: creating tablet
I20250903 20:05:22.320789 27502 tablet_service.cc:1468] Processing CreateTablet for tablet a868191630454a1eb3a4301664178a9a (DEFAULT_TABLE table=default.Table_8c666d5ff1be4e47b55808f90d025198 [id=6e20879f3ad948faa7333872b9382536]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:22.321050 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 86e33bd235c14042a578a4a1c70f11b7 consensus state has no leader
I20250903 20:05:22.322336 27494 tablet_service.cc:1468] Processing CreateTablet for tablet 838041eaf62e4d40a8cfdc0dab5494ba (DEFAULT_TABLE table=default.Table_d7d1a268897341aeb586cae942a487b3 [id=e2d04c38564b4e83a9ad30cf5c017382]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.323228 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15d369d9d483414c899b1eceffbf7a61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.324529 27362 tablet_service.cc:1468] Processing CreateTablet for tablet 838041eaf62e4d40a8cfdc0dab5494ba (DEFAULT_TABLE table=default.Table_d7d1a268897341aeb586cae942a487b3 [id=e2d04c38564b4e83a9ad30cf5c017382]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:22.325099 27184 leader_election.cc:343] T 7c398634d6304a12b35c0579fba4c813 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:22.325220 27184 leader_election.cc:304] T 7c398634d6304a12b35c0579fba4c813 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:22.325470 28245 raft_consensus.cc:2747] T 7c398634d6304a12b35c0579fba4c813 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:22.326290 27687 ts_tablet_manager.cc:1918] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:22.326450 27687 log.cc:1199] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/deb84fc3465443d99c11097e53f5f575
I20250903 20:05:22.322197 27496 tablet_service.cc:1468] Processing CreateTablet for tablet 16601b34e4764a8aa14ab95ef8778202 (DEFAULT_TABLE table=default.Table_e73660c6b8c7464795b0571cfbd8b801 [id=11a727d0949240be86b3f6bb346adcfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.327553 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a868191630454a1eb3a4301664178a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.328518 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 838041eaf62e4d40a8cfdc0dab5494ba. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:22.328600 27320 leader_election.cc:343] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:22.328961 27320 leader_election.cc:343] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:22.329730 27687 ts_tablet_manager.cc:1939] T deb84fc3465443d99c11097e53f5f575 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:22.331346 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 564) took 2248 ms. Trace:
I20250903 20:05:22.333156 27687 rpcz_store.cc:276] 0903 20:05:20.082858 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.082921 (+ 63us) service_pool.cc:225] Handling call
0903 20:05:22.010379 (+1927458us) ts_tablet_manager.cc:1145] Deleting tablet deb84fc3465443d99c11097e53f5f575
0903 20:05:22.010395 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:22.180656 (+170261us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:22.296000 (+115344us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:22.331280 (+ 35280us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1927215}
I20250903 20:05:22.334523 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 495) took 3557 ms. Trace:
I20250903 20:05:22.334714 27685 rpcz_store.cc:276] 0903 20:05:18.777299 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.777352 (+ 53us) service_pool.cc:225] Handling call
0903 20:05:22.334344 (+3556992us) ts_tablet_manager.cc:1145] Deleting tablet 1223743918814d88b401782ca4b521dd
0903 20:05:22.334359 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:22.334495 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3556747}
W20250903 20:05:22.335479 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 1223743918814d88b401782ca4b521dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1223743918814d88b401782ca4b521dd already in progress: creating tablet
W20250903 20:05:22.339412 28299 raft_consensus.cc:668] T fbfabb6ff1714d2e921f45256b543759 P e72ef1dbcc5143cf9f04a808f101c80f: failed to trigger leader election: Illegal state: RaftConsensus is not running: State = Stopped
I20250903 20:05:22.340377 27685 tablet_replica.cc:331] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:22.340682 27685 raft_consensus.cc:2241] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:22.343204 27684 ts_tablet_manager.cc:1905] T fbfabb6ff1714d2e921f45256b543759 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:22.344313 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a7492190504c4a388eea5ab3ebfe7b83" } new_table_name: "default.Table_da2a44a186144172b97e2630bf1c103d" modify_external_catalogs: true
I20250903 20:05:22.345326 27489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f2124e9baae4c819aeb51cef79c9ff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.345853 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16601b34e4764a8aa14ab95ef8778202. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.347254 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_c7ba7021073b42a8b57cb33054f9a141"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:22.347855 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7ba7021073b42a8b57cb33054f9a141 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:22.348534 27491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14f97a610f25488eb510ba0ce966470d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:22.351213 27051 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.65.125:44059: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20250903 20:05:22.352793 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e12222227f7f4000b4601d80888e1b89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.354449 27685 raft_consensus.cc:2270] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:22.347379 27695 tablet_bootstrap.cc:492] T 8d2d64f103fc44d7b390559b92cbf4b4 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:22.355007 27695 ts_tablet_manager.cc:1397] T 8d2d64f103fc44d7b390559b92cbf4b4 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.309s user 0.003s sys 0.007s
I20250903 20:05:22.355984 27695 raft_consensus.cc:357] T 8d2d64f103fc44d7b390559b92cbf4b4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:22.356252 27687 tablet_replica.cc:331] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:22.356253 27695 raft_consensus.cc:383] T 8d2d64f103fc44d7b390559b92cbf4b4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:22.356317 27695 raft_consensus.cc:738] T 8d2d64f103fc44d7b390559b92cbf4b4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:22.356573 27695 consensus_queue.cc:260] T 8d2d64f103fc44d7b390559b92cbf4b4 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:22.358140 27494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 838041eaf62e4d40a8cfdc0dab5494ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.358137 27695 ts_tablet_manager.cc:1428] T 8d2d64f103fc44d7b390559b92cbf4b4 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250903 20:05:22.358707 27695 tablet_bootstrap.cc:492] T 885fff47fea14b84ac6b34a57b827b4b P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:22.361536 27695 tablet_bootstrap.cc:654] T 885fff47fea14b84ac6b34a57b827b4b P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:22.362396 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet deb84fc3465443d99c11097e53f5f575 (table default.Table_a642c53ef0da461ba2f33dd4010e7129 [id=95919dc5590942bd8e47047598b6d6bc]) successfully deleted
I20250903 20:05:22.363052 27685 ts_tablet_manager.cc:1905] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:22.363431 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebc222df5d1445fea88939e1152142b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.353964 27693 tablet_bootstrap.cc:492] T ac1b0dc95fca49b4af6697c0a70e9730 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:22.366513 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a868191630454a1eb3a4301664178a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.366710 27687 raft_consensus.cc:2241] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:22.366989 27687 pending_rounds.cc:70] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528: Trying to abort 1 pending ops.
I20250903 20:05:22.367702 27490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df5f439f5aca4231af229159811b7a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.368312 27693 ts_tablet_manager.cc:1397] T ac1b0dc95fca49b4af6697c0a70e9730 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.519s user 0.009s sys 0.005s
I20250903 20:05:22.368394 27498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79c0cc0c30ab42eab0be415e3db9589f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:22.368889 27066 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 69f8eef52bee431a86f0eef45fd11f9f consensus state has no leader
I20250903 20:05:22.373021 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_444ca50a57904488bdecef2ff5522be1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:22.373697 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_444ca50a57904488bdecef2ff5522be1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:22.374739 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_8a3e1bffc7f340f1b4b6d7dcb66fdcfb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:22.374884 28299 raft_consensus.cc:491] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250903 20:05:22.375339 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a3e1bffc7f340f1b4b6d7dcb66fdcfb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:22.376797 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 45f7068bafaf48d290ded9d1272c0e46 consensus state has no leader
I20250903 20:05:22.377169 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b880e180787d4987961ba1ef18c3ca8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:22.377693 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b880e180787d4987961ba1ef18c3ca8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:22.378758 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d2d64f103fc44d7b390559b92cbf4b4 with delete_type TABLET_DATA_DELETED (replaced by tablet a38f65e992884635aae9942681c0dc4f) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:22.378778 28299 raft_consensus.cc:513] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:22.379508 28299 leader_election.cc:290] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:22.379774 27693 raft_consensus.cc:357] T ac1b0dc95fca49b4af6697c0a70e9730 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:22.380060 27693 raft_consensus.cc:383] T ac1b0dc95fca49b4af6697c0a70e9730 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:22.380142 27693 raft_consensus.cc:738] T ac1b0dc95fca49b4af6697c0a70e9730 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:22.380647 27693 consensus_queue.cc:260] T ac1b0dc95fca49b4af6697c0a70e9730 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:22.381196 27693 ts_tablet_manager.cc:1428] T ac1b0dc95fca49b4af6697c0a70e9730 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250903 20:05:22.385720 27687 pending_rounds.cc:77] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384959152103424 op_type: NO_OP noop_request { }
I20250903 20:05:22.385910 27693 tablet_bootstrap.cc:492] T 369c750ff5bf449e989d95ffd89497d4 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:22.397228 27687 raft_consensus.cc:2887] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:22.397526 27687 raft_consensus.cc:2270] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:22.397930 27066 catalog_manager.cc:5582] T 5fa7cc0c03234223af382982ca737e27 P 51578fb7bceb4e119b3ab1e58c4f5528 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66). New cstate: current_term: 1 leader_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:22.398537 27066 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 5fa7cc0c03234223af382982ca737e27 (table default.Table_30f001eba314494d93e8ecac2f0bc3dc [id=66eb535587dc49e5b1ee72124cca0374]). Expected version 1 got 0
I20250903 20:05:22.399569 27695 tablet_bootstrap.cc:492] T 885fff47fea14b84ac6b34a57b827b4b P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:22.399706 27695 ts_tablet_manager.cc:1397] T 885fff47fea14b84ac6b34a57b827b4b P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.001s
I20250903 20:05:22.400460 27695 raft_consensus.cc:357] T 885fff47fea14b84ac6b34a57b827b4b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:22.401074 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5253d82a41494ce59bb96928179dfdd9 consensus state has no leader
I20250903 20:05:22.405328 27687 ts_tablet_manager.cc:1905] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:22.406685 27693 tablet_bootstrap.cc:654] T 369c750ff5bf449e989d95ffd89497d4 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:22.400619 27695 raft_consensus.cc:383] T 885fff47fea14b84ac6b34a57b827b4b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:22.413424 27695 raft_consensus.cc:738] T 885fff47fea14b84ac6b34a57b827b4b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:22.413762 27695 consensus_queue.cc:260] T 885fff47fea14b84ac6b34a57b827b4b P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:22.414436 27695 ts_tablet_manager.cc:1428] T 885fff47fea14b84ac6b34a57b827b4b P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.015s user 0.001s sys 0.002s
I20250903 20:05:22.416296 27695 tablet_bootstrap.cc:492] T fa05f54ebc324b40855fa3288760e9e9 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:22.417804 27695 tablet_bootstrap.cc:654] T fa05f54ebc324b40855fa3288760e9e9 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:22.418655 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 9a71290c27ac4a22bd9aa8e14e5834f5
I20250903 20:05:22.419701 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 8093e36d79894ed3aa8e92c0bca4f93d
I20250903 20:05:22.420496 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 53881ad30bc843839008ace4012a3f70
I20250903 20:05:22.421370 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 4e99665f920f4de590bd6d9e918c7182
I20250903 20:05:22.422835 27693 tablet_bootstrap.cc:492] T 369c750ff5bf449e989d95ffd89497d4 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:22.422986 27693 ts_tablet_manager.cc:1397] T 369c750ff5bf449e989d95ffd89497d4 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.000s
I20250903 20:05:22.423863 27693 raft_consensus.cc:357] T 369c750ff5bf449e989d95ffd89497d4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:22.424052 27693 raft_consensus.cc:383] T 369c750ff5bf449e989d95ffd89497d4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:22.424163 27693 raft_consensus.cc:738] T 369c750ff5bf449e989d95ffd89497d4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:22.424455 27693 consensus_queue.cc:260] T 369c750ff5bf449e989d95ffd89497d4 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:22.425113 27365 tablet_service.cc:1468] Processing CreateTablet for tablet b991186c15e04d1a95afc5e61e79802f (DEFAULT_TABLE table=default.Table_4f8ccb14c2844148b738bf1cc6b94ce7 [id=a80d0cd8bb974d578002493ee28b055e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.425558 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b991186c15e04d1a95afc5e61e79802f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:22.425925 27321 consensus_peers.cc:489] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Couldn't send request to peer 89f8f0ee68b940db9ce250c0604f934a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f78a951bb024cf290116b0fe438ddb4. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:22.426483 27693 ts_tablet_manager.cc:1428] T 369c750ff5bf449e989d95ffd89497d4 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:22.426806 27693 tablet_bootstrap.cc:492] T 07565ea2b5f34dcfb407ff3bfde17243 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:22.428671 27693 tablet_bootstrap.cc:654] T 07565ea2b5f34dcfb407ff3bfde17243 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:22.430562 27349 tablet_service.cc:1468] Processing CreateTablet for tablet ed7c564138cc43e5a3bb4e92fb5060b6 (DEFAULT_TABLE table=default.Table_57cd5123dc7042c1887af6e833af4d7a [id=4b42213db8534ac3b3b97318f5a38e61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.431035 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed7c564138cc43e5a3bb4e92fb5060b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.434636 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 3766a0fa511c4e25bf832b094d26c160 (DEFAULT_TABLE table=default.Table_004a021331544f1bbdd8a1ede39f7590 [id=bbbde9e4eff54241b58349126f7a4dd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.435254 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3766a0fa511c4e25bf832b094d26c160. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.435384 27347 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e99665f920f4de590bd6d9e918c7182 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:22.435941 27348 tablet_service.cc:1468] Processing CreateTablet for tablet 8c0fb063ad0441e396e240e4156f07d9 (DEFAULT_TABLE table=default.Table_7d6fa591f0c44be9af702f89f2a924f3 [id=5f6cffe39859435889ffaa2df26c193f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.437384 27350 tablet_service.cc:1515] Processing DeleteTablet for tablet 8093e36d79894ed3aa8e92c0bca4f93d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:22.437567 27358 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a71290c27ac4a22bd9aa8e14e5834f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:22.435278 27351 tablet_service.cc:1515] Processing DeleteTablet for tablet 53881ad30bc843839008ace4012a3f70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:22.438002 27357 tablet_service.cc:1468] Processing CreateTablet for tablet e4d631bff80747b6b00be65f1325d80d (DEFAULT_TABLE table=default.Table_dc971e5490844bea94ad3c6b8cf15d9d [id=5cdd26db1c56496b807e8cd941cdb678]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.438179 27218 tablet_service.cc:1468] Processing CreateTablet for tablet f7c58bbb3c394d7bac43ae3e03c77666 (DEFAULT_TABLE table=default.Table_ced71ca60b354c2fb16776250f6db517 [id=d8e6d95363f04681bd07e9f4206b1e1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.438386 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c0fb063ad0441e396e240e4156f07d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.438592 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7c58bbb3c394d7bac43ae3e03c77666. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.424999 27685 ts_tablet_manager.cc:1918] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:22.437824 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet 53881ad30bc843839008ace4012a3f70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:22.439283 27685 log.cc:1199] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/6f0cba70240f4f38a4fa5ccbcd99b3b1
I20250903 20:05:22.437983 27223 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e99665f920f4de590bd6d9e918c7182 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:22.439615 27685 ts_tablet_manager.cc:1939] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:22.439917 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 496) took 3652 ms. Trace:
I20250903 20:05:22.439987 27685 rpcz_store.cc:276] 0903 20:05:18.787220 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.787270 (+ 50us) service_pool.cc:225] Handling call
0903 20:05:22.340306 (+3553036us) ts_tablet_manager.cc:1145] Deleting tablet 6f0cba70240f4f38a4fa5ccbcd99b3b1
0903 20:05:22.340321 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:22.376251 (+ 35930us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:22.422168 (+ 45917us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:22.439902 (+ 17734us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3548648}
I20250903 20:05:22.440384 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 498) took 3629 ms. Trace:
I20250903 20:05:22.440455 27685 rpcz_store.cc:276] 0903 20:05:18.810519 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.810583 (+ 64us) service_pool.cc:225] Handling call
0903 20:05:22.440260 (+3629677us) ts_tablet_manager.cc:1145] Deleting tablet 512908bc4d6941ec81b145ee5795405f
0903 20:05:22.440269 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:22.440369 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3613176}
I20250903 20:05:22.440697 27685 tablet_replica.cc:331] T 369c750ff5bf449e989d95ffd89497d4 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:22.440860 27685 raft_consensus.cc:2241] T 369c750ff5bf449e989d95ffd89497d4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:22.440996 27685 raft_consensus.cc:2270] T 369c750ff5bf449e989d95ffd89497d4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:22.441485 27685 ts_tablet_manager.cc:1905] T 369c750ff5bf449e989d95ffd89497d4 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:22.442504 27213 tablet_service.cc:1468] Processing CreateTablet for tablet 82ee2034cc5f4a57b72ceab3f2dbe53c (DEFAULT_TABLE table=default.Table_639629ff34c84a8e8dc097b924db94c4 [id=a3d5379c4b994b909efb7446bd3420ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.442875 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82ee2034cc5f4a57b72ceab3f2dbe53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.443601 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4d631bff80747b6b00be65f1325d80d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.446499 27227 tablet_service.cc:1468] Processing CreateTablet for tablet 8c0fb063ad0441e396e240e4156f07d9 (DEFAULT_TABLE table=default.Table_7d6fa591f0c44be9af702f89f2a924f3 [id=5f6cffe39859435889ffaa2df26c193f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.446858 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c0fb063ad0441e396e240e4156f07d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.450307 27224 tablet_service.cc:1468] Processing CreateTablet for tablet b991186c15e04d1a95afc5e61e79802f (DEFAULT_TABLE table=default.Table_4f8ccb14c2844148b738bf1cc6b94ce7 [id=a80d0cd8bb974d578002493ee28b055e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.450703 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b991186c15e04d1a95afc5e61e79802f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.454582 27219 tablet_service.cc:1468] Processing CreateTablet for tablet ed7c564138cc43e5a3bb4e92fb5060b6 (DEFAULT_TABLE table=default.Table_57cd5123dc7042c1887af6e833af4d7a [id=4b42213db8534ac3b3b97318f5a38e61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.454977 27219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed7c564138cc43e5a3bb4e92fb5060b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.458629 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 3766a0fa511c4e25bf832b094d26c160 (DEFAULT_TABLE table=default.Table_004a021331544f1bbdd8a1ede39f7590 [id=bbbde9e4eff54241b58349126f7a4dd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.459229 27211 tablet_service.cc:1468] Processing CreateTablet for tablet e4d631bff80747b6b00be65f1325d80d (DEFAULT_TABLE table=default.Table_dc971e5490844bea94ad3c6b8cf15d9d [id=5cdd26db1c56496b807e8cd941cdb678]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.459573 27229 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a71290c27ac4a22bd9aa8e14e5834f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:22.459786 27228 tablet_service.cc:1515] Processing DeleteTablet for tablet 8093e36d79894ed3aa8e92c0bca4f93d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:22.459973 27212 tablet_service.cc:1515] Processing DeleteTablet for tablet 885fff47fea14b84ac6b34a57b827b4b with delete_type TABLET_DATA_DELETED (replaced by tablet 5f27f525fdab407280aa23df1ae71b43) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:22.463027 27051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.125:44059, user_credentials={real_user=slave}} blocked reactor thread for 111755us
I20250903 20:05:22.463084 28340 consensus_queue.cc:1035] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:22.471978 27210 tablet_service.cc:1468] Processing CreateTablet for tablet 15072aa10c5540a09f3ec4592e377d2a (DEFAULT_TABLE table=default.Table_59c192b7b71c41f4b9a349690a5e4ea3 [id=eef3a780ec3a4cbc867782988b3d4518]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.473845 28245 raft_consensus.cc:491] T 516830fc42e84af693b3af52caee30ee P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:22.473995 28245 raft_consensus.cc:513] T 516830fc42e84af693b3af52caee30ee P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:22.474478 27693 tablet_bootstrap.cc:492] T 07565ea2b5f34dcfb407ff3bfde17243 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:22.474637 27693 ts_tablet_manager.cc:1397] T 07565ea2b5f34dcfb407ff3bfde17243 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.003s
I20250903 20:05:22.474678 28245 leader_election.cc:290] T 516830fc42e84af693b3af52caee30ee P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:22.475248 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 6f0cba70240f4f38a4fa5ccbcd99b3b1 (table default.Table_5ca5478ef07f4acdbbedb24af75f9e04 [id=7fb235b85bba4f7ab1b77bc5d2b9b58c]) successfully deleted
W20250903 20:05:22.475610 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 512908bc4d6941ec81b145ee5795405f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 512908bc4d6941ec81b145ee5795405f already in progress: creating tablet
I20250903 20:05:22.477421 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "516830fc42e84af693b3af52caee30ee" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:22.478040 27693 raft_consensus.cc:357] T 07565ea2b5f34dcfb407ff3bfde17243 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:22.478299 27693 raft_consensus.cc:383] T 07565ea2b5f34dcfb407ff3bfde17243 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:22.478391 27693 raft_consensus.cc:738] T 07565ea2b5f34dcfb407ff3bfde17243 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:22.478639 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a76a17b64a74fbfbb4f2e92a02ae1cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:11 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:22.479884 27220 tablet_service.cc:1468] Processing CreateTablet for tablet bcd798ce9b314423ba22f47c187915ce (DEFAULT_TABLE table=default.Table_cb6d16bb91ba492b9f25371c40efa85e [id=a2463b8beed4489e96d172802c5788a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.480455 27695 tablet_bootstrap.cc:492] T fa05f54ebc324b40855fa3288760e9e9 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:22.480487 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3766a0fa511c4e25bf832b094d26c160. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.480589 27695 ts_tablet_manager.cc:1397] T fa05f54ebc324b40855fa3288760e9e9 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.065s user 0.000s sys 0.007s
I20250903 20:05:22.481058 27211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4d631bff80747b6b00be65f1325d80d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.481277 27695 raft_consensus.cc:357] T fa05f54ebc324b40855fa3288760e9e9 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:22.481446 27695 raft_consensus.cc:383] T fa05f54ebc324b40855fa3288760e9e9 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:22.481506 27695 raft_consensus.cc:738] T fa05f54ebc324b40855fa3288760e9e9 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:22.481081 27693 consensus_queue.cc:260] T 07565ea2b5f34dcfb407ff3bfde17243 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:22.482352 27210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15072aa10c5540a09f3ec4592e377d2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.482892 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d2625760f0884f809e7aa5e436cddec1" } new_table_name: "default.Table_397211088a974a09bd7f031d5c7ca1cc" modify_external_catalogs: true
I20250903 20:05:22.483610 27693 ts_tablet_manager.cc:1428] T 07565ea2b5f34dcfb407ff3bfde17243 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20250903 20:05:22.484891 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_71f1cf3f76a8469695fdc49c38ba8771"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:22.485550 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71f1cf3f76a8469695fdc49c38ba8771 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:22.486033 27693 tablet_bootstrap.cc:492] T bf2545037c214bb1b5060575e3cdf81a P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:22.487226 27058 master_service.cc:948] ReplaceTablet: received request to replace tablet c6d67cd0c0304c77ba571b414878dad2 from {username='slave'} at 127.0.0.1:45936
W20250903 20:05:22.490533 27184 leader_election.cc:343] T 516830fc42e84af693b3af52caee30ee P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:22.492612 27693 tablet_bootstrap.cc:654] T bf2545037c214bb1b5060575e3cdf81a P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:22.494100 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_98cd4cb4df914678b7bb584d97bd0ca9" } modify_external_catalogs: true
I20250903 20:05:22.495839 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_98cd4cb4df914678b7bb584d97bd0ca9" } modify_external_catalogs: true
I20250903 20:05:22.501487 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcd798ce9b314423ba22f47c187915ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.507155 27352 tablet_service.cc:1468] Processing CreateTablet for tablet f7c58bbb3c394d7bac43ae3e03c77666 (DEFAULT_TABLE table=default.Table_ced71ca60b354c2fb16776250f6db517 [id=d8e6d95363f04681bd07e9f4206b1e1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.507786 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7c58bbb3c394d7bac43ae3e03c77666. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.508538 27693 tablet_bootstrap.cc:492] T bf2545037c214bb1b5060575e3cdf81a P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:22.508729 27693 ts_tablet_manager.cc:1397] T bf2545037c214bb1b5060575e3cdf81a P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.005s
I20250903 20:05:22.509626 27693 raft_consensus.cc:357] T bf2545037c214bb1b5060575e3cdf81a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:22.509840 27693 raft_consensus.cc:383] T bf2545037c214bb1b5060575e3cdf81a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:22.509938 27693 raft_consensus.cc:738] T bf2545037c214bb1b5060575e3cdf81a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:22.510178 27693 consensus_queue.cc:260] T bf2545037c214bb1b5060575e3cdf81a P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:22.510694 27693 ts_tablet_manager.cc:1428] T bf2545037c214bb1b5060575e3cdf81a P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250903 20:05:22.511031 27693 tablet_bootstrap.cc:492] T fbfabb6ff1714d2e921f45256b543759 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:22.515211 27065 master_service.cc:948] ReplaceTablet: received request to replace tablet 659a127405d341c7b074d3445854256f from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:22.521870 27693 tablet_bootstrap.cc:654] T fbfabb6ff1714d2e921f45256b543759 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:22.522132 28245 raft_consensus.cc:491] T ac9b5d4f39db4f05a15316585040d142 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:22.522346 28245 raft_consensus.cc:513] T ac9b5d4f39db4f05a15316585040d142 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:22.522619 27356 tablet_service.cc:1468] Processing CreateTablet for tablet 82ee2034cc5f4a57b72ceab3f2dbe53c (DEFAULT_TABLE table=default.Table_639629ff34c84a8e8dc097b924db94c4 [id=a3d5379c4b994b909efb7446bd3420ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.523144 28245 leader_election.cc:290] T ac9b5d4f39db4f05a15316585040d142 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:22.524405 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac9b5d4f39db4f05a15316585040d142" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:22.525015 27184 leader_election.cc:343] T ac9b5d4f39db4f05a15316585040d142 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:22.525790 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82ee2034cc5f4a57b72ceab3f2dbe53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.535410 28245 raft_consensus.cc:491] T 885fff47fea14b84ac6b34a57b827b4b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:22.535558 28245 raft_consensus.cc:513] T 885fff47fea14b84ac6b34a57b827b4b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:22.536199 28245 leader_election.cc:290] T 885fff47fea14b84ac6b34a57b827b4b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:22.537052 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "885fff47fea14b84ac6b34a57b827b4b" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:22.537632 27184 leader_election.cc:343] T 885fff47fea14b84ac6b34a57b827b4b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:22.538532 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 15072aa10c5540a09f3ec4592e377d2a (DEFAULT_TABLE table=default.Table_59c192b7b71c41f4b9a349690a5e4ea3 [id=eef3a780ec3a4cbc867782988b3d4518]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.539156 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15072aa10c5540a09f3ec4592e377d2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.541576 27695 consensus_queue.cc:260] T fa05f54ebc324b40855fa3288760e9e9 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:22.545297 27693 tablet_bootstrap.cc:492] T fbfabb6ff1714d2e921f45256b543759 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:22.545480 27693 ts_tablet_manager.cc:1397] T fbfabb6ff1714d2e921f45256b543759 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.001s
I20250903 20:05:22.546778 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_5fec15415a2b4c22826e4038aaf75aa9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:22.547580 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5fec15415a2b4c22826e4038aaf75aa9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:22.551123 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_fd3053c3354a4359b2742e9350d19d23" } new_table_name: "default.Table_10a7df6626f24353ae6661cbbdf3efb0" modify_external_catalogs: true
I20250903 20:05:22.551126 27693 raft_consensus.cc:357] T fbfabb6ff1714d2e921f45256b543759 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:22.551314 27693 raft_consensus.cc:383] T fbfabb6ff1714d2e921f45256b543759 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:22.551380 27693 raft_consensus.cc:738] T fbfabb6ff1714d2e921f45256b543759 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:22.551618 27693 consensus_queue.cc:260] T fbfabb6ff1714d2e921f45256b543759 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:22.552102 27693 ts_tablet_manager.cc:1428] T fbfabb6ff1714d2e921f45256b543759 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250903 20:05:22.552448 27693 tablet_bootstrap.cc:492] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:22.552446 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_8d0b63b0e60f4b2ba191abfd7f8680b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:22.552978 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d0b63b0e60f4b2ba191abfd7f8680b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:22.555971 28245 raft_consensus.cc:491] T 8d2d64f103fc44d7b390559b92cbf4b4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:22.556111 28245 raft_consensus.cc:513] T 8d2d64f103fc44d7b390559b92cbf4b4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:22.556670 28245 leader_election.cc:290] T 8d2d64f103fc44d7b390559b92cbf4b4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:22.557443 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d2d64f103fc44d7b390559b92cbf4b4" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:22.558007 27184 leader_election.cc:343] T 8d2d64f103fc44d7b390559b92cbf4b4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:22.560092 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_19e55aadaad1461e81d6b242dfa41c39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:22.560675 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19e55aadaad1461e81d6b242dfa41c39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:22.562014 27693 tablet_bootstrap.cc:654] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:22.565194 27224 tablet_service.cc:1515] Processing DeleteTablet for tablet fa05f54ebc324b40855fa3288760e9e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:22.574288 27695 ts_tablet_manager.cc:1428] T fa05f54ebc324b40855fa3288760e9e9 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.094s user 0.000s sys 0.002s
I20250903 20:05:22.574858 27362 tablet_service.cc:1468] Processing CreateTablet for tablet bcd798ce9b314423ba22f47c187915ce (DEFAULT_TABLE table=default.Table_cb6d16bb91ba492b9f25371c40efa85e [id=a2463b8beed4489e96d172802c5788a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.575456 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcd798ce9b314423ba22f47c187915ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.578565 27058 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 5fa7cc0c03234223af382982ca737e27 (table default.Table_30f001eba314494d93e8ecac2f0bc3dc [id=66eb535587dc49e5b1ee72124cca0374]). Expected version 1 got 0
I20250903 20:05:22.579165 27058 catalog_manager.cc:5582] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66). New cstate: current_term: 1 leader_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:22.579618 27058 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 6238740ce1894e899be9a40321f0ab73 (table default.Table_0fef1d019eff4323b7ba56368614aadc [id=53074cd97dec40c7b0e1444183d6707a]). Expected version 2 got 0
I20250903 20:05:22.584877 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 659a127405d341c7b074d3445854256f
I20250903 20:05:22.585451 27065 catalog_manager.cc:6652] ReplaceTablet: tablet 659a127405d341c7b074d3445854256f deleted and replaced by tablet 443d77f2593b44afa3e5aae4c5b0a651
I20250903 20:05:22.591578 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 659a127405d341c7b074d3445854256f with delete_type TABLET_DATA_DELETED (replaced by tablet 443d77f2593b44afa3e5aae4c5b0a651) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:22.592594 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet f9375fff5f5945078c4412aa21c06f0d
I20250903 20:05:22.594686 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 369c750ff5bf449e989d95ffd89497d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:22.595438 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 07565ea2b5f34dcfb407ff3bfde17243 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:22.596288 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet f9375fff5f5945078c4412aa21c06f0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:22.597136 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 0ab15dd783ff49868d6f9707369a5d3a (DEFAULT_TABLE table=default.Table_b87f7ee9a2444f9892fcadaa791f93dc [id=35d21449d9e94f2eb2bd7f326ce4d043]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.597506 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ab15dd783ff49868d6f9707369a5d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.604561 27351 tablet_service.cc:1468] Processing CreateTablet for tablet 2478fe89414b4845b0ba377c1cde98b7 (DEFAULT_TABLE table=default.Table_c7ba7021073b42a8b57cb33054f9a141 [id=62966767ca30457f86722ec3ab6cbd32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.604951 27351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2478fe89414b4845b0ba377c1cde98b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.605651 27357 tablet_service.cc:1468] Processing CreateTablet for tablet 760f2b8862604d5fae9097e15c1980f3 (DEFAULT_TABLE table=default.Table_375c4a1643654cb09b0e80840c461183 [id=e32e58d1c89d4de9ae27afd4685ba313]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.606117 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 760f2b8862604d5fae9097e15c1980f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.610543 27348 tablet_service.cc:1468] Processing CreateTablet for tablet 0c59083353b34b3c97dd634742350a9b (DEFAULT_TABLE table=default.Table_6f2ad684529148418295c9b463e0a0f7 [id=28f0772fb64f4c248e4a86180c976d25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.611016 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c59083353b34b3c97dd634742350a9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.611362 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 0c59083353b34b3c97dd634742350a9b (DEFAULT_TABLE table=default.Table_6f2ad684529148418295c9b463e0a0f7 [id=28f0772fb64f4c248e4a86180c976d25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.611734 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c59083353b34b3c97dd634742350a9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.614116 27358 tablet_service.cc:1468] Processing CreateTablet for tablet 5962685d79044081a75532e19bfa2bce (DEFAULT_TABLE table=default.Table_260dc7423c364e949e0f0a46af6cf67d [id=ef255544a52d4b46b4ef88d2c732d2a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.614495 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5962685d79044081a75532e19bfa2bce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.614506 27227 tablet_service.cc:1468] Processing CreateTablet for tablet 5962685d79044081a75532e19bfa2bce (DEFAULT_TABLE table=default.Table_260dc7423c364e949e0f0a46af6cf67d [id=ef255544a52d4b46b4ef88d2c732d2a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.614818 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5962685d79044081a75532e19bfa2bce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.614897 27347 tablet_service.cc:1515] Processing DeleteTablet for tablet bf2545037c214bb1b5060575e3cdf81a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:22.618382 27350 tablet_service.cc:1515] Processing DeleteTablet for tablet fbfabb6ff1714d2e921f45256b543759 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:22.619962 27213 tablet_service.cc:1468] Processing CreateTablet for tablet 2478fe89414b4845b0ba377c1cde98b7 (DEFAULT_TABLE table=default.Table_c7ba7021073b42a8b57cb33054f9a141 [id=62966767ca30457f86722ec3ab6cbd32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.620229 27365 tablet_service.cc:1468] Processing CreateTablet for tablet a9adbac18a97487b93009784023348ad (DEFAULT_TABLE table=default.Table_62256f24380d4b02949f23a0c3c07d0a [id=e845e2f2f26e4f4ea3c7ed035a7f2aaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.620461 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2478fe89414b4845b0ba377c1cde98b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.620647 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9adbac18a97487b93009784023348ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.622390 27211 tablet_service.cc:1515] Processing DeleteTablet for tablet f9375fff5f5945078c4412aa21c06f0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:22.623273 27224 tablet_service.cc:1468] Processing CreateTablet for tablet 0ab15dd783ff49868d6f9707369a5d3a (DEFAULT_TABLE table=default.Table_b87f7ee9a2444f9892fcadaa791f93dc [id=35d21449d9e94f2eb2bd7f326ce4d043]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.623759 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ab15dd783ff49868d6f9707369a5d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.624837 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "885fff47fea14b84ac6b34a57b827b4b" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:22.625289 27498 tablet_service.cc:1468] Processing CreateTablet for tablet bcd798ce9b314423ba22f47c187915ce (DEFAULT_TABLE table=default.Table_cb6d16bb91ba492b9f25371c40efa85e [id=a2463b8beed4489e96d172802c5788a8]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:22.625440 27183 leader_election.cc:343] T 885fff47fea14b84ac6b34a57b827b4b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:22.625576 27183 leader_election.cc:304] T 885fff47fea14b84ac6b34a57b827b4b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:22.625880 28245 raft_consensus.cc:2747] T 885fff47fea14b84ac6b34a57b827b4b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:22.628144 27523 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d2d64f103fc44d7b390559b92cbf4b4" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:22.628670 27183 leader_election.cc:343] T 8d2d64f103fc44d7b390559b92cbf4b4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:22.628799 27183 leader_election.cc:304] T 8d2d64f103fc44d7b390559b92cbf4b4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:22.629043 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "516830fc42e84af693b3af52caee30ee" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:22.629148 28245 raft_consensus.cc:2747] T 8d2d64f103fc44d7b390559b92cbf4b4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:22.629540 27214 tablet_service.cc:1468] Processing CreateTablet for tablet a9adbac18a97487b93009784023348ad (DEFAULT_TABLE table=default.Table_62256f24380d4b02949f23a0c3c07d0a [id=e845e2f2f26e4f4ea3c7ed035a7f2aaa]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:22.629793 27183 leader_election.cc:343] T 516830fc42e84af693b3af52caee30ee P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:22.629895 27183 leader_election.cc:304] T 516830fc42e84af693b3af52caee30ee P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:22.630048 27493 tablet_service.cc:1468] Processing CreateTablet for tablet f7c58bbb3c394d7bac43ae3e03c77666 (DEFAULT_TABLE table=default.Table_ced71ca60b354c2fb16776250f6db517 [id=d8e6d95363f04681bd07e9f4206b1e1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.630254 28245 raft_consensus.cc:2747] T 516830fc42e84af693b3af52caee30ee P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:22.630735 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9adbac18a97487b93009784023348ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.631728 27491 tablet_service.cc:1468] Processing CreateTablet for tablet 15072aa10c5540a09f3ec4592e377d2a (DEFAULT_TABLE table=default.Table_59c192b7b71c41f4b9a349690a5e4ea3 [id=eef3a780ec3a4cbc867782988b3d4518]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.634600 27219 tablet_service.cc:1468] Processing CreateTablet for tablet 760f2b8862604d5fae9097e15c1980f3 (DEFAULT_TABLE table=default.Table_375c4a1643654cb09b0e80840c461183 [id=e32e58d1c89d4de9ae27afd4685ba313]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.635200 27219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 760f2b8862604d5fae9097e15c1980f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.631603 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac9b5d4f39db4f05a15316585040d142" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:22.636436 27183 leader_election.cc:343] T ac9b5d4f39db4f05a15316585040d142 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:22.636543 27183 leader_election.cc:304] T ac9b5d4f39db4f05a15316585040d142 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:22.636909 28245 raft_consensus.cc:2747] T ac9b5d4f39db4f05a15316585040d142 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:22.637652 27498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcd798ce9b314423ba22f47c187915ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.639382 28245 raft_consensus.cc:491] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:22.639549 28245 raft_consensus.cc:513] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:22.640270 28245 leader_election.cc:290] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:22.641311 28413 tablet.cc:1722] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20250903 20:05:22.641639 27051 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 8330ab3e4b6f4560b9e41a9df280c211 (table default.Table_63be5f347200408ba649d6672260ac6e [id=07e066ded3ea4623958b36fd3874a354], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8330ab3e4b6f4560b9e41a9df280c211 consensus state has no leader
I20250903 20:05:22.641965 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "639ecb9caa4a423899ab8eb35bb716d7" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:22.643270 28412 tablet.cc:1722] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20250903 20:05:22.642822 27184 leader_election.cc:343] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:22.643715 27210 tablet_service.cc:1515] Processing DeleteTablet for tablet 659a127405d341c7b074d3445854256f with delete_type TABLET_DATA_DELETED (replaced by tablet 443d77f2593b44afa3e5aae4c5b0a651) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:22.648293 27695 tablet_bootstrap.cc:492] T cb4d99139516405bb65c7480ddf259f1 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:22.650676 27695 tablet_bootstrap.cc:654] T cb4d99139516405bb65c7480ddf259f1 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:22.651986 27692 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.564s user 0.063s sys 0.129s
I20250903 20:05:22.653268 27499 tablet_service.cc:1468] Processing CreateTablet for tablet b991186c15e04d1a95afc5e61e79802f (DEFAULT_TABLE table=default.Table_4f8ccb14c2844148b738bf1cc6b94ce7 [id=a80d0cd8bb974d578002493ee28b055e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.657543 27490 tablet_service.cc:1468] Processing CreateTablet for tablet 82ee2034cc5f4a57b72ceab3f2dbe53c (DEFAULT_TABLE table=default.Table_639629ff34c84a8e8dc097b924db94c4 [id=a3d5379c4b994b909efb7446bd3420ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.659942 27491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15072aa10c5540a09f3ec4592e377d2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.661494 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7c58bbb3c394d7bac43ae3e03c77666. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.662379 27497 tablet_service.cc:1468] Processing CreateTablet for tablet 8c0fb063ad0441e396e240e4156f07d9 (DEFAULT_TABLE table=default.Table_7d6fa591f0c44be9af702f89f2a924f3 [id=5f6cffe39859435889ffaa2df26c193f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.662824 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c0fb063ad0441e396e240e4156f07d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.663836 27504 tablet_service.cc:1468] Processing CreateTablet for tablet ed7c564138cc43e5a3bb4e92fb5060b6 (DEFAULT_TABLE table=default.Table_57cd5123dc7042c1887af6e833af4d7a [id=4b42213db8534ac3b3b97318f5a38e61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.664995 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed7c564138cc43e5a3bb4e92fb5060b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.664920 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3bf3d1e0cfbb43c2a04f1983c51e9192" } new_table_name: "default.Table_1feef4303b4e43809a0a3f1a79d68b30" modify_external_catalogs: true
I20250903 20:05:22.670353 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b991186c15e04d1a95afc5e61e79802f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.670557 27501 tablet_service.cc:1468] Processing CreateTablet for tablet 3766a0fa511c4e25bf832b094d26c160 (DEFAULT_TABLE table=default.Table_004a021331544f1bbdd8a1ede39f7590 [id=bbbde9e4eff54241b58349126f7a4dd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.670663 27695 tablet_bootstrap.cc:492] T cb4d99139516405bb65c7480ddf259f1 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:22.670787 27695 ts_tablet_manager.cc:1397] T cb4d99139516405bb65c7480ddf259f1 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20250903 20:05:22.671567 27695 raft_consensus.cc:357] T cb4d99139516405bb65c7480ddf259f1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:22.671746 27695 raft_consensus.cc:383] T cb4d99139516405bb65c7480ddf259f1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:22.671809 27695 raft_consensus.cc:738] T cb4d99139516405bb65c7480ddf259f1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:22.672047 27695 consensus_queue.cc:260] T cb4d99139516405bb65c7480ddf259f1 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:22.672538 27695 ts_tablet_manager.cc:1428] T cb4d99139516405bb65c7480ddf259f1 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:22.673915 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_1ce4ddcb0e6046bd9087209ae27c7552"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:22.674613 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ce4ddcb0e6046bd9087209ae27c7552 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:22.674968 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_30721ff3b68e4c08bdb5845184d8ada7" } modify_external_catalogs: true
I20250903 20:05:22.675194 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_30721ff3b68e4c08bdb5845184d8ada7" } modify_external_catalogs: true
I20250903 20:05:22.675388 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_455b30dcbf624a178417e64410fbb531" } new_table_name: "default.Table_ab328ca9be0d4484a401d9fdfd047d15" modify_external_catalogs: true
I20250903 20:05:22.676079 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b57aafd78ceb4611b5b6f4dd3b2bad30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:22.676591 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b57aafd78ceb4611b5b6f4dd3b2bad30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:22.677261 27501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3766a0fa511c4e25bf832b094d26c160. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.677822 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_3628bc6ae45543e39f5bc3443d68e5b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:22.678370 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3628bc6ae45543e39f5bc3443d68e5b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:22.679436 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c5eaadc5a45c45c780ebebbabf4717a0" } new_table_name: "default.Table_61979f075b5b4555b56d523ab4931364" modify_external_catalogs: true
I20250903 20:05:22.680712 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "639ecb9caa4a423899ab8eb35bb716d7" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:22.680836 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_da2a44a186144172b97e2630bf1c103d" } modify_external_catalogs: true
I20250903 20:05:22.680997 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_da2a44a186144172b97e2630bf1c103d" } modify_external_catalogs: true
W20250903 20:05:22.681254 27500 consensus_meta.cc:287] T 20d826f1b1d64746bd8e409a12b74eb1 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent flushing consensus metadata: real 0.517s user 0.004s sys 0.001s
W20250903 20:05:22.681315 27183 leader_election.cc:343] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:22.681435 27183 leader_election.cc:304] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:22.681712 27692 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.578s user 0.063s sys 0.130s
I20250903 20:05:22.682010 28245 raft_consensus.cc:2747] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:22.682651 27692 tablet_bootstrap.cc:492] T 5fa7cc0c03234223af382982ca737e27 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:22.682801 27692 ts_tablet_manager.cc:1397] T 5fa7cc0c03234223af382982ca737e27 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.611s user 0.063s sys 0.130s
I20250903 20:05:22.683542 27692 raft_consensus.cc:357] T 5fa7cc0c03234223af382982ca737e27 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:22.683737 27692 raft_consensus.cc:383] T 5fa7cc0c03234223af382982ca737e27 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:22.683832 27692 raft_consensus.cc:738] T 5fa7cc0c03234223af382982ca737e27 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:22.684106 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_61ee151cc2184644b05a60d8feb9e3c4" } modify_external_catalogs: true
I20250903 20:05:22.684303 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_61ee151cc2184644b05a60d8feb9e3c4" } modify_external_catalogs: true
I20250903 20:05:22.685655 27695 tablet_bootstrap.cc:492] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:22.688702 27695 tablet_bootstrap.cc:654] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:22.695123 27492 tablet_service.cc:1468] Processing CreateTablet for tablet e4d631bff80747b6b00be65f1325d80d (DEFAULT_TABLE table=default.Table_dc971e5490844bea94ad3c6b8cf15d9d [id=5cdd26db1c56496b807e8cd941cdb678]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.697237 27490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82ee2034cc5f4a57b72ceab3f2dbe53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.697702 27505 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a71290c27ac4a22bd9aa8e14e5834f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:22.699505 27066 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c3c4f008ae4a422d9f6d0272097e9686 consensus state has no leader
I20250903 20:05:22.699667 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 8093e36d79894ed3aa8e92c0bca4f93d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:22.699856 27492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4d631bff80747b6b00be65f1325d80d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.699990 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ef30e0f336384b50bf8293fe525252cf" } modify_external_catalogs: true
I20250903 20:05:22.700193 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ef30e0f336384b50bf8293fe525252cf" } modify_external_catalogs: true
I20250903 20:05:22.708756 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_789d6e974e2f44ff8d04b1a39adcb6a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:22.709483 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_789d6e974e2f44ff8d04b1a39adcb6a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:22.711082 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 43fb27c857d543b7b78ea6108c406085 consensus state has no leader
I20250903 20:05:22.711521 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_9b93f568af044c7aa27fb10e5225af5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:22.712113 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b93f568af044c7aa27fb10e5225af5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:22.716763 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 53881ad30bc843839008ace4012a3f70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:22.717937 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_4bc2d6a26f9b41f59aaee304f156a6c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:22.718576 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4bc2d6a26f9b41f59aaee304f156a6c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:22.723970 27505 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e99665f920f4de590bd6d9e918c7182 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:22.724428 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet f9375fff5f5945078c4412aa21c06f0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:22.727404 27507 tablet_service.cc:1468] Processing CreateTablet for tablet a9adbac18a97487b93009784023348ad (DEFAULT_TABLE table=default.Table_62256f24380d4b02949f23a0c3c07d0a [id=e845e2f2f26e4f4ea3c7ed035a7f2aaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.727835 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9adbac18a97487b93009784023348ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.728963 27488 tablet_service.cc:1468] Processing CreateTablet for tablet 760f2b8862604d5fae9097e15c1980f3 (DEFAULT_TABLE table=default.Table_375c4a1643654cb09b0e80840c461183 [id=e32e58d1c89d4de9ae27afd4685ba313]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.729442 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 760f2b8862604d5fae9097e15c1980f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.730851 27496 tablet_service.cc:1468] Processing CreateTablet for tablet 5962685d79044081a75532e19bfa2bce (DEFAULT_TABLE table=default.Table_260dc7423c364e949e0f0a46af6cf67d [id=ef255544a52d4b46b4ef88d2c732d2a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.731195 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5962685d79044081a75532e19bfa2bce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.732020 27494 tablet_service.cc:1468] Processing CreateTablet for tablet 0c59083353b34b3c97dd634742350a9b (DEFAULT_TABLE table=default.Table_6f2ad684529148418295c9b463e0a0f7 [id=28f0772fb64f4c248e4a86180c976d25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.732317 27494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c59083353b34b3c97dd634742350a9b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:22.733542 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 14f97a610f25488eb510ba0ce966470d consensus state has no leader
I20250903 20:05:22.734884 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b368ef35ad84486787f041c9a392c027" } new_table_name: "default.Table_0d67f3bd51344527af6248e556039647" modify_external_catalogs: true
I20250903 20:05:22.744855 27489 tablet_service.cc:1515] Processing DeleteTablet for tablet 659a127405d341c7b074d3445854256f with delete_type TABLET_DATA_DELETED (replaced by tablet 443d77f2593b44afa3e5aae4c5b0a651) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:22.754540 27500 tablet_service.cc:1468] Processing CreateTablet for tablet 0ab15dd783ff49868d6f9707369a5d3a (DEFAULT_TABLE table=default.Table_b87f7ee9a2444f9892fcadaa791f93dc [id=35d21449d9e94f2eb2bd7f326ce4d043]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.748272 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 2478fe89414b4845b0ba377c1cde98b7 (DEFAULT_TABLE table=default.Table_c7ba7021073b42a8b57cb33054f9a141 [id=62966767ca30457f86722ec3ab6cbd32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.761761 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ab15dd783ff49868d6f9707369a5d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.763641 27684 ts_tablet_manager.cc:1918] T fbfabb6ff1714d2e921f45256b543759 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:22.766304 27502 tablet_service.cc:1468] Processing CreateTablet for tablet ddc3f18eeba240cf97986c20f3cf75fa (DEFAULT_TABLE table=default.Table_5fec15415a2b4c22826e4038aaf75aa9 [id=1d7cd268a3494bf1a21d9d02c8bd0bf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.772194 27356 tablet_service.cc:1468] Processing CreateTablet for tablet ddc3f18eeba240cf97986c20f3cf75fa (DEFAULT_TABLE table=default.Table_5fec15415a2b4c22826e4038aaf75aa9 [id=1d7cd268a3494bf1a21d9d02c8bd0bf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.772747 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddc3f18eeba240cf97986c20f3cf75fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.775210 27349 tablet_service.cc:1468] Processing CreateTablet for tablet 44fef873a0644dd988322847f333ec95 (DEFAULT_TABLE table=default.Table_71f1cf3f76a8469695fdc49c38ba8771 [id=9cba15e1bcf04d03aadd26976221f9ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.775628 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44fef873a0644dd988322847f333ec95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.777117 27219 tablet_service.cc:1468] Processing CreateTablet for tablet ddc3f18eeba240cf97986c20f3cf75fa (DEFAULT_TABLE table=default.Table_5fec15415a2b4c22826e4038aaf75aa9 [id=1d7cd268a3494bf1a21d9d02c8bd0bf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.777558 27219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddc3f18eeba240cf97986c20f3cf75fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.780627 27684 log.cc:1199] T fbfabb6ff1714d2e921f45256b543759 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/fbfabb6ff1714d2e921f45256b543759
I20250903 20:05:22.784073 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 44fef873a0644dd988322847f333ec95 (DEFAULT_TABLE table=default.Table_71f1cf3f76a8469695fdc49c38ba8771 [id=9cba15e1bcf04d03aadd26976221f9ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.784565 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44fef873a0644dd988322847f333ec95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.785815 27348 tablet_service.cc:1468] Processing CreateTablet for tablet 6b1fbefaabc540d8a3db2a86f4920044 (DEFAULT_TABLE table=default.Table_8a3e1bffc7f340f1b4b6d7dcb66fdcfb [id=789b21c529784f3fb7097ec6e7ba9553]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.786267 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b1fbefaabc540d8a3db2a86f4920044. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.787528 27224 tablet_service.cc:1468] Processing CreateTablet for tablet 6b1fbefaabc540d8a3db2a86f4920044 (DEFAULT_TABLE table=default.Table_8a3e1bffc7f340f1b4b6d7dcb66fdcfb [id=789b21c529784f3fb7097ec6e7ba9553]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.787901 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b1fbefaabc540d8a3db2a86f4920044. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.788861 27357 tablet_service.cc:1468] Processing CreateTablet for tablet d7d37849132d4420bb5fcb57c11cfb6b (DEFAULT_TABLE table=default.Table_8d0b63b0e60f4b2ba191abfd7f8680b4 [id=ac07acc78bc74351a8dccf6dc0ebc4a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.789230 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7d37849132d4420bb5fcb57c11cfb6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.790907 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2478fe89414b4845b0ba377c1cde98b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.792513 27358 tablet_service.cc:1468] Processing CreateTablet for tablet 5934304e4ebd46f0a04e7c58c6607379 (DEFAULT_TABLE table=default.Table_b880e180787d4987961ba1ef18c3ca8f [id=3a9c97d38a584650bdc8af85cd849932]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.792883 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5934304e4ebd46f0a04e7c58c6607379. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.793967 27213 tablet_service.cc:1468] Processing CreateTablet for tablet d7d37849132d4420bb5fcb57c11cfb6b (DEFAULT_TABLE table=default.Table_8d0b63b0e60f4b2ba191abfd7f8680b4 [id=ac07acc78bc74351a8dccf6dc0ebc4a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.794061 27503 tablet_service.cc:1468] Processing CreateTablet for tablet 44fef873a0644dd988322847f333ec95 (DEFAULT_TABLE table=default.Table_71f1cf3f76a8469695fdc49c38ba8771 [id=9cba15e1bcf04d03aadd26976221f9ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.794405 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7d37849132d4420bb5fcb57c11cfb6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.802047 27684 ts_tablet_manager.cc:1939] T fbfabb6ff1714d2e921f45256b543759 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:22.802778 27244 raft_consensus.cc:1273] T 5fa7cc0c03234223af382982ca737e27 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Refusing update from remote peer 51578fb7bceb4e119b3ab1e58c4f5528: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 20:05:22.802762 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 5934304e4ebd46f0a04e7c58c6607379 (DEFAULT_TABLE table=default.Table_b880e180787d4987961ba1ef18c3ca8f [id=3a9c97d38a584650bdc8af85cd849932]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.803524 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5934304e4ebd46f0a04e7c58c6607379. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.803416 28298 consensus_queue.cc:1035] T 5fa7cc0c03234223af382982ca737e27 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:22.807945 27349 tablet_service.cc:1468] Processing CreateTablet for tablet 7baeb72c8bd440c39c07f34a6cb5e3ea (DEFAULT_TABLE table=default.Table_19e55aadaad1461e81d6b242dfa41c39 [id=9da1236f96c24c029aae89acadf41154]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.808449 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7baeb72c8bd440c39c07f34a6cb5e3ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.809140 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddc3f18eeba240cf97986c20f3cf75fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.812740 27498 tablet_service.cc:1468] Processing CreateTablet for tablet 5934304e4ebd46f0a04e7c58c6607379 (DEFAULT_TABLE table=default.Table_b880e180787d4987961ba1ef18c3ca8f [id=3a9c97d38a584650bdc8af85cd849932]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.817528 27210 tablet_service.cc:1468] Processing CreateTablet for tablet 7baeb72c8bd440c39c07f34a6cb5e3ea (DEFAULT_TABLE table=default.Table_19e55aadaad1461e81d6b242dfa41c39 [id=9da1236f96c24c029aae89acadf41154]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.818006 27210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7baeb72c8bd440c39c07f34a6cb5e3ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.819157 28340 raft_consensus.cc:491] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:22.819499 28340 raft_consensus.cc:513] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:22.820955 28340 leader_election.cc:290] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:22.821827 27321 leader_election.cc:343] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 9a76a17b64a74fbfbb4f2e92a02ae1cd
I20250903 20:05:22.823423 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44fef873a0644dd988322847f333ec95. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:22.826579 27462 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.024s sys 0.055s
I20250903 20:05:22.827589 27348 tablet_service.cc:1468] Processing CreateTablet for tablet c50e887828514d15935312d2e573f8ff (DEFAULT_TABLE table=default.Table_444ca50a57904488bdecef2ff5522be1 [id=d130af2954954522bca7e2d71c4e09d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.827842 27211 tablet_service.cc:1468] Processing CreateTablet for tablet c50e887828514d15935312d2e573f8ff (DEFAULT_TABLE table=default.Table_444ca50a57904488bdecef2ff5522be1 [id=d130af2954954522bca7e2d71c4e09d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.828011 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c50e887828514d15935312d2e573f8ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.828239 27211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c50e887828514d15935312d2e573f8ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.829749 27489 tablet_service.cc:1468] Processing CreateTablet for tablet d7d37849132d4420bb5fcb57c11cfb6b (DEFAULT_TABLE table=default.Table_8d0b63b0e60f4b2ba191abfd7f8680b4 [id=ac07acc78bc74351a8dccf6dc0ebc4a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.831787 27499 tablet_service.cc:1468] Processing CreateTablet for tablet 6b1fbefaabc540d8a3db2a86f4920044 (DEFAULT_TABLE table=default.Table_8a3e1bffc7f340f1b4b6d7dcb66fdcfb [id=789b21c529784f3fb7097ec6e7ba9553]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.836352 27498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5934304e4ebd46f0a04e7c58c6607379. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.839161 27356 tablet_service.cc:1468] Processing CreateTablet for tablet 443d77f2593b44afa3e5aae4c5b0a651 (DEFAULT_TABLE table=default.Table_30721ff3b68e4c08bdb5845184d8ada7 [id=c807184f006b4eac8c34c84733b6a5fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.839691 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 443d77f2593b44afa3e5aae4c5b0a651. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:22.852064 27321 consensus_peers.cc:489] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Couldn't send request to peer 89f8f0ee68b940db9ce250c0604f934a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c543ac4df33429b9787c6d32bf1ebeb. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:22.852722 27488 tablet_service.cc:1468] Processing CreateTablet for tablet 7baeb72c8bd440c39c07f34a6cb5e3ea (DEFAULT_TABLE table=default.Table_19e55aadaad1461e81d6b242dfa41c39 [id=9da1236f96c24c029aae89acadf41154]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.853278 27497 tablet_service.cc:1468] Processing CreateTablet for tablet c50e887828514d15935312d2e573f8ff (DEFAULT_TABLE table=default.Table_444ca50a57904488bdecef2ff5522be1 [id=d130af2954954522bca7e2d71c4e09d4]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:22.853813 27462 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.506s user 0.025s sys 0.055s
I20250903 20:05:22.854012 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 649) took 1129 ms. Trace:
I20250903 20:05:22.854028 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 443d77f2593b44afa3e5aae4c5b0a651 (DEFAULT_TABLE table=default.Table_30721ff3b68e4c08bdb5845184d8ada7 [id=c807184f006b4eac8c34c84733b6a5fe]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:22.854228 27320 consensus_peers.cc:489] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9129a3e773a4fecb28ceee27778a294. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:22.854408 27684 rpcz_store.cc:276] 0903 20:05:21.695685 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.695737 (+ 52us) service_pool.cc:225] Handling call
0903 20:05:22.141380 (+445643us) ts_tablet_manager.cc:1145] Deleting tablet fbfabb6ff1714d2e921f45256b543759
0903 20:05:22.141392 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:22.402655 (+261263us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:22.759303 (+356648us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:22.824680 (+ 65377us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":445428}
I20250903 20:05:22.854668 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 443d77f2593b44afa3e5aae4c5b0a651. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.855237 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet fbfabb6ff1714d2e921f45256b543759 (table default.Table_aab58ad5394041179974552994469305 [id=1889dee4122f4e69ac2e3d22fb4126a6]) successfully deleted
I20250903 20:05:22.855638 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a76a17b64a74fbfbb4f2e92a02ae1cd" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:22.855819 27380 raft_consensus.cc:2466] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:22.855912 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 651) took 1025 ms. Trace:
I20250903 20:05:22.856088 27684 rpcz_store.cc:276] 0903 20:05:21.829968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.830718 (+ 750us) service_pool.cc:225] Handling call
0903 20:05:22.855650 (+1024932us) ts_tablet_manager.cc:1145] Deleting tablet dbfdab4332234b2a93fd3c901a9cb645
0903 20:05:22.855682 (+ 32us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:22.855834 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":981256}
W20250903 20:05:22.856793 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet dbfdab4332234b2a93fd3c901a9cb645 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dbfdab4332234b2a93fd3c901a9cb645 already in progress: creating tablet
I20250903 20:05:22.857298 27684 tablet_replica.cc:331] T 369c750ff5bf449e989d95ffd89497d4 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:22.857695 27684 raft_consensus.cc:2241] T 369c750ff5bf449e989d95ffd89497d4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:22.858057 27684 raft_consensus.cc:2270] T 369c750ff5bf449e989d95ffd89497d4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:22.858371 27462 raft_consensus.cc:685] T 369c750ff5bf449e989d95ffd89497d4 P e72ef1dbcc5143cf9f04a808f101c80f: failed to submit failure detected task: Service unavailable: Thread pool token was shut down
I20250903 20:05:22.859515 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 443d77f2593b44afa3e5aae4c5b0a651 (DEFAULT_TABLE table=default.Table_30721ff3b68e4c08bdb5845184d8ada7 [id=c807184f006b4eac8c34c84733b6a5fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:22.865000 28396 raft_consensus.cc:491] T 07565ea2b5f34dcfb407ff3bfde17243 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:22.865425 28396 raft_consensus.cc:513] T 07565ea2b5f34dcfb407ff3bfde17243 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:22.866917 28396 leader_election.cc:290] T 07565ea2b5f34dcfb407ff3bfde17243 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:22.873584 28412 tablet.cc:1722] T 5fa7cc0c03234223af382982ca737e27 P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:22.890385 27489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7d37849132d4420bb5fcb57c11cfb6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.890385 27692 consensus_queue.cc:260] T 5fa7cc0c03234223af382982ca737e27 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:22.890956 27692 ts_tablet_manager.cc:1428] T 5fa7cc0c03234223af382982ca737e27 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.208s user 0.000s sys 0.002s
I20250903 20:05:22.896251 28413 tablet.cc:1722] T 5fa7cc0c03234223af382982ca737e27 P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:22.897318 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c50e887828514d15935312d2e573f8ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.901558 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b1fbefaabc540d8a3db2a86f4920044. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.907997 27522 raft_consensus.cc:3058] T 5fa7cc0c03234223af382982ca737e27 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:22.908977 27692 tablet_bootstrap.cc:492] T ac1b0dc95fca49b4af6697c0a70e9730 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:22.910775 27522 raft_consensus.cc:1273] T 5fa7cc0c03234223af382982ca737e27 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Refusing update from remote peer 51578fb7bceb4e119b3ab1e58c4f5528: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 20:05:22.911664 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a76a17b64a74fbfbb4f2e92a02ae1cd" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:22.911823 27524 raft_consensus.cc:2466] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
W20250903 20:05:22.911854 27321 consensus_peers.cc:489] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Couldn't send request to peer 89f8f0ee68b940db9ce250c0604f934a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57a1167457b84ea8a493be232b1d088a. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:22.913022 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7baeb72c8bd440c39c07f34a6cb5e3ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.913988 27247 raft_consensus.cc:1273] T 6238740ce1894e899be9a40321f0ab73 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Refusing update from remote peer 51578fb7bceb4e119b3ab1e58c4f5528: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 20:05:22.914539 28315 consensus_queue.cc:1035] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:22.918459 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 443d77f2593b44afa3e5aae4c5b0a651. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:22.922896 28315 raft_consensus.cc:491] T ac1b0dc95fca49b4af6697c0a70e9730 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:22.923089 28315 raft_consensus.cc:513] T ac1b0dc95fca49b4af6697c0a70e9730 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:22.923878 28315 leader_election.cc:290] T ac1b0dc95fca49b4af6697c0a70e9730 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:22.924621 27244 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac1b0dc95fca49b4af6697c0a70e9730" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:22.924835 27244 raft_consensus.cc:2466] T ac1b0dc95fca49b4af6697c0a70e9730 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:22.925029 27684 ts_tablet_manager.cc:1905] T 369c750ff5bf449e989d95ffd89497d4 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:22.925514 27321 leader_election.cc:304] T ac1b0dc95fca49b4af6697c0a70e9730 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:22.925791 28315 raft_consensus.cc:2802] T ac1b0dc95fca49b4af6697c0a70e9730 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:22.925853 28315 raft_consensus.cc:491] T ac1b0dc95fca49b4af6697c0a70e9730 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:22.925905 28315 raft_consensus.cc:3058] T ac1b0dc95fca49b4af6697c0a70e9730 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:22.928649 28315 raft_consensus.cc:513] T ac1b0dc95fca49b4af6697c0a70e9730 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:22.930727 27244 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac1b0dc95fca49b4af6697c0a70e9730" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:22.930923 27244 raft_consensus.cc:3058] T ac1b0dc95fca49b4af6697c0a70e9730 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:22.934303 28315 leader_election.cc:290] T ac1b0dc95fca49b4af6697c0a70e9730 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:22.935868 28441 tablet.cc:1722] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:22.938020 27244 raft_consensus.cc:2466] T ac1b0dc95fca49b4af6697c0a70e9730 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:22.938446 27685 ts_tablet_manager.cc:1918] T 369c750ff5bf449e989d95ffd89497d4 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:22.938527 27321 leader_election.cc:304] T ac1b0dc95fca49b4af6697c0a70e9730 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:22.938702 27685 log.cc:1199] T 369c750ff5bf449e989d95ffd89497d4 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/369c750ff5bf449e989d95ffd89497d4
I20250903 20:05:22.939038 27685 ts_tablet_manager.cc:1939] T 369c750ff5bf449e989d95ffd89497d4 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:22.939383 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 511) took 4038 ms. Trace:
I20250903 20:05:22.939512 27685 rpcz_store.cc:276] 0903 20:05:18.900616 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:18.900968 (+ 352us) service_pool.cc:225] Handling call
0903 20:05:22.440660 (+3539692us) ts_tablet_manager.cc:1145] Deleting tablet 369c750ff5bf449e989d95ffd89497d4
0903 20:05:22.440669 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:22.571452 (+130783us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:22.938326 (+366874us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:22.939368 (+ 1042us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3530162}
I20250903 20:05:22.940083 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 369c750ff5bf449e989d95ffd89497d4 (table default.Table_b0b8582c6c054baab4688a30cfe5f767 [id=feefb00b97c34fa0a5ed8af66004830e]) successfully deleted
I20250903 20:05:22.940510 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 522) took 3942 ms. Trace:
I20250903 20:05:22.940649 27685 rpcz_store.cc:276] 0903 20:05:18.997945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:19.000747 (+ 2802us) service_pool.cc:225] Handling call
0903 20:05:22.940397 (+3939650us) ts_tablet_manager.cc:1145] Deleting tablet f1c3c8e46a2b40ccb8d5bb61843cecd1
0903 20:05:22.940407 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:22.940496 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3939399}
I20250903 20:05:22.940831 28298 raft_consensus.cc:2802] T ac1b0dc95fca49b4af6697c0a70e9730 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:22.940932 28298 raft_consensus.cc:695] T ac1b0dc95fca49b4af6697c0a70e9730 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
W20250903 20:05:22.941124 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet f1c3c8e46a2b40ccb8d5bb61843cecd1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1c3c8e46a2b40ccb8d5bb61843cecd1 already in progress: creating tablet
I20250903 20:05:22.941170 28298 consensus_queue.cc:237] T ac1b0dc95fca49b4af6697c0a70e9730 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:22.941406 27685 tablet_replica.cc:331] T bf2545037c214bb1b5060575e3cdf81a P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:22.941567 27685 raft_consensus.cc:2241] T bf2545037c214bb1b5060575e3cdf81a P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:22.941690 27685 raft_consensus.cc:2270] T bf2545037c214bb1b5060575e3cdf81a P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:22.942219 27685 ts_tablet_manager.cc:1905] T bf2545037c214bb1b5060575e3cdf81a P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:22.942252 28413 tablet.cc:1722] T 6238740ce1894e899be9a40321f0ab73 P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:22.964733 27062 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_10a7df6626f24353ae6661cbbdf3efb0 [id=416ff29f66d14564826a6734dde1c604] alter complete (version 4)
I20250903 20:05:22.965903 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet cb4d99139516405bb65c7480ddf259f1 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d4b8d4bf29e408cbdf9557c8b0d8a4b) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:22.967178 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_cb368db83ae64dc784cc454f9224c3e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:22.968058 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb368db83ae64dc784cc454f9224c3e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:22.969143 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07565ea2b5f34dcfb407ff3bfde17243" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:22.969363 27380 raft_consensus.cc:2466] T 07565ea2b5f34dcfb407ff3bfde17243 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:22.969450 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_a6683ce4dd5d4470a5b08bddd66ee17f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:22.970192 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6683ce4dd5d4470a5b08bddd66ee17f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:22.971318 27247 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07565ea2b5f34dcfb407ff3bfde17243" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:22.971522 27247 raft_consensus.cc:2466] T 07565ea2b5f34dcfb407ff3bfde17243 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
W20250903 20:05:22.975374 27320 consensus_peers.cc:489] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f78a951bb024cf290116b0fe438ddb4. This is attempt 1: this message will repeat every 5th retry.
W20250903 20:05:22.975834 27320 consensus_peers.cc:489] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c543ac4df33429b9787c6d32bf1ebeb. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:22.980458 27687 ts_tablet_manager.cc:1918] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:22.989106 28245 raft_consensus.cc:491] T cb4d99139516405bb65c7480ddf259f1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:22.989274 28245 raft_consensus.cc:513] T cb4d99139516405bb65c7480ddf259f1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:22.989902 28245 leader_election.cc:290] T cb4d99139516405bb65c7480ddf259f1 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:22.991947 27059 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 5fa7cc0c03234223af382982ca737e27 (table default.Table_30f001eba314494d93e8ecac2f0bc3dc [id=66eb535587dc49e5b1ee72124cca0374]). Expected version 1 got 0
I20250903 20:05:22.992308 27059 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet c8a293c6fa034e9fb29682439cb3bcc2 (table default.Table_397211088a974a09bd7f031d5c7ca1cc [id=5cfc8827e4ca4f4a94424c48d6f71eb5]). Expected version 5 got 4
I20250903 20:05:22.994957 27685 ts_tablet_manager.cc:1918] T bf2545037c214bb1b5060575e3cdf81a P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:22.995098 27685 log.cc:1199] T bf2545037c214bb1b5060575e3cdf81a P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/bf2545037c214bb1b5060575e3cdf81a
I20250903 20:05:22.994943 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_7532aa050a154ef1824ea9f89f029692"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:22.995436 27685 ts_tablet_manager.cc:1939] T bf2545037c214bb1b5060575e3cdf81a P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:22.995925 27320 leader_election.cc:304] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:22.996382 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 533) took 3946 ms. Trace:
I20250903 20:05:22.996369 28315 consensus_queue.cc:1035] T 5fa7cc0c03234223af382982ca737e27 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:22.996402 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_81922f27b2de428ba78422c5be13fcd9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:22.996460 27685 rpcz_store.cc:276] 0903 20:05:19.049599 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:19.085304 (+ 35705us) service_pool.cc:225] Handling call
0903 20:05:22.941366 (+3856062us) ts_tablet_manager.cc:1145] Deleting tablet bf2545037c214bb1b5060575e3cdf81a
0903 20:05:22.941375 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:22.983784 (+ 42409us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:22.994609 (+ 10825us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:22.996365 (+ 1756us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3803322}
I20250903 20:05:22.980578 27687 log.cc:1199] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/1fef9400227449ed892daff588999caa
W20250903 20:05:22.997432 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4fa886f5be0f489290921e61dcaa4916 consensus state has no leader
I20250903 20:05:22.997851 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_9fa3fddcea2e482baf53322b4072b3b6" } new_table_name: "default.Table_c1366331d88b435486be5218d2777500" modify_external_catalogs: true
I20250903 20:05:22.998791 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 535) took 3803 ms. Trace:
I20250903 20:05:22.999357 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 09fa32312c9648fbbd6fd327e9738066
I20250903 20:05:22.999667 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb4d99139516405bb65c7480ddf259f1" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:23.000051 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 5253d82a41494ce59bb96928179dfdd9
W20250903 20:05:23.000484 27184 leader_election.cc:343] T cb4d99139516405bb65c7480ddf259f1 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:23.000754 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet cd6d6bb514134bdf8e3bc0aaad180732
I20250903 20:05:23.001734 27687 ts_tablet_manager.cc:1939] T 1fef9400227449ed892daff588999caa P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:23.001915 28340 raft_consensus.cc:2802] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:23.002022 28340 raft_consensus.cc:491] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:23.002103 28340 raft_consensus.cc:3058] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:23.004091 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb4d99139516405bb65c7480ddf259f1" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:23.004611 27183 leader_election.cc:343] T cb4d99139516405bb65c7480ddf259f1 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:23.004719 27183 leader_election.cc:304] T cb4d99139516405bb65c7480ddf259f1 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:23.005091 28245 raft_consensus.cc:2747] T cb4d99139516405bb65c7480ddf259f1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:23.005353 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 09fa32312c9648fbbd6fd327e9738066 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:23.005651 27462 leader_election.cc:343] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 9a76a17b64a74fbfbb4f2e92a02ae1cd
I20250903 20:05:23.007006 27488 tablet_service.cc:1515] Processing DeleteTablet for tablet 5253d82a41494ce59bb96928179dfdd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:23.007288 27462 leader_election.cc:304] T 07565ea2b5f34dcfb407ff3bfde17243 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:23.007499 28299 raft_consensus.cc:2802] T 07565ea2b5f34dcfb407ff3bfde17243 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:23.007560 28299 raft_consensus.cc:491] T 07565ea2b5f34dcfb407ff3bfde17243 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:23.007615 28299 raft_consensus.cc:3058] T 07565ea2b5f34dcfb407ff3bfde17243 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:23.010308 27684 ts_tablet_manager.cc:1918] T 369c750ff5bf449e989d95ffd89497d4 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:23.010180 28299 raft_consensus.cc:513] T 07565ea2b5f34dcfb407ff3bfde17243 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:23.011209 28299 leader_election.cc:290] T 07565ea2b5f34dcfb407ff3bfde17243 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:23.011585 27462 leader_election.cc:304] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:23.011888 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 443d77f2593b44afa3e5aae4c5b0a651
I20250903 20:05:23.012005 28396 raft_consensus.cc:2802] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:23.012076 28396 raft_consensus.cc:491] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:23.012133 28396 raft_consensus.cc:3058] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:23.012357 27692 tablet_bootstrap.cc:654] T ac1b0dc95fca49b4af6697c0a70e9730 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:23.016994 28396 raft_consensus.cc:513] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:23.017591 28396 leader_election.cc:290] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:23.010449 27684 log.cc:1199] T 369c750ff5bf449e989d95ffd89497d4 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/369c750ff5bf449e989d95ffd89497d4
I20250903 20:05:23.018318 27684 ts_tablet_manager.cc:1939] T 369c750ff5bf449e989d95ffd89497d4 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:23.019234 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 650) took 1189 ms. Trace:
I20250903 20:05:23.019309 27684 rpcz_store.cc:276] 0903 20:05:21.829821 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.830453 (+ 632us) service_pool.cc:225] Handling call
0903 20:05:22.857149 (+1026696us) ts_tablet_manager.cc:1145] Deleting tablet 369c750ff5bf449e989d95ffd89497d4
0903 20:05:22.857161 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:22.963947 (+106786us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.010160 (+ 46213us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.019213 (+ 9053us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":950524}
I20250903 20:05:23.019940 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 369c750ff5bf449e989d95ffd89497d4 (table default.Table_b0b8582c6c054baab4688a30cfe5f767 [id=feefb00b97c34fa0a5ed8af66004830e]) successfully deleted
I20250903 20:05:23.020516 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 662) took 1037 ms. Trace:
I20250903 20:05:23.020640 27684 rpcz_store.cc:276] 0903 20:05:21.983447 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.984069 (+ 622us) service_pool.cc:225] Handling call
0903 20:05:23.020279 (+1036210us) ts_tablet_manager.cc:1145] Deleting tablet 4157633525034a019fd6a80185892cec
0903 20:05:23.020292 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.020494 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1035915}
W20250903 20:05:23.021255 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 4157633525034a019fd6a80185892cec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4157633525034a019fd6a80185892cec already in progress: creating tablet
I20250903 20:05:23.022034 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 663) took 1038 ms. Trace:
I20250903 20:05:23.022240 27684 rpcz_store.cc:276] 0903 20:05:21.983544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.984390 (+ 846us) service_pool.cc:225] Handling call
0903 20:05:23.021543 (+1037153us) ts_tablet_manager.cc:1145] Deleting tablet 36e1061915574407b6e7695be8b1e55a
0903 20:05:23.021826 (+ 283us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.021992 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1036955}
W20250903 20:05:23.022950 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 36e1061915574407b6e7695be8b1e55a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36e1061915574407b6e7695be8b1e55a already in progress: creating tablet
I20250903 20:05:23.023579 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 664) took 1039 ms. Trace:
I20250903 20:05:23.023749 27684 rpcz_store.cc:276] 0903 20:05:21.983615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.984619 (+ 1004us) service_pool.cc:225] Handling call
0903 20:05:23.023383 (+1038764us) ts_tablet_manager.cc:1145] Deleting tablet 17e7deb74389448db37289563b86c253
0903 20:05:23.023395 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.023538 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1038564}
W20250903 20:05:23.024363 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 17e7deb74389448db37289563b86c253 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17e7deb74389448db37289563b86c253 already in progress: creating tablet
I20250903 20:05:23.024917 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 665) took 1041 ms. Trace:
I20250903 20:05:23.025107 27684 rpcz_store.cc:276] 0903 20:05:21.983690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.984820 (+ 1130us) service_pool.cc:225] Handling call
0903 20:05:23.024725 (+1039905us) ts_tablet_manager.cc:1145] Deleting tablet cd92e3dafca6419f8eba7a4ef51a911e
0903 20:05:23.024760 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.024900 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1039737}
W20250903 20:05:23.025780 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet cd92e3dafca6419f8eba7a4ef51a911e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd92e3dafca6419f8eba7a4ef51a911e already in progress: creating tablet
I20250903 20:05:23.026427 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 666) took 1042 ms. Trace:
I20250903 20:05:23.030882 27692 tablet_bootstrap.cc:492] T ac1b0dc95fca49b4af6697c0a70e9730 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:23.031052 27692 ts_tablet_manager.cc:1397] T ac1b0dc95fca49b4af6697c0a70e9730 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.122s user 0.003s sys 0.003s
I20250903 20:05:23.031955 27692 raft_consensus.cc:357] T ac1b0dc95fca49b4af6697c0a70e9730 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:23.032146 27692 raft_consensus.cc:383] T ac1b0dc95fca49b4af6697c0a70e9730 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:23.032202 27692 raft_consensus.cc:738] T ac1b0dc95fca49b4af6697c0a70e9730 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:23.032437 27692 consensus_queue.cc:260] T ac1b0dc95fca49b4af6697c0a70e9730 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:23.033301 27695 tablet_bootstrap.cc:492] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:23.033480 27695 ts_tablet_manager.cc:1397] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.348s user 0.096s sys 0.119s
I20250903 20:05:23.034261 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7b5eb542da2845c79bd273f704aa9159" } modify_external_catalogs: true
I20250903 20:05:23.034314 27695 raft_consensus.cc:357] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.034492 27695 raft_consensus.cc:383] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:23.034547 27695 raft_consensus.cc:738] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:23.034512 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7b5eb542da2845c79bd273f704aa9159" } modify_external_catalogs: true
I20250903 20:05:23.034812 27695 consensus_queue.cc:260] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.035358 27695 ts_tablet_manager.cc:1428] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20250903 20:05:23.036741 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a4fb5fb54cb740698336598c09b58699 consensus state has no leader
I20250903 20:05:23.037178 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3f5617eea30e4038945f21ba8ca6be65" } new_table_name: "default.Table_20e9f856d5e24febbdb3b95098a17f0c" modify_external_catalogs: true
I20250903 20:05:23.037395 27695 tablet_bootstrap.cc:492] T fe902f1b58924be1ac3a5585413f7789 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:23.039144 27695 tablet_bootstrap.cc:654] T fe902f1b58924be1ac3a5585413f7789 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:23.039373 27692 ts_tablet_manager.cc:1428] T ac1b0dc95fca49b4af6697c0a70e9730 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20250903 20:05:23.039690 27692 tablet_bootstrap.cc:492] T 6238740ce1894e899be9a40321f0ab73 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:23.043181 27695 tablet_bootstrap.cc:492] T fe902f1b58924be1ac3a5585413f7789 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:23.043323 27695 ts_tablet_manager.cc:1397] T fe902f1b58924be1ac3a5585413f7789 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20250903 20:05:23.026641 27684 rpcz_store.cc:276] 0903 20:05:21.983757 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.985014 (+ 1257us) service_pool.cc:225] Handling call
0903 20:05:23.026146 (+1041132us) ts_tablet_manager.cc:1145] Deleting tablet e9126e4a18c546c28927f22c9f16426e
0903 20:05:23.026178 (+ 32us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.026378 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1040978}
I20250903 20:05:23.000264 27685 rpcz_store.cc:276] 0903 20:05:19.195207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:19.195420 (+ 213us) service_pool.cc:225] Handling call
0903 20:05:22.998279 (+3802859us) ts_tablet_manager.cc:1145] Deleting tablet 64c4dd2645034a7eaf243f5474603f10
0903 20:05:22.998292 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:22.998679 (+ 387us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3798424}
I20250903 20:05:23.045059 27695 raft_consensus.cc:357] T fe902f1b58924be1ac3a5585413f7789 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.045239 27695 raft_consensus.cc:383] T fe902f1b58924be1ac3a5585413f7789 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:23.045300 27695 raft_consensus.cc:738] T fe902f1b58924be1ac3a5585413f7789 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:23.046135 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 569) took 2914 ms. Trace:
I20250903 20:05:23.046377 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 536) took 3846 ms. Trace:
I20250903 20:05:23.046809 27685 rpcz_store.cc:276] 0903 20:05:19.200072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:19.200381 (+ 309us) service_pool.cc:225] Handling call
0903 20:05:23.045910 (+3845529us) ts_tablet_manager.cc:1145] Deleting tablet 90baf83357494a0aae325908c118b471
0903 20:05:23.045922 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.046264 (+ 342us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3843410}
I20250903 20:05:23.046272 27687 rpcz_store.cc:276] 0903 20:05:20.131357 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.134117 (+ 2760us) service_pool.cc:225] Handling call
0903 20:05:22.356160 (+2222043us) ts_tablet_manager.cc:1145] Deleting tablet 1fef9400227449ed892daff588999caa
0903 20:05:22.356174 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:22.412741 (+ 56567us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:22.980286 (+567545us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.046106 (+ 65820us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2221701}
I20250903 20:05:23.047528 27687 tablet_replica.cc:331] T 8b3121fce1c44d16a0b083313dc67b7a P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:23.047726 27687 raft_consensus.cc:2241] T 8b3121fce1c44d16a0b083313dc67b7a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:23.047866 27687 raft_consensus.cc:2270] T 8b3121fce1c44d16a0b083313dc67b7a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:23.048394 27687 ts_tablet_manager.cc:1905] T 8b3121fce1c44d16a0b083313dc67b7a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:23.053772 27695 consensus_queue.cc:260] T fe902f1b58924be1ac3a5585413f7789 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.054420 27695 ts_tablet_manager.cc:1428] T fe902f1b58924be1ac3a5585413f7789 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20250903 20:05:23.054735 27695 tablet_bootstrap.cc:492] T cd92e3dafca6419f8eba7a4ef51a911e P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:23.055315 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 9767293447214a7eb68ed03d7373db7f
I20250903 20:05:23.058982 27695 tablet_bootstrap.cc:654] T cd92e3dafca6419f8eba7a4ef51a911e P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:23.074889 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ac1b0dc95fca49b4af6697c0a70e9730 consensus state has no leader
I20250903 20:05:23.076478 28340 raft_consensus.cc:513] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:23.077349 28340 leader_election.cc:290] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:23.077391 27065 catalog_manager.cc:5582] T ac1b0dc95fca49b4af6697c0a70e9730 P 51578fb7bceb4e119b3ab1e58c4f5528 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66). New cstate: current_term: 1 leader_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:23.077950 27065 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet ac1b0dc95fca49b4af6697c0a70e9730 (table default.Table_20e9f856d5e24febbdb3b95098a17f0c [id=6952319d78524829b3bc6c3de6425a67]). Expected version 1 got 0
I20250903 20:05:23.080186 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_c27464eb362a44268d9c085034dd3ff4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:23.081319 27321 leader_election.cc:343] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 9a76a17b64a74fbfbb4f2e92a02ae1cd
I20250903 20:05:23.082087 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 534) took 4026 ms. Trace:
I20250903 20:05:23.082268 27685 rpcz_store.cc:276] 0903 20:05:19.056095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:19.208806 (+152711us) service_pool.cc:225] Handling call
0903 20:05:23.081848 (+3873042us) ts_tablet_manager.cc:1145] Deleting tablet 8400e23873fd4be18de2998afc64a689
0903 20:05:23.081861 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.082034 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3792057}
I20250903 20:05:23.082973 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 543) took 3139 ms. Trace:
I20250903 20:05:23.083101 27685 rpcz_store.cc:276] 0903 20:05:19.943989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:19.944100 (+ 111us) service_pool.cc:225] Handling call
0903 20:05:23.082798 (+3138698us) ts_tablet_manager.cc:1145] Deleting tablet f31e3e5ea72940de953c7dd787515c62
0903 20:05:23.082807 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.082934 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3138419}
I20250903 20:05:23.083735 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 544) took 3138 ms. Trace:
I20250903 20:05:23.083827 27685 rpcz_store.cc:276] 0903 20:05:19.945720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:19.945789 (+ 69us) service_pool.cc:225] Handling call
0903 20:05:23.083559 (+3137770us) ts_tablet_manager.cc:1145] Deleting tablet e3e82f37c6fa4bebb0115e6087919a27
0903 20:05:23.083571 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.083696 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3137554}
I20250903 20:05:23.084431 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 545) took 3134 ms. Trace:
I20250903 20:05:23.084520 27685 rpcz_store.cc:276] 0903 20:05:19.949912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:19.949975 (+ 63us) service_pool.cc:225] Handling call
0903 20:05:23.084259 (+3134284us) ts_tablet_manager.cc:1145] Deleting tablet 3919f7f95a954f59b68f56d0051c0109
0903 20:05:23.084267 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.084396 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3133993}
I20250903 20:05:22.996942 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet bf2545037c214bb1b5060575e3cdf81a (table default.Table_143fea9b5e3d4acbb92adc441aeede47 [id=42c554f2f532414a88b50818e2dbb458]) successfully deleted
W20250903 20:05:23.085129 27052 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 88213.4us
I20250903 20:05:23.085927 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 1fef9400227449ed892daff588999caa (table default.Table_d3945d5a36d84041b35394333dc704f9 [id=5568f788d1414c83b4359b5adf28632a]) successfully deleted
I20250903 20:05:23.086122 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_53e3669346c9467e8e27c3a33c6e1660"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:23.086495 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 64c4dd2645034a7eaf243f5474603f10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64c4dd2645034a7eaf243f5474603f10 already in progress: creating tablet
I20250903 20:05:23.086593 27356 tablet_service.cc:1515] Processing DeleteTablet for tablet 5253d82a41494ce59bb96928179dfdd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:23.086772 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 90baf83357494a0aae325908c118b471 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90baf83357494a0aae325908c118b471 already in progress: creating tablet
I20250903 20:05:23.086828 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet 9767293447214a7eb68ed03d7373db7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:23 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:23.086892 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53e3669346c9467e8e27c3a33c6e1660 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:23.087003 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet cd6d6bb514134bdf8e3bc0aaad180732 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:23.086993 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 8400e23873fd4be18de2998afc64a689 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8400e23873fd4be18de2998afc64a689 already in progress: creating tablet
I20250903 20:05:23.087173 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 09fa32312c9648fbbd6fd327e9738066 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:23.087234 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet f31e3e5ea72940de953c7dd787515c62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f31e3e5ea72940de953c7dd787515c62 already in progress: creating tablet
W20250903 20:05:23.087443 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet e3e82f37c6fa4bebb0115e6087919a27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3e82f37c6fa4bebb0115e6087919a27 already in progress: creating tablet
W20250903 20:05:23.087667 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 3919f7f95a954f59b68f56d0051c0109 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3919f7f95a954f59b68f56d0051c0109 already in progress: creating tablet
I20250903 20:05:23.088140 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet 09fa32312c9648fbbd6fd327e9738066 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:23.088277 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 546) took 3132 ms. Trace:
I20250903 20:05:23.088364 27685 rpcz_store.cc:276] 0903 20:05:19.956179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:19.956250 (+ 71us) service_pool.cc:225] Handling call
0903 20:05:23.084931 (+3128681us) ts_tablet_manager.cc:1145] Deleting tablet bc98762790f441038972366b1cc2c55b
0903 20:05:23.084940 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.088260 (+ 3320us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3128426}
W20250903 20:05:23.088754 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet bc98762790f441038972366b1cc2c55b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc98762790f441038972366b1cc2c55b already in progress: creating tablet
I20250903 20:05:23.089015 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5bf171fc5f3d4fe7b00fb86fd3e60d1e" } new_table_name: "default.Table_8dd7940c3a2c406c96060423df335b7d" modify_external_catalogs: true
I20250903 20:05:23.089226 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet 5253d82a41494ce59bb96928179dfdd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:23.089356 27687 ts_tablet_manager.cc:1918] T 8b3121fce1c44d16a0b083313dc67b7a P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:23.089375 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 547) took 3128 ms. Trace:
I20250903 20:05:23.089465 27687 log.cc:1199] T 8b3121fce1c44d16a0b083313dc67b7a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/8b3121fce1c44d16a0b083313dc67b7a
I20250903 20:05:23.089429 27685 rpcz_store.cc:276] 0903 20:05:19.960523 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:19.960582 (+ 59us) service_pool.cc:225] Handling call
0903 20:05:23.089263 (+3128681us) ts_tablet_manager.cc:1145] Deleting tablet 314856bf269f4df7aa25f5a30cec0d5b
0903 20:05:23.089272 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.089360 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3128439}
I20250903 20:05:23.089800 27687 ts_tablet_manager.cc:1939] T 8b3121fce1c44d16a0b083313dc67b7a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
W20250903 20:05:23.089849 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 314856bf269f4df7aa25f5a30cec0d5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 314856bf269f4df7aa25f5a30cec0d5b already in progress: creating tablet
I20250903 20:05:23.090533 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 548) took 3109 ms. Trace:
W20250903 20:05:23.091169 27693 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.012s sys 0.037s
W20250903 20:05:23.091243 27693 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.528s user 0.012s sys 0.037s
I20250903 20:05:23.091295 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_a1bcf29f05be4350b110670fdbf011d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:23.091681 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet 443d77f2593b44afa3e5aae4c5b0a651 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:23.091883 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 443d77f2593b44afa3e5aae4c5b0a651 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:23.091923 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1bcf29f05be4350b110670fdbf011d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:23.092758 27693 tablet_bootstrap.cc:492] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:23.092878 27693 ts_tablet_manager.cc:1397] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.541s user 0.014s sys 0.037s
W20250903 20:05:23.093214 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81922f27b2de428ba78422c5be13fcd9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:23.093611 27693 raft_consensus.cc:357] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:23.093784 27693 raft_consensus.cc:383] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:23.093849 27693 raft_consensus.cc:738] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:23.094045 27693 consensus_queue.cc:260] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:23.094321 28298 raft_consensus.cc:491] T bf2545037c214bb1b5060575e3cdf81a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250903 20:05:23.094385 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7532aa050a154ef1824ea9f89f029692 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:23.094424 28298 raft_consensus.cc:513] T bf2545037c214bb1b5060575e3cdf81a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:23.095793 27321 leader_election.cc:343] T bf2545037c214bb1b5060575e3cdf81a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: bf2545037c214bb1b5060575e3cdf81a
I20250903 20:05:23.095942 28298 leader_election.cc:290] T bf2545037c214bb1b5060575e3cdf81a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:23.095973 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_00da0bb3fb0c4aa1b29a87d3c8844955" } new_table_name: "default.Table_59e7e14a9ba945bf9969e024a530cb07" modify_external_catalogs: true
I20250903 20:05:23.090620 27685 rpcz_store.cc:276] 0903 20:05:19.981099 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:19.981167 (+ 68us) service_pool.cc:225] Handling call
0903 20:05:23.090403 (+3109236us) ts_tablet_manager.cc:1145] Deleting tablet 514e382479014fa7bf1f404efff11c86
0903 20:05:23.090415 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.090514 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3108967}
I20250903 20:05:23.097745 27227 tablet_service.cc:1515] Processing DeleteTablet for tablet 9767293447214a7eb68ed03d7373db7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:23 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:23.097944 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet cd6d6bb514134bdf8e3bc0aaad180732 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:23.098073 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 514e382479014fa7bf1f404efff11c86 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 514e382479014fa7bf1f404efff11c86 already in progress: creating tablet
I20250903 20:05:23.098414 27693 ts_tablet_manager.cc:1428] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250903 20:05:23.098479 27685 tablet_replica.cc:331] T 07565ea2b5f34dcfb407ff3bfde17243 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:23.098832 27685 raft_consensus.cc:2241] T 07565ea2b5f34dcfb407ff3bfde17243 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:23.099006 27685 raft_consensus.cc:2270] T 07565ea2b5f34dcfb407ff3bfde17243 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:23.099352 27693 tablet_bootstrap.cc:492] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:23.101341 27685 ts_tablet_manager.cc:1905] T 07565ea2b5f34dcfb407ff3bfde17243 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:23.106307 27062 master_service.cc:948] ReplaceTablet: received request to replace tablet 02b3e262dd7841dfbb0520377eb94743 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:23.107626 27693 tablet_bootstrap.cc:654] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:23.108017 27685 ts_tablet_manager.cc:1918] T 07565ea2b5f34dcfb407ff3bfde17243 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:23.108120 27685 log.cc:1199] T 07565ea2b5f34dcfb407ff3bfde17243 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/07565ea2b5f34dcfb407ff3bfde17243
I20250903 20:05:23.108907 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 570) took 2977 ms. Trace:
I20250903 20:05:23.108989 27687 rpcz_store.cc:276] 0903 20:05:20.131490 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.132410 (+ 920us) service_pool.cc:225] Handling call
0903 20:05:23.047473 (+2915063us) ts_tablet_manager.cc:1145] Deleting tablet 8b3121fce1c44d16a0b083313dc67b7a
0903 20:05:23.047485 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.072008 (+ 24523us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.088605 (+ 16597us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.108881 (+ 20276us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2890514}
I20250903 20:05:23.109517 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 8b3121fce1c44d16a0b083313dc67b7a (table default.Table_41538df113834b45ae2213b105fb9706 [id=158a2fc69fce4c54917277cc90439126]) successfully deleted
I20250903 20:05:23.109849 27687 tablet_replica.cc:331] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:23.110031 27687 raft_consensus.cc:2241] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:23.110365 27687 pending_rounds.cc:70] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528: Trying to abort 1 pending ops.
I20250903 20:05:23.110411 27687 pending_rounds.cc:77] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384958480842752 op_type: NO_OP noop_request { }
I20250903 20:05:23.110503 27687 raft_consensus.cc:2887] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:23.110580 27687 raft_consensus.cc:2270] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:23.111117 27687 ts_tablet_manager.cc:1905] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:23.113911 28396 raft_consensus.cc:491] T 5fa7cc0c03234223af382982ca737e27 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:23.114065 28396 raft_consensus.cc:513] T 5fa7cc0c03234223af382982ca737e27 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:23.128163 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07565ea2b5f34dcfb407ff3bfde17243" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:23.128283 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a76a17b64a74fbfbb4f2e92a02ae1cd" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:23.128373 27380 raft_consensus.cc:3058] T 07565ea2b5f34dcfb407ff3bfde17243 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:23.128501 27381 raft_consensus.cc:2391] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
I20250903 20:05:23.130477 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07565ea2b5f34dcfb407ff3bfde17243" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:23.131575 28315 raft_consensus.cc:491] T 369c750ff5bf449e989d95ffd89497d4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:23.131691 28315 raft_consensus.cc:513] T 369c750ff5bf449e989d95ffd89497d4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:23.132231 28315 leader_election.cc:290] T 369c750ff5bf449e989d95ffd89497d4 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:23.132370 27380 raft_consensus.cc:2466] T 07565ea2b5f34dcfb407ff3bfde17243 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
W20250903 20:05:23.133152 27321 leader_election.cc:343] T 369c750ff5bf449e989d95ffd89497d4 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 369c750ff5bf449e989d95ffd89497d4
I20250903 20:05:23.137524 27693 tablet_bootstrap.cc:492] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:23.138192 27693 ts_tablet_manager.cc:1397] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
W20250903 20:05:23.139288 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c27464eb362a44268d9c085034dd3ff4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:23.139475 27693 raft_consensus.cc:357] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:23.140846 27693 raft_consensus.cc:383] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:23.141170 27693 raft_consensus.cc:738] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:23.141649 27693 consensus_queue.cc:260] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:23.143270 27693 ts_tablet_manager.cc:1428] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250903 20:05:23.143851 27693 tablet_bootstrap.cc:492] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:23.147568 27685 ts_tablet_manager.cc:1939] T 07565ea2b5f34dcfb407ff3bfde17243 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:23.148701 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 549) took 3132 ms. Trace:
I20250903 20:05:23.151875 27685 rpcz_store.cc:276] 0903 20:05:20.016598 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.016806 (+ 208us) service_pool.cc:225] Handling call
0903 20:05:23.098424 (+3081618us) ts_tablet_manager.cc:1145] Deleting tablet 07565ea2b5f34dcfb407ff3bfde17243
0903 20:05:23.098437 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.104780 (+ 6343us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.107901 (+ 3121us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.148675 (+ 40774us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3081347}
I20250903 20:05:23.154819 27693 tablet_bootstrap.cc:654] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:23.157155 27687 ts_tablet_manager.cc:1918] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:23.158833 27064 master_service.cc:948] ReplaceTablet: received request to replace tablet cb2f50683e904dd39ee6eb128717d38c from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:23.160993 27060 master_service.cc:948] ReplaceTablet: received request to replace tablet 8605bc329627438998812ed028327f53 from {username='slave'} at 127.0.0.1:45936
W20250903 20:05:23.161958 27066 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ef7dd1ec6fcc47ec81facd6ae0207b97 consensus state has no leader
I20250903 20:05:23.161573 27687 log.cc:1199] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/3f78a951bb024cf290116b0fe438ddb4
I20250903 20:05:23.163065 27687 ts_tablet_manager.cc:1939] T 3f78a951bb024cf290116b0fe438ddb4 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:23.163903 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 573) took 2767 ms. Trace:
I20250903 20:05:23.163991 27687 rpcz_store.cc:276] 0903 20:05:20.396614 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.396954 (+ 340us) service_pool.cc:225] Handling call
0903 20:05:23.109790 (+2712836us) ts_tablet_manager.cc:1145] Deleting tablet 3f78a951bb024cf290116b0fe438ddb4
0903 20:05:23.109802 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.144622 (+ 34820us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.157041 (+ 12419us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.163883 (+ 6842us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2705215}
I20250903 20:05:23.164444 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 3f78a951bb024cf290116b0fe438ddb4 (table default.Table_835ced1f8b194192809c3b6fbf4d1d96 [id=23f448924ec04ef08a541a14bd2ce5df]) successfully deleted
I20250903 20:05:23.166590 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 07565ea2b5f34dcfb407ff3bfde17243 (table default.Table_4810ae9efbde43bc8c237c6e135aa5aa [id=dbeb164ccbf544cf947af9bf656559c7]) successfully deleted
I20250903 20:05:23.166911 27685 tablet_replica.cc:331] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:23.167502 27685 raft_consensus.cc:2241] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:23.169528 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac1b0dc95fca49b4af6697c0a70e9730" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:23.169715 27524 raft_consensus.cc:3058] T ac1b0dc95fca49b4af6697c0a70e9730 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:23.169996 27462 leader_election.cc:343] T 07565ea2b5f34dcfb407ff3bfde17243 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: Tablet not RUNNING: STOPPED
W20250903 20:05:23.170159 27462 leader_election.cc:343] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 9a76a17b64a74fbfbb4f2e92a02ae1cd
I20250903 20:05:23.170583 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 667) took 1186 ms. Trace:
I20250903 20:05:23.170667 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 574) took 2764 ms. Trace:
I20250903 20:05:23.170732 27687 rpcz_store.cc:276] 0903 20:05:20.409547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.409625 (+ 78us) service_pool.cc:225] Handling call
0903 20:05:23.170368 (+2760743us) ts_tablet_manager.cc:1145] Deleting tablet 5f2eefeebbfc47249e2f72b7c52e905d
0903 20:05:23.170380 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.170647 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2746844}
I20250903 20:05:23.171234 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 575) took 2753 ms. Trace:
I20250903 20:05:23.171307 27687 rpcz_store.cc:276] 0903 20:05:20.418111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.418226 (+ 115us) service_pool.cc:225] Handling call
0903 20:05:23.171125 (+2752899us) ts_tablet_manager.cc:1145] Deleting tablet 4789bd72aa454b639df76bd5ad379850
0903 20:05:23.171132 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.171218 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2746345}
I20250903 20:05:23.171847 27693 tablet_bootstrap.cc:492] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:23.172257 27693 ts_tablet_manager.cc:1397] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.002s
I20250903 20:05:23.172405 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 580) took 2740 ms. Trace:
I20250903 20:05:23.172766 27687 rpcz_store.cc:276] 0903 20:05:20.432142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.433333 (+ 1191us) service_pool.cc:225] Handling call
0903 20:05:23.172306 (+2738973us) ts_tablet_manager.cc:1145] Deleting tablet 64d53f2071b0449ca61caa6c264a3641
0903 20:05:23.172314 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.172389 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2738758}
W20250903 20:05:23.173213 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 5f2eefeebbfc47249e2f72b7c52e905d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f2eefeebbfc47249e2f72b7c52e905d already in progress: creating tablet
I20250903 20:05:23.172999 27693 raft_consensus.cc:357] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.173545 27693 raft_consensus.cc:383] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250903 20:05:23.173539 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 4789bd72aa454b639df76bd5ad379850 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4789bd72aa454b639df76bd5ad379850 already in progress: creating tablet
I20250903 20:05:23.173630 27693 raft_consensus.cc:738] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
W20250903 20:05:23.173790 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 64d53f2071b0449ca61caa6c264a3641 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64d53f2071b0449ca61caa6c264a3641 already in progress: creating tablet
I20250903 20:05:23.173853 27693 consensus_queue.cc:260] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.170657 27684 rpcz_store.cc:276] 0903 20:05:21.983911 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.985195 (+ 1284us) service_pool.cc:225] Handling call
0903 20:05:23.170467 (+1185272us) ts_tablet_manager.cc:1145] Deleting tablet c49767d618be4fd18f30e5124ce0cb84
0903 20:05:23.170475 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.170562 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1185098}
I20250903 20:05:23.174161 27684 tablet_replica.cc:331] T 07565ea2b5f34dcfb407ff3bfde17243 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:23.174376 27684 raft_consensus.cc:2241] T 07565ea2b5f34dcfb407ff3bfde17243 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:23.174517 27684 raft_consensus.cc:2270] T 07565ea2b5f34dcfb407ff3bfde17243 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:23.174526 27693 ts_tablet_manager.cc:1428] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:23.174772 27693 tablet_bootstrap.cc:492] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
W20250903 20:05:23.174773 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b32c82c3d8c4420cb6e3a95129550076 consensus state has no leader
I20250903 20:05:23.175019 27684 ts_tablet_manager.cc:1905] T 07565ea2b5f34dcfb407ff3bfde17243 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:23.176465 28396 leader_election.cc:290] T 5fa7cc0c03234223af382982ca737e27 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:23.177496 27693 tablet_bootstrap.cc:654] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:23.178393 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 584) took 2745 ms. Trace:
I20250903 20:05:23.178459 27687 rpcz_store.cc:276] 0903 20:05:20.432550 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.434260 (+ 1710us) service_pool.cc:225] Handling call
0903 20:05:23.178285 (+2744025us) ts_tablet_manager.cc:1145] Deleting tablet 1bf570855b204b96acd070587c434463
0903 20:05:23.178293 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.178377 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2743853}
W20250903 20:05:23.178833 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 1bf570855b204b96acd070587c434463 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1bf570855b204b96acd070587c434463 already in progress: creating tablet
I20250903 20:05:23.182024 27488 tablet_service.cc:1515] Processing DeleteTablet for tablet 9767293447214a7eb68ed03d7373db7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:23 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:23.182108 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 588) took 2746 ms. Trace:
I20250903 20:05:23.182168 27687 rpcz_store.cc:276] 0903 20:05:20.435990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.436193 (+ 203us) service_pool.cc:225] Handling call
0903 20:05:23.181965 (+2745772us) ts_tablet_manager.cc:1145] Deleting tablet 259bc73ca9df47e9874692725a60bb1c
0903 20:05:23.181996 (+ 31us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.182091 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2745594}
W20250903 20:05:23.183432 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 259bc73ca9df47e9874692725a60bb1c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 259bc73ca9df47e9874692725a60bb1c already in progress: creating tablet
W20250903 20:05:23.183495 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet e9126e4a18c546c28927f22c9f16426e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9126e4a18c546c28927f22c9f16426e already in progress: creating tablet
W20250903 20:05:23.183708 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet c49767d618be4fd18f30e5124ce0cb84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c49767d618be4fd18f30e5124ce0cb84 already in progress: creating tablet
I20250903 20:05:23.184561 27687 tablet_replica.cc:331] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:23.184532 27523 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a76a17b64a74fbfbb4f2e92a02ae1cd" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:23.184749 27523 raft_consensus.cc:2391] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 51578fb7bceb4e119b3ab1e58c4f5528 in current term 1: Already voted for candidate e72ef1dbcc5143cf9f04a808f101c80f in this term.
I20250903 20:05:23.184885 27687 raft_consensus.cc:2241] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:23.185295 27687 raft_consensus.cc:2270] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:23.186352 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet cd6d6bb514134bdf8e3bc0aaad180732 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:23.186826 27687 ts_tablet_manager.cc:1905] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:23.187114 27692 tablet_bootstrap.cc:654] T 6238740ce1894e899be9a40321f0ab73 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:23.189088 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet cb2f50683e904dd39ee6eb128717d38c
I20250903 20:05:23.190459 27348 tablet_service.cc:1515] Processing DeleteTablet for tablet cb2f50683e904dd39ee6eb128717d38c with delete_type TABLET_DATA_DELETED (replaced by tablet 72fcb7b0cba34313a7dc2b8c29e120fc) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:23.190554 27064 catalog_manager.cc:6652] ReplaceTablet: tablet cb2f50683e904dd39ee6eb128717d38c deleted and replaced by tablet 72fcb7b0cba34313a7dc2b8c29e120fc
I20250903 20:05:23.187304 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac1b0dc95fca49b4af6697c0a70e9730" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:23.192153 27488 tablet_service.cc:1515] Processing DeleteTablet for tablet 443d77f2593b44afa3e5aae4c5b0a651 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:23.200857 27066 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet ac1b0dc95fca49b4af6697c0a70e9730 (table default.Table_20e9f856d5e24febbdb3b95098a17f0c [id=6952319d78524829b3bc6c3de6425a67]). Expected version 1 got 0
I20250903 20:05:23.203370 27488 tablet_service.cc:1515] Processing DeleteTablet for tablet cb2f50683e904dd39ee6eb128717d38c with delete_type TABLET_DATA_DELETED (replaced by tablet 72fcb7b0cba34313a7dc2b8c29e120fc) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:23.210083 27687 ts_tablet_manager.cc:1918] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:23.214577 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a7a9140280274ad2ac9a71b678fc8ca7" } new_table_name: "default.Table_f8346aab1d1f4850b85a74c2d00decca" modify_external_catalogs: true
I20250903 20:05:23.216382 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_5951c39f7b88401e867c14601b0c1ea1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:23.216969 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5951c39f7b88401e867c14601b0c1ea1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:23.218083 27687 log.cc:1199] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/57a1167457b84ea8a493be232b1d088a
I20250903 20:05:23.218205 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_5a979a57ff9e4e9796aa94e6398ead05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:23.218724 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a979a57ff9e4e9796aa94e6398ead05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:23.219401 27687 ts_tablet_manager.cc:1939] T 57a1167457b84ea8a493be232b1d088a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:23.219861 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_193a0ff511204b3b8130c17e783b51ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:23.220366 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_193a0ff511204b3b8130c17e783b51ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:23.227212 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 597) took 2739 ms. Trace:
I20250903 20:05:23.228356 27687 rpcz_store.cc:276] 0903 20:05:20.487398 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.487515 (+ 117us) service_pool.cc:225] Handling call
0903 20:05:23.184519 (+2697004us) ts_tablet_manager.cc:1145] Deleting tablet 57a1167457b84ea8a493be232b1d088a
0903 20:05:23.184527 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.200299 (+ 15772us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.209945 (+ 9646us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.227182 (+ 17237us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2696784}
I20250903 20:05:23.233948 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 57a1167457b84ea8a493be232b1d088a (table default.Table_b2a82a2c310140caaeac5d0dcb4bb61a [id=e2548ad469e54a41af7c0181357d2467]) successfully deleted
I20250903 20:05:23.234808 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 596) took 2748 ms. Trace:
I20250903 20:05:23.235365 27687 rpcz_store.cc:276] 0903 20:05:20.486663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.486839 (+ 176us) service_pool.cc:225] Handling call
0903 20:05:23.234636 (+2747797us) ts_tablet_manager.cc:1145] Deleting tablet 847d0a0a3d2b45938cec9058c77f6231
0903 20:05:23.234652 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.234783 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2738930}
W20250903 20:05:23.236234 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 847d0a0a3d2b45938cec9058c77f6231 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 847d0a0a3d2b45938cec9058c77f6231 already in progress: creating tablet
I20250903 20:05:23.236243 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b69c590ef46648439e74d958bf9ffdf3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:23.236963 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b69c590ef46648439e74d958bf9ffdf3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:23.236980 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 599) took 2706 ms. Trace:
I20250903 20:05:23.237550 27687 rpcz_store.cc:276] 0903 20:05:20.530867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.530991 (+ 124us) service_pool.cc:225] Handling call
0903 20:05:23.236814 (+2705823us) ts_tablet_manager.cc:1145] Deleting tablet 4e9a6a6e05464b7b97ae624f51402cf0
0903 20:05:23.236830 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.236957 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2702834}
W20250903 20:05:23.238291 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 4e9a6a6e05464b7b97ae624f51402cf0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e9a6a6e05464b7b97ae624f51402cf0 already in progress: creating tablet
I20250903 20:05:23.238575 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b1d2232da9bd4385b1b65355d3be0b4d" } new_table_name: "default.Table_b7fa325810b64d068d80bcb17c18ae5d" modify_external_catalogs: true
I20250903 20:05:23.239085 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 598) took 2716 ms. Trace:
I20250903 20:05:23.239528 27687 rpcz_store.cc:276] 0903 20:05:20.530390 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.530641 (+ 251us) service_pool.cc:225] Handling call
0903 20:05:23.238721 (+2708080us) ts_tablet_manager.cc:1145] Deleting tablet f2629f46704e4a0fa7922533928ca2ed
0903 20:05:23.238736 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.239054 (+ 318us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2589314}
W20250903 20:05:23.240494 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet f2629f46704e4a0fa7922533928ca2ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2629f46704e4a0fa7922533928ca2ed already in progress: creating tablet
I20250903 20:05:23.241057 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 601) took 2648 ms. Trace:
I20250903 20:05:23.241560 27687 rpcz_store.cc:276] 0903 20:05:20.592743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.598829 (+ 6086us) service_pool.cc:225] Handling call
0903 20:05:23.240911 (+2642082us) ts_tablet_manager.cc:1145] Deleting tablet cac38daf002c4ae698d72903a277a587
0903 20:05:23.240926 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.241036 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2586636}
W20250903 20:05:23.242439 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet cac38daf002c4ae698d72903a277a587 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cac38daf002c4ae698d72903a277a587 already in progress: creating tablet
I20250903 20:05:23.242947 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 600) took 2650 ms. Trace:
I20250903 20:05:23.243861 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_32ee815023e84a3dadbc1ec42d582b42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:23.244261 27687 rpcz_store.cc:276] 0903 20:05:20.592628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.598891 (+ 6263us) service_pool.cc:225] Handling call
0903 20:05:23.242839 (+2643948us) ts_tablet_manager.cc:1145] Deleting tablet 200c06f296b94eab94f35f312c5a443b
0903 20:05:23.242848 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.242931 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2575596}
W20250903 20:05:23.244635 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32ee815023e84a3dadbc1ec42d582b42 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:23.245203 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 200c06f296b94eab94f35f312c5a443b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 200c06f296b94eab94f35f312c5a443b already in progress: creating tablet
I20250903 20:05:23.245734 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 605) took 2558 ms. Trace:
I20250903 20:05:23.246227 27687 rpcz_store.cc:276] 0903 20:05:20.689926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.689996 (+ 70us) service_pool.cc:225] Handling call
0903 20:05:23.245593 (+2555597us) ts_tablet_manager.cc:1145] Deleting tablet 615f42e95bb24c8480f73d371fd63b9c
0903 20:05:23.245606 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.245714 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2554970}
W20250903 20:05:23.247112 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 615f42e95bb24c8480f73d371fd63b9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 615f42e95bb24c8480f73d371fd63b9c already in progress: creating tablet
I20250903 20:05:23.247610 27687 tablet_replica.cc:331] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:23.248101 27687 raft_consensus.cc:2241] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:23.248829 27687 raft_consensus.cc:2270] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:23.189803 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet cb2f50683e904dd39ee6eb128717d38c with delete_type TABLET_DATA_DELETED (replaced by tablet 72fcb7b0cba34313a7dc2b8c29e120fc) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:23.241271 27695 tablet_bootstrap.cc:492] T cd92e3dafca6419f8eba7a4ef51a911e P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:23.252065 27695 ts_tablet_manager.cc:1397] T cd92e3dafca6419f8eba7a4ef51a911e P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.197s user 0.025s sys 0.031s
I20250903 20:05:23.252172 27685 raft_consensus.cc:2270] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:23.252887 27685 ts_tablet_manager.cc:1905] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:23.253144 27695 raft_consensus.cc:357] T cd92e3dafca6419f8eba7a4ef51a911e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.253401 27695 raft_consensus.cc:383] T cd92e3dafca6419f8eba7a4ef51a911e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:23.253496 27695 raft_consensus.cc:738] T cd92e3dafca6419f8eba7a4ef51a911e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:23.253785 27695 consensus_queue.cc:260] T cd92e3dafca6419f8eba7a4ef51a911e P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.254110 27687 ts_tablet_manager.cc:1905] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:23.257508 27695 ts_tablet_manager.cc:1428] T cd92e3dafca6419f8eba7a4ef51a911e P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250903 20:05:23.258594 27695 tablet_bootstrap.cc:492] T 45d9b6ff520348628d19671127b9426c P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:23.258893 28451 tablet.cc:1722] T e8bc99de4ffe43d58351d170b7c61d8a P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250903 20:05:23.260102 27695 tablet_bootstrap.cc:654] T 45d9b6ff520348628d19671127b9426c P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:23.260623 27524 raft_consensus.cc:2466] T ac1b0dc95fca49b4af6697c0a70e9730 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:23.264561 27684 ts_tablet_manager.cc:1918] T 07565ea2b5f34dcfb407ff3bfde17243 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:23.264781 27684 log.cc:1199] T 07565ea2b5f34dcfb407ff3bfde17243 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/07565ea2b5f34dcfb407ff3bfde17243
I20250903 20:05:23.264915 28451 tablet.cc:1722] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250903 20:05:23.265151 27684 ts_tablet_manager.cc:1939] T 07565ea2b5f34dcfb407ff3bfde17243 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:23.265664 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 668) took 1163 ms. Trace:
I20250903 20:05:23.265805 27684 rpcz_store.cc:276] 0903 20:05:22.102162 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.102260 (+ 98us) service_pool.cc:225] Handling call
0903 20:05:23.174116 (+1071856us) ts_tablet_manager.cc:1145] Deleting tablet 07565ea2b5f34dcfb407ff3bfde17243
0903 20:05:23.174127 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.260829 (+ 86702us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.264479 (+ 3650us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.265645 (+ 1166us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1071618}
I20250903 20:05:23.266381 27684 tablet_replica.cc:331] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:23.266748 27684 raft_consensus.cc:2241] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:23.266970 27684 raft_consensus.cc:2270] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:23.267231 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 07565ea2b5f34dcfb407ff3bfde17243 (table default.Table_4810ae9efbde43bc8c237c6e135aa5aa [id=dbeb164ccbf544cf947af9bf656559c7]) successfully deleted
I20250903 20:05:23.267581 27684 ts_tablet_manager.cc:1905] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:23.270013 27244 raft_consensus.cc:1273] T ac1b0dc95fca49b4af6697c0a70e9730 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Refusing update from remote peer 51578fb7bceb4e119b3ab1e58c4f5528: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 20:05:23.270750 28298 consensus_queue.cc:1035] T ac1b0dc95fca49b4af6697c0a70e9730 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:23.275494 27695 tablet_bootstrap.cc:492] T 45d9b6ff520348628d19671127b9426c P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:23.275729 27695 ts_tablet_manager.cc:1397] T 45d9b6ff520348628d19671127b9426c P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.005s
I20250903 20:05:23.276628 27695 raft_consensus.cc:357] T 45d9b6ff520348628d19671127b9426c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.276870 27695 raft_consensus.cc:383] T 45d9b6ff520348628d19671127b9426c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:23.276964 27695 raft_consensus.cc:738] T 45d9b6ff520348628d19671127b9426c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:23.277251 27695 consensus_queue.cc:260] T 45d9b6ff520348628d19671127b9426c P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.277814 27695 ts_tablet_manager.cc:1428] T 45d9b6ff520348628d19671127b9426c P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:23.279713 27695 tablet_bootstrap.cc:492] T c6d67cd0c0304c77ba571b414878dad2 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:23.281471 27695 tablet_bootstrap.cc:654] T c6d67cd0c0304c77ba571b414878dad2 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:23.305938 27687 ts_tablet_manager.cc:1918] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:23.307318 27687 log.cc:1199] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/c9129a3e773a4fecb28ceee27778a294
I20250903 20:05:23.307955 27687 ts_tablet_manager.cc:1939] T c9129a3e773a4fecb28ceee27778a294 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:23.310370 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 606) took 2591 ms. Trace:
I20250903 20:05:23.311118 27695 tablet_bootstrap.cc:492] T c6d67cd0c0304c77ba571b414878dad2 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:23.311091 27687 rpcz_store.cc:276] 0903 20:05:20.719370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.719428 (+ 58us) service_pool.cc:225] Handling call
0903 20:05:23.247551 (+2528123us) ts_tablet_manager.cc:1145] Deleting tablet c9129a3e773a4fecb28ceee27778a294
0903 20:05:23.247564 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.281301 (+ 33737us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.305778 (+ 24477us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.310337 (+ 4559us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2527891}
I20250903 20:05:23.311553 27695 ts_tablet_manager.cc:1397] T c6d67cd0c0304c77ba571b414878dad2 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.001s
I20250903 20:05:23.312073 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet c9129a3e773a4fecb28ceee27778a294 (table default.Table_b2824f7f5f324e6ebcacb1b0645de8ec [id=50898d980302442fbd70cb88f8f4faba]) successfully deleted
I20250903 20:05:23.312758 27695 raft_consensus.cc:357] T c6d67cd0c0304c77ba571b414878dad2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:23.313045 27462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 142713us
I20250903 20:05:23.313337 27462 leader_election.cc:304] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
W20250903 20:05:23.313663 27462 raft_consensus.cc:2684] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250903 20:05:23.313743 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 607) took 2592 ms. Trace:
I20250903 20:05:23.314033 27462 leader_election.cc:304] T 07565ea2b5f34dcfb407ff3bfde17243 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
W20250903 20:05:23.314356 27462 raft_consensus.cc:2684] T 07565ea2b5f34dcfb407ff3bfde17243 P e72ef1dbcc5143cf9f04a808f101c80f: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250903 20:05:23.314407 27695 raft_consensus.cc:383] T c6d67cd0c0304c77ba571b414878dad2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:23.314648 27695 raft_consensus.cc:738] T c6d67cd0c0304c77ba571b414878dad2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:23.315150 27695 consensus_queue.cc:260] T c6d67cd0c0304c77ba571b414878dad2 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.316715 27246 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5fa7cc0c03234223af382982ca737e27" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:23.315810 27320 leader_election.cc:343] T bf2545037c214bb1b5060575e3cdf81a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: bf2545037c214bb1b5060575e3cdf81a
I20250903 20:05:23.317510 27320 leader_election.cc:304] T bf2545037c214bb1b5060575e3cdf81a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:23.318084 27320 consensus_peers.cc:489] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20250903 20:05:23.318341 27320 leader_election.cc:343] T 369c750ff5bf449e989d95ffd89497d4 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 369c750ff5bf449e989d95ffd89497d4
I20250903 20:05:23.318416 27695 ts_tablet_manager.cc:1428] T c6d67cd0c0304c77ba571b414878dad2 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250903 20:05:23.318434 27320 leader_election.cc:304] T 369c750ff5bf449e989d95ffd89497d4 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:23.318523 28413 tablet.cc:1722] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250903 20:05:23.318670 28340 raft_consensus.cc:2747] T bf2545037c214bb1b5060575e3cdf81a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:23.318758 27695 tablet_bootstrap.cc:492] T e78e6d212daf4589a6b4b65711e356cb P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:23.318790 27320 leader_election.cc:304] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:23.319013 28340 raft_consensus.cc:2747] T 369c750ff5bf449e989d95ffd89497d4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:23.319166 28340 raft_consensus.cc:2747] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250903 20:05:23.320554 27524 raft_consensus.cc:1273] T ac1b0dc95fca49b4af6697c0a70e9730 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Refusing update from remote peer 51578fb7bceb4e119b3ab1e58c4f5528: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 20:05:23.322685 28412 tablet.cc:1722] T c8a293c6fa034e9fb29682439cb3bcc2 P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250903 20:05:23.319911 27382 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5fa7cc0c03234223af382982ca737e27" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:23.325537 27695 tablet_bootstrap.cc:654] T e78e6d212daf4589a6b4b65711e356cb P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:23.327499 27692 tablet_bootstrap.cc:492] T 6238740ce1894e899be9a40321f0ab73 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:23.327840 27692 ts_tablet_manager.cc:1397] T 6238740ce1894e899be9a40321f0ab73 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.288s user 0.004s sys 0.009s
I20250903 20:05:23.333734 27693 tablet_bootstrap.cc:492] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:23.333974 27693 ts_tablet_manager.cc:1397] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.159s user 0.011s sys 0.013s
I20250903 20:05:23.334594 27695 tablet_bootstrap.cc:492] T e78e6d212daf4589a6b4b65711e356cb P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:23.334822 27693 raft_consensus.cc:357] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.335080 27693 raft_consensus.cc:383] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:23.335196 27693 raft_consensus.cc:738] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:23.335307 27695 ts_tablet_manager.cc:1397] T e78e6d212daf4589a6b4b65711e356cb P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
I20250903 20:05:23.335713 27693 consensus_queue.cc:260] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.336465 27695 raft_consensus.cc:357] T e78e6d212daf4589a6b4b65711e356cb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.337800 27695 raft_consensus.cc:383] T e78e6d212daf4589a6b4b65711e356cb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:23.338110 27695 raft_consensus.cc:738] T e78e6d212daf4589a6b4b65711e356cb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:23.338562 27695 consensus_queue.cc:260] T e78e6d212daf4589a6b4b65711e356cb P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.340473 27059 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_30f001eba314494d93e8ecac2f0bc3dc [id=66eb535587dc49e5b1ee72124cca0374] alter complete (version 1)
I20250903 20:05:23.341073 27695 ts_tablet_manager.cc:1428] T e78e6d212daf4589a6b4b65711e356cb P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250903 20:05:23.341809 27693 ts_tablet_manager.cc:1428] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
W20250903 20:05:23.341828 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a686ea49a29a4dd7b6e5dce5e1f9935a consensus state has no leader
I20250903 20:05:23.342875 27695 tablet_bootstrap.cc:492] T cd6d6bb514134bdf8e3bc0aaad180732 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:23.343160 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_cbe4f0209a3345bc9ae4a50a260346c8" } new_table_name: "default.Table_78426e4139c24ecd9133e4564ed77d31" modify_external_catalogs: true
I20250903 20:05:23.345064 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ac1cd4ca0afa4c3494fa9953c49c838e" } modify_external_catalogs: true
I20250903 20:05:23.345260 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ac1cd4ca0afa4c3494fa9953c49c838e" } modify_external_catalogs: true
I20250903 20:05:23.345997 27693 tablet_bootstrap.cc:492] T 912eee5b48ef467bbd6bbe34f8aa390c P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:23.346042 27695 tablet_bootstrap.cc:654] T cd6d6bb514134bdf8e3bc0aaad180732 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:23.346853 27687 rpcz_store.cc:276] 0903 20:05:20.721623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.721675 (+ 52us) service_pool.cc:225] Handling call
0903 20:05:23.313581 (+2591906us) ts_tablet_manager.cc:1145] Deleting tablet ebf0a9aaaea1408c8c9caf28f00dbff7
0903 20:05:23.313592 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.313722 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2545015}
W20250903 20:05:23.350526 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet ebf0a9aaaea1408c8c9caf28f00dbff7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ebf0a9aaaea1408c8c9caf28f00dbff7 already in progress: creating tablet
I20250903 20:05:23.350942 27687 tablet_replica.cc:331] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:23.351166 27687 raft_consensus.cc:2241] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:23.352090 27687 pending_rounds.cc:70] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528: Trying to abort 1 pending ops.
I20250903 20:05:23.352669 27695 tablet_bootstrap.cc:492] T cd6d6bb514134bdf8e3bc0aaad180732 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:23.352841 27695 ts_tablet_manager.cc:1397] T cd6d6bb514134bdf8e3bc0aaad180732 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.002s
I20250903 20:05:23.353955 27695 raft_consensus.cc:357] T cd6d6bb514134bdf8e3bc0aaad180732 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.352825 27687 pending_rounds.cc:77] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384959158804480 op_type: NO_OP noop_request { }
I20250903 20:05:23.354337 27687 raft_consensus.cc:2887] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:23.354511 27687 raft_consensus.cc:2270] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:23.356107 27462 leader_election.cc:304] T 5fa7cc0c03234223af382982ca737e27 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:23.356173 28412 tablet.cc:1722] T ac1b0dc95fca49b4af6697c0a70e9730 P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:23.357012 28454 tablet.cc:1722] T 5fa7cc0c03234223af382982ca737e27 P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:23.357347 28299 raft_consensus.cc:2747] T 5fa7cc0c03234223af382982ca737e27 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:23.354208 27695 raft_consensus.cc:383] T cd6d6bb514134bdf8e3bc0aaad180732 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:23.358348 27695 raft_consensus.cc:738] T cd6d6bb514134bdf8e3bc0aaad180732 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:23.358632 27695 consensus_queue.cc:260] T cd6d6bb514134bdf8e3bc0aaad180732 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.359735 28340 consensus_queue.cc:1035] T ac1b0dc95fca49b4af6697c0a70e9730 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:23.359293 27695 ts_tablet_manager.cc:1428] T cd6d6bb514134bdf8e3bc0aaad180732 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250903 20:05:23.360770 27695 tablet_bootstrap.cc:492] T dbfdab4332234b2a93fd3c901a9cb645 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:23.363009 27695 tablet_bootstrap.cc:654] T dbfdab4332234b2a93fd3c901a9cb645 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:23.367980 27692 raft_consensus.cc:357] T 6238740ce1894e899be9a40321f0ab73 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:23.369876 28413 tablet.cc:1722] T ac1b0dc95fca49b4af6697c0a70e9730 P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:23.372824 27692 raft_consensus.cc:383] T 6238740ce1894e899be9a40321f0ab73 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:23.373014 27692 raft_consensus.cc:738] T 6238740ce1894e899be9a40321f0ab73 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:23.373351 27692 consensus_queue.cc:260] T 6238740ce1894e899be9a40321f0ab73 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:23.373998 27692 ts_tablet_manager.cc:1428] T 6238740ce1894e899be9a40321f0ab73 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.046s user 0.001s sys 0.001s
I20250903 20:05:23.374382 27692 tablet_bootstrap.cc:492] T de8ac9f984be46ec8f021928de12a090 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:23.374414 28245 raft_consensus.cc:491] T fa05f54ebc324b40855fa3288760e9e9 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:23.374554 28245 raft_consensus.cc:513] T fa05f54ebc324b40855fa3288760e9e9 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.375445 28245 leader_election.cc:290] T fa05f54ebc324b40855fa3288760e9e9 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:23.376282 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa05f54ebc324b40855fa3288760e9e9" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:23.376636 27382 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa05f54ebc324b40855fa3288760e9e9" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:23.377323 27183 leader_election.cc:343] T fa05f54ebc324b40855fa3288760e9e9 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:23.377831 27184 leader_election.cc:343] T fa05f54ebc324b40855fa3288760e9e9 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:23.377931 27184 leader_election.cc:304] T fa05f54ebc324b40855fa3288760e9e9 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:23.378007 28340 raft_consensus.cc:491] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:23.378144 28340 raft_consensus.cc:513] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:23.378448 28245 raft_consensus.cc:2747] T fa05f54ebc324b40855fa3288760e9e9 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:23.378732 28340 leader_election.cc:290] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:23.379083 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_9b93f568af044c7aa27fb10e5225af5b" } modify_external_catalogs: true
I20250903 20:05:23.379364 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_9b93f568af044c7aa27fb10e5225af5b" } modify_external_catalogs: true
I20250903 20:05:23.385712 27058 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 6238740ce1894e899be9a40321f0ab73 (table default.Table_0fef1d019eff4323b7ba56368614aadc [id=53074cd97dec40c7b0e1444183d6707a]). Expected version 2 got 0
I20250903 20:05:23.390296 27244 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0bdcbaa0169c4ab7ba6ec038b50482fe" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:23.391033 27321 leader_election.cc:343] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: Tablet not RUNNING: STOPPED
I20250903 20:05:23.391872 27693 tablet_bootstrap.cc:654] T 912eee5b48ef467bbd6bbe34f8aa390c P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:23.395368 27348 tablet_service.cc:1468] Processing CreateTablet for tablet d7d40e8ab3024073a845ba3b6589a571 (DEFAULT_TABLE table=default.Table_3628bc6ae45543e39f5bc3443d68e5b9 [id=2fecdca3e2da497daae8b14ecf383884]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:23.396965 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7d40e8ab3024073a845ba3b6589a571. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:23.399719 27507 tablet_service.cc:1468] Processing CreateTablet for tablet d7d40e8ab3024073a845ba3b6589a571 (DEFAULT_TABLE table=default.Table_3628bc6ae45543e39f5bc3443d68e5b9 [id=2fecdca3e2da497daae8b14ecf383884]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:23.400154 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7d40e8ab3024073a845ba3b6589a571. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:23.401392 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 92c0365f08ec47b581b1bfed736524b2 (DEFAULT_TABLE table=default.Table_b57aafd78ceb4611b5b6f4dd3b2bad30 [id=bfadb551a576441ab74523cb464ff406]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:23.401619 27227 tablet_service.cc:1468] Processing CreateTablet for tablet 41201cfd6c2f4346bdbd194d3865073d (DEFAULT_TABLE table=default.Table_1ce4ddcb0e6046bd9087209ae27c7552 [id=e8d563562fca4024af330c4300937310]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:23.401810 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92c0365f08ec47b581b1bfed736524b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:23.401947 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41201cfd6c2f4346bdbd194d3865073d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:23.405427 27685 ts_tablet_manager.cc:1918] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:23.405608 27685 log.cc:1199] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/0bdcbaa0169c4ab7ba6ec038b50482fe
I20250903 20:05:23.406001 27685 ts_tablet_manager.cc:1939] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:23.406409 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 550) took 3389 ms. Trace:
I20250903 20:05:23.406577 27685 rpcz_store.cc:276] 0903 20:05:20.016735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.017122 (+ 387us) service_pool.cc:225] Handling call
0903 20:05:23.166855 (+3149733us) ts_tablet_manager.cc:1145] Deleting tablet 0bdcbaa0169c4ab7ba6ec038b50482fe
0903 20:05:23.166866 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.256421 (+ 89555us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.405347 (+148926us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.406391 (+ 1044us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3149506}
I20250903 20:05:23.406958 27362 tablet_service.cc:1468] Processing CreateTablet for tablet 41201cfd6c2f4346bdbd194d3865073d (DEFAULT_TABLE table=default.Table_1ce4ddcb0e6046bd9087209ae27c7552 [id=e8d563562fca4024af330c4300937310]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:23.407146 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 0bdcbaa0169c4ab7ba6ec038b50482fe (table default.Table_05d58d894486468a8a15aaa4b4c0cabb [id=e69321e60a3a4e71a463a0011c0c6d65]) successfully deleted
I20250903 20:05:23.407380 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41201cfd6c2f4346bdbd194d3865073d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:23.407678 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 557) took 3163 ms. Trace:
I20250903 20:05:23.407817 27685 rpcz_store.cc:276] 0903 20:05:20.244038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.244223 (+ 185us) service_pool.cc:225] Handling call
0903 20:05:23.407553 (+3163330us) ts_tablet_manager.cc:1145] Deleting tablet 4789bd72aa454b639df76bd5ad379850
0903 20:05:23.407563 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.407662 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3163104}
W20250903 20:05:23.408341 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 4789bd72aa454b639df76bd5ad379850 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4789bd72aa454b639df76bd5ad379850 already in progress: creating tablet
I20250903 20:05:23.401392 27218 tablet_service.cc:1468] Processing CreateTablet for tablet d7d40e8ab3024073a845ba3b6589a571 (DEFAULT_TABLE table=default.Table_3628bc6ae45543e39f5bc3443d68e5b9 [id=2fecdca3e2da497daae8b14ecf383884]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:23.409125 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7d40e8ab3024073a845ba3b6589a571. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:23.409224 27462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 53138.6us
I20250903 20:05:23.410050 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0bdcbaa0169c4ab7ba6ec038b50482fe" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:23.410661 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 92c0365f08ec47b581b1bfed736524b2 (DEFAULT_TABLE table=default.Table_b57aafd78ceb4611b5b6f4dd3b2bad30 [id=bfadb551a576441ab74523cb464ff406]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:23.411034 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92c0365f08ec47b581b1bfed736524b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:23.419663 28298 raft_consensus.cc:491] T fbfabb6ff1714d2e921f45256b543759 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:23.419725 27684 ts_tablet_manager.cc:1918] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:23.419843 27684 log.cc:1199] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/9a76a17b64a74fbfbb4f2e92a02ae1cd
I20250903 20:05:23.419831 28298 raft_consensus.cc:513] T fbfabb6ff1714d2e921f45256b543759 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:23.420161 27684 ts_tablet_manager.cc:1939] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:23.420493 28315 raft_consensus.cc:491] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:23.420604 28315 raft_consensus.cc:513] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.421020 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 669) took 1318 ms. Trace:
I20250903 20:05:23.421217 28315 leader_election.cc:290] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:23.421118 27684 rpcz_store.cc:276] 0903 20:05:22.102660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.102711 (+ 51us) service_pool.cc:225] Handling call
0903 20:05:23.266331 (+1163620us) ts_tablet_manager.cc:1145] Deleting tablet 9a76a17b64a74fbfbb4f2e92a02ae1cd
0903 20:05:23.266342 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.332500 (+ 66158us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.419595 (+ 87095us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.420979 (+ 1384us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1163369}
I20250903 20:05:23.422027 27692 tablet_bootstrap.cc:654] T de8ac9f984be46ec8f021928de12a090 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:23.422618 28298 leader_election.cc:290] T fbfabb6ff1714d2e921f45256b543759 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:23.423692 27321 leader_election.cc:343] T fbfabb6ff1714d2e921f45256b543759 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: fbfabb6ff1714d2e921f45256b543759
W20250903 20:05:23.424114 27320 leader_election.cc:343] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:23.424208 27320 leader_election.cc:304] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:23.424612 28340 raft_consensus.cc:2747] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:23.425891 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 9a76a17b64a74fbfbb4f2e92a02ae1cd (table default.Table_e822637ebd11495cbb3c65907c6a7c78 [id=297ca0b9100044d783d2c610c137f77b]) successfully deleted
I20250903 20:05:23.426311 27488 tablet_service.cc:1468] Processing CreateTablet for tablet 41201cfd6c2f4346bdbd194d3865073d (DEFAULT_TABLE table=default.Table_1ce4ddcb0e6046bd9087209ae27c7552 [id=e8d563562fca4024af330c4300937310]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:23.426775 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41201cfd6c2f4346bdbd194d3865073d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:23.430364 28245 raft_consensus.cc:491] T fe902f1b58924be1ac3a5585413f7789 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:23.430518 28245 raft_consensus.cc:513] T fe902f1b58924be1ac3a5585413f7789 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.430584 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 92c0365f08ec47b581b1bfed736524b2 (DEFAULT_TABLE table=default.Table_b57aafd78ceb4611b5b6f4dd3b2bad30 [id=bfadb551a576441ab74523cb464ff406]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:23.430977 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92c0365f08ec47b581b1bfed736524b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:23.431219 28245 leader_election.cc:290] T fe902f1b58924be1ac3a5585413f7789 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:23.431809 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe902f1b58924be1ac3a5585413f7789" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:23.432209 27382 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe902f1b58924be1ac3a5585413f7789" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:23.432690 27184 leader_election.cc:343] T fe902f1b58924be1ac3a5585413f7789 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:23.432911 27183 leader_election.cc:343] T fe902f1b58924be1ac3a5585413f7789 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:23.433007 27183 leader_election.cc:304] T fe902f1b58924be1ac3a5585413f7789 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:23.433380 28245 raft_consensus.cc:2747] T fe902f1b58924be1ac3a5585413f7789 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:23.436888 27244 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9bdc390de4a74a396003c9c3c2b71" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:23.437111 27244 raft_consensus.cc:2466] T 4be9bdc390de4a74a396003c9c3c2b71 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:23.437475 27321 leader_election.cc:304] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:23.437759 28298 raft_consensus.cc:2802] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:23.437835 28298 raft_consensus.cc:491] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:23.437913 28298 raft_consensus.cc:3058] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:23.440476 28298 raft_consensus.cc:513] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.441665 27246 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9bdc390de4a74a396003c9c3c2b71" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:23.441823 27246 raft_consensus.cc:3058] T 4be9bdc390de4a74a396003c9c3c2b71 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:23.444475 27246 raft_consensus.cc:2466] T 4be9bdc390de4a74a396003c9c3c2b71 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:23.445426 27321 leader_election.cc:304] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:23.445796 28340 raft_consensus.cc:2802] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:23.445902 28340 raft_consensus.cc:695] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:23.445914 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 678) took 1133 ms. Trace:
I20250903 20:05:23.446008 27684 rpcz_store.cc:276] 0903 20:05:22.312122 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.312212 (+ 90us) service_pool.cc:225] Handling call
0903 20:05:23.445747 (+1133535us) ts_tablet_manager.cc:1145] Deleting tablet 441bb228c5c54aebb6aba2bc5761bd00
0903 20:05:23.445759 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.445873 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1131335}
I20250903 20:05:23.446146 28340 consensus_queue.cc:237] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:23.446506 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 441bb228c5c54aebb6aba2bc5761bd00 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 441bb228c5c54aebb6aba2bc5761bd00 already in progress: creating tablet
I20250903 20:05:23.447258 28245 raft_consensus.cc:491] T cd92e3dafca6419f8eba7a4ef51a911e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:23.447388 28245 raft_consensus.cc:513] T cd92e3dafca6419f8eba7a4ef51a911e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.447903 28245 leader_election.cc:290] T cd92e3dafca6419f8eba7a4ef51a911e P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:23.448529 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 562) took 3201 ms. Trace:
I20250903 20:05:23.448781 27382 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd92e3dafca6419f8eba7a4ef51a911e" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:23.449580 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd92e3dafca6419f8eba7a4ef51a911e" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:23.447521 28298 leader_election.cc:290] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:23.450152 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 680) took 1137 ms. Trace:
W20250903 20:05:23.449301 27184 leader_election.cc:343] T cd92e3dafca6419f8eba7a4ef51a911e P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:23.450917 27183 leader_election.cc:343] T cd92e3dafca6419f8eba7a4ef51a911e P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:23.451144 27183 leader_election.cc:304] T cd92e3dafca6419f8eba7a4ef51a911e P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:23.451499 28245 raft_consensus.cc:2747] T cd92e3dafca6419f8eba7a4ef51a911e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:23.396555 27687 ts_tablet_manager.cc:1905] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:23.457345 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_86a571e18d164bd5beae624857d8526d" } modify_external_catalogs: true
I20250903 20:05:23.457556 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_86a571e18d164bd5beae624857d8526d" } modify_external_catalogs: true
W20250903 20:05:23.458928 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2dc0f714cdb44ece90615de5a11bc71f consensus state has no leader
I20250903 20:05:23.459208 27061 master_service.cc:948] ReplaceTablet: received request to replace tablet c49767d618be4fd18f30e5124ce0cb84 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:23.459861 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d1b7ba8a5d3440f7be2eb198bbf5e7fa" } modify_external_catalogs: true
I20250903 20:05:23.460042 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d1b7ba8a5d3440f7be2eb198bbf5e7fa" } modify_external_catalogs: true
I20250903 20:05:23.461639 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b5110c69d0364c50a92330c1824b8416" } modify_external_catalogs: true
I20250903 20:05:23.461830 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b5110c69d0364c50a92330c1824b8416" } modify_external_catalogs: true
I20250903 20:05:23.463210 27059 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_0fef1d019eff4323b7ba56368614aadc [id=53074cd97dec40c7b0e1444183d6707a] alter complete (version 2)
I20250903 20:05:23.463673 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_392e604d6bdd46bfbe7e3b6d21529019"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:23.464250 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_392e604d6bdd46bfbe7e3b6d21529019 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:23.466146 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_95b24ad570fd42158c2e3dc68edf37a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:23.466711 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95b24ad570fd42158c2e3dc68edf37a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:23.473027 27320 leader_election.cc:343] T fbfabb6ff1714d2e921f45256b543759 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: fbfabb6ff1714d2e921f45256b543759
I20250903 20:05:23.473150 27320 leader_election.cc:304] T fbfabb6ff1714d2e921f45256b543759 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:23.473590 28340 raft_consensus.cc:2747] T fbfabb6ff1714d2e921f45256b543759 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:23.450840 27685 rpcz_store.cc:276] 0903 20:05:20.246960 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.247180 (+ 220us) service_pool.cc:225] Handling call
0903 20:05:23.448401 (+3201221us) ts_tablet_manager.cc:1145] Deleting tablet 64d53f2071b0449ca61caa6c264a3641
0903 20:05:23.448412 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.448510 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3196481}
W20250903 20:05:23.477358 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 64d53f2071b0449ca61caa6c264a3641 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64d53f2071b0449ca61caa6c264a3641 already in progress: creating tablet
I20250903 20:05:23.477494 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 556) took 3233 ms. Trace:
I20250903 20:05:23.477578 27685 rpcz_store.cc:276] 0903 20:05:20.243935 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.246300 (+ 2365us) service_pool.cc:225] Handling call
0903 20:05:23.477348 (+3231048us) ts_tablet_manager.cc:1145] Deleting tablet 5f2eefeebbfc47249e2f72b7c52e905d
0903 20:05:23.477359 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.477475 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3225236}
W20250903 20:05:23.478159 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 5f2eefeebbfc47249e2f72b7c52e905d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f2eefeebbfc47249e2f72b7c52e905d already in progress: creating tablet
I20250903 20:05:23.478739 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 565) took 3208 ms. Trace:
I20250903 20:05:23.478859 27685 rpcz_store.cc:276] 0903 20:05:20.270471 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.270542 (+ 71us) service_pool.cc:225] Handling call
0903 20:05:23.478631 (+3208089us) ts_tablet_manager.cc:1145] Deleting tablet 1bf570855b204b96acd070587c434463
0903 20:05:23.478640 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.478723 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3199763}
W20250903 20:05:23.479310 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 1bf570855b204b96acd070587c434463 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1bf570855b204b96acd070587c434463 already in progress: creating tablet
I20250903 20:05:23.479722 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 570) took 3193 ms. Trace:
I20250903 20:05:23.479835 27685 rpcz_store.cc:276] 0903 20:05:20.286031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.286104 (+ 73us) service_pool.cc:225] Handling call
0903 20:05:23.479613 (+3193509us) ts_tablet_manager.cc:1145] Deleting tablet 259bc73ca9df47e9874692725a60bb1c
0903 20:05:23.479623 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.479704 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3189018}
W20250903 20:05:23.480301 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 259bc73ca9df47e9874692725a60bb1c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 259bc73ca9df47e9874692725a60bb1c already in progress: creating tablet
I20250903 20:05:23.480634 27685 tablet_replica.cc:331] T e9eb24dd5a6243c49a3389320a3f78a0 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:23.480857 27685 raft_consensus.cc:2241] T e9eb24dd5a6243c49a3389320a3f78a0 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:23.481079 27685 raft_consensus.cc:2270] T e9eb24dd5a6243c49a3389320a3f78a0 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:23.481658 27685 ts_tablet_manager.cc:1905] T e9eb24dd5a6243c49a3389320a3f78a0 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:23.487269 27693 tablet_bootstrap.cc:492] T 912eee5b48ef467bbd6bbe34f8aa390c P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:23.487514 27693 ts_tablet_manager.cc:1397] T 912eee5b48ef467bbd6bbe34f8aa390c P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.142s user 0.000s sys 0.003s
I20250903 20:05:23.488379 27693 raft_consensus.cc:357] T 912eee5b48ef467bbd6bbe34f8aa390c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:23.488682 27693 raft_consensus.cc:383] T 912eee5b48ef467bbd6bbe34f8aa390c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:23.488797 27693 raft_consensus.cc:738] T 912eee5b48ef467bbd6bbe34f8aa390c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:23.489079 27693 consensus_queue.cc:260] T 912eee5b48ef467bbd6bbe34f8aa390c P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:23.493357 27693 ts_tablet_manager.cc:1428] T 912eee5b48ef467bbd6bbe34f8aa390c P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20250903 20:05:23.497766 27693 tablet_bootstrap.cc:492] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:23.499626 27693 tablet_bootstrap.cc:654] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:23.503335 27687 ts_tablet_manager.cc:1918] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:23.504436 27687 log.cc:1199] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/4c543ac4df33429b9787c6d32bf1ebeb
I20250903 20:05:23.504865 27687 ts_tablet_manager.cc:1939] T 4c543ac4df33429b9787c6d32bf1ebeb P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:23.505685 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 612) took 2729 ms. Trace:
I20250903 20:05:23.511468 27687 rpcz_store.cc:276] 0903 20:05:20.776487 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.777592 (+ 1105us) service_pool.cc:225] Handling call
0903 20:05:23.350880 (+2573288us) ts_tablet_manager.cc:1145] Deleting tablet 4c543ac4df33429b9787c6d32bf1ebeb
0903 20:05:23.350892 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.471474 (+120582us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.503207 (+ 31733us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.505663 (+ 2456us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2566473}
I20250903 20:05:23.512137 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 4c543ac4df33429b9787c6d32bf1ebeb (table default.Table_26a46b6cf89f459b99526bc640729603 [id=ec0522912bda4887b0c40e98ac12768a]) successfully deleted
I20250903 20:05:23.512621 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 611) took 2736 ms. Trace:
I20250903 20:05:23.512758 27687 rpcz_store.cc:276] 0903 20:05:20.776340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.777843 (+ 1503us) service_pool.cc:225] Handling call
0903 20:05:23.512467 (+2734624us) ts_tablet_manager.cc:1145] Deleting tablet 39c2ebe4ada5426590ec3769c8662644
0903 20:05:23.512479 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.512603 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2709666}
I20250903 20:05:23.513455 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 610) took 2740 ms. Trace:
I20250903 20:05:23.513576 27687 rpcz_store.cc:276] 0903 20:05:20.772973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.773047 (+ 74us) service_pool.cc:225] Handling call
0903 20:05:23.513364 (+2740317us) ts_tablet_manager.cc:1145] Deleting tablet 1bf69315f07a470bb2b897ad7c590729
0903 20:05:23.513373 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.513441 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2709653}
W20250903 20:05:23.513993 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 1bf69315f07a470bb2b897ad7c590729 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1bf69315f07a470bb2b897ad7c590729 already in progress: creating tablet
I20250903 20:05:23.514334 27687 tablet_replica.cc:331] T e9eb24dd5a6243c49a3389320a3f78a0 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:23.514535 27687 raft_consensus.cc:2241] T e9eb24dd5a6243c49a3389320a3f78a0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:23.514750 27687 raft_consensus.cc:2270] T e9eb24dd5a6243c49a3389320a3f78a0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:23.515334 27687 ts_tablet_manager.cc:1905] T e9eb24dd5a6243c49a3389320a3f78a0 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:23.466625 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9bdc390de4a74a396003c9c3c2b71" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:23.516290 27684 rpcz_store.cc:276] 0903 20:05:22.312842 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.313272 (+ 430us) service_pool.cc:225] Handling call
0903 20:05:23.450029 (+1136757us) ts_tablet_manager.cc:1145] Deleting tablet 16601b34e4764a8aa14ab95ef8778202
0903 20:05:23.450038 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.450116 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1131021}
I20250903 20:05:23.467182 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9bdc390de4a74a396003c9c3c2b71" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
W20250903 20:05:23.516748 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 16601b34e4764a8aa14ab95ef8778202 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16601b34e4764a8aa14ab95ef8778202 already in progress: creating tablet
I20250903 20:05:23.474891 28454 tablet.cc:1722] T ac1b0dc95fca49b4af6697c0a70e9730 P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250903 20:05:23.517503 27320 leader_election.cc:343] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:23.517803 27693 tablet_bootstrap.cc:492] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:23.499897 27692 tablet_bootstrap.cc:492] T de8ac9f984be46ec8f021928de12a090 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
W20250903 20:05:23.518069 27320 leader_election.cc:343] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:23.518512 27692 ts_tablet_manager.cc:1397] T de8ac9f984be46ec8f021928de12a090 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.144s user 0.005s sys 0.000s
I20250903 20:05:23.518872 27693 ts_tablet_manager.cc:1397] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20250903 20:05:23.519302 27692 raft_consensus.cc:357] T de8ac9f984be46ec8f021928de12a090 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:23.519562 27692 raft_consensus.cc:383] T de8ac9f984be46ec8f021928de12a090 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:23.519608 27693 raft_consensus.cc:357] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.519690 27692 raft_consensus.cc:738] T de8ac9f984be46ec8f021928de12a090 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:23.519982 27692 consensus_queue.cc:260] T de8ac9f984be46ec8f021928de12a090 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:23.520673 27692 ts_tablet_manager.cc:1428] T de8ac9f984be46ec8f021928de12a090 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:23.521008 27692 tablet_bootstrap.cc:492] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:23.524435 27693 raft_consensus.cc:383] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:23.524597 27693 raft_consensus.cc:738] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:23.524829 27692 tablet_bootstrap.cc:654] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:23.524953 27693 consensus_queue.cc:260] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:23.526804 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 9a71290c27ac4a22bd9aa8e14e5834f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a71290c27ac4a22bd9aa8e14e5834f5 already in progress: creating tablet
W20250903 20:05:23.527716 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 8093e36d79894ed3aa8e92c0bca4f93d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8093e36d79894ed3aa8e92c0bca4f93d already in progress: creating tablet
W20250903 20:05:23.528332 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 53881ad30bc843839008ace4012a3f70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53881ad30bc843839008ace4012a3f70 already in progress: creating tablet
W20250903 20:05:23.528937 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 4e99665f920f4de590bd6d9e918c7182 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e99665f920f4de590bd6d9e918c7182 already in progress: creating tablet
W20250903 20:05:23.529697 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet f9375fff5f5945078c4412aa21c06f0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9375fff5f5945078c4412aa21c06f0d already in progress: creating tablet
W20250903 20:05:23.530766 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 09fa32312c9648fbbd6fd327e9738066 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09fa32312c9648fbbd6fd327e9738066 already in progress: creating tablet
W20250903 20:05:23.531354 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 5253d82a41494ce59bb96928179dfdd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5253d82a41494ce59bb96928179dfdd9 already in progress: creating tablet
W20250903 20:05:23.532016 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 9767293447214a7eb68ed03d7373db7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9767293447214a7eb68ed03d7373db7f already in progress: creating tablet
I20250903 20:05:23.532155 27693 ts_tablet_manager.cc:1428] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
W20250903 20:05:23.532699 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet cd6d6bb514134bdf8e3bc0aaad180732 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd6d6bb514134bdf8e3bc0aaad180732 already in progress: creating tablet
W20250903 20:05:23.533274 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 443d77f2593b44afa3e5aae4c5b0a651 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 443d77f2593b44afa3e5aae4c5b0a651 already in progress: creating tablet
W20250903 20:05:23.533849 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet cb2f50683e904dd39ee6eb128717d38c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb2f50683e904dd39ee6eb128717d38c already in progress: creating tablet
I20250903 20:05:23.537149 27693 tablet_bootstrap.cc:492] T ac9b5d4f39db4f05a15316585040d142 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:23.540781 27693 tablet_bootstrap.cc:654] T ac9b5d4f39db4f05a15316585040d142 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:23.548837 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 79c0cc0c30ab42eab0be415e3db9589f consensus state has no leader
I20250903 20:05:23.550541 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet b6b4713936134681aaef1732d46c6ab2
I20250903 20:05:23.551332 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet b6b4713936134681aaef1732d46c6ab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:23 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:23.551658 27227 tablet_service.cc:1515] Processing DeleteTablet for tablet b6b4713936134681aaef1732d46c6ab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:23 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:23.552436 28475 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.65.125:44059 every 8 connection(s)
I20250903 20:05:23.552789 28184 rpc_server.cc:307] RPC server started. Bound to: 127.26.65.125:44059
I20250903 20:05:23.553265 27695 tablet_bootstrap.cc:492] T dbfdab4332234b2a93fd3c901a9cb645 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:23.553478 27695 ts_tablet_manager.cc:1397] T dbfdab4332234b2a93fd3c901a9cb645 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.193s user 0.014s sys 0.026s
I20250903 20:05:23.553668 27066 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet de8ac9f984be46ec8f021928de12a090 (table default.Table_eaff993b7bb7437580aae238faa74b3e [id=2650c8259ef4489695c56b6a2ba898d3]). Expected version 2 got 0
I20250903 20:05:23.555197 28184 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/data/info.pb
I20250903 20:05:23.555244 27695 raft_consensus.cc:357] T dbfdab4332234b2a93fd3c901a9cb645 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.555579 27695 raft_consensus.cc:383] T dbfdab4332234b2a93fd3c901a9cb645 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:23.555756 27695 raft_consensus.cc:738] T dbfdab4332234b2a93fd3c901a9cb645 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:23.557039 27687 ts_tablet_manager.cc:1918] T e9eb24dd5a6243c49a3389320a3f78a0 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:23.556252 27695 consensus_queue.cc:260] T dbfdab4332234b2a93fd3c901a9cb645 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.560279 27687 log.cc:1199] T e9eb24dd5a6243c49a3389320a3f78a0 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/e9eb24dd5a6243c49a3389320a3f78a0
I20250903 20:05:23.560773 27695 ts_tablet_manager.cc:1428] T dbfdab4332234b2a93fd3c901a9cb645 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20250903 20:05:23.560976 27687 ts_tablet_manager.cc:1939] T e9eb24dd5a6243c49a3389320a3f78a0 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:23.562887 27695 tablet_bootstrap.cc:492] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:23.565181 26885 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskH7eH49/build/asan/bin/kudu as pid 28184
I20250903 20:05:23.567214 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 613) took 2782 ms. Trace:
I20250903 20:05:23.567353 27687 rpcz_store.cc:276] 0903 20:05:20.785213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.787462 (+ 2249us) service_pool.cc:225] Handling call
0903 20:05:23.514291 (+2726829us) ts_tablet_manager.cc:1145] Deleting tablet e9eb24dd5a6243c49a3389320a3f78a0
0903 20:05:23.514300 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.550308 (+ 36008us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.556912 (+ 6604us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.567165 (+ 10253us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2701751}
I20250903 20:05:23.568652 27693 tablet_bootstrap.cc:492] T ac9b5d4f39db4f05a15316585040d142 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:23.568926 27693 ts_tablet_manager.cc:1397] T ac9b5d4f39db4f05a15316585040d142 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.003s
I20250903 20:05:23.569769 27693 raft_consensus.cc:357] T ac9b5d4f39db4f05a15316585040d142 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.570024 27693 raft_consensus.cc:383] T ac9b5d4f39db4f05a15316585040d142 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:23.570176 27693 raft_consensus.cc:738] T ac9b5d4f39db4f05a15316585040d142 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:23.570531 27693 consensus_queue.cc:260] T ac9b5d4f39db4f05a15316585040d142 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.574478 27063 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet a1c71f40603c4834a0556ab514c0b5e4
I20250903 20:05:23.574908 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1c67ff00c09c4c3485ae43755a6f7c1f" } new_table_name: "default.Table_5169afe15fc24518bcb50ddc2d74e306" modify_external_catalogs: true
I20250903 20:05:23.575479 27693 ts_tablet_manager.cc:1428] T ac9b5d4f39db4f05a15316585040d142 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250903 20:05:23.578384 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet b6b4713936134681aaef1732d46c6ab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:23 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:23.578770 27684 tablet_replica.cc:331] T b6b4713936134681aaef1732d46c6ab2 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:23.579027 27684 raft_consensus.cc:2241] T b6b4713936134681aaef1732d46c6ab2 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:23.579190 27693 tablet_bootstrap.cc:492] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:23.579309 27684 raft_consensus.cc:2270] T b6b4713936134681aaef1732d46c6ab2 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:23.579946 27684 ts_tablet_manager.cc:1905] T b6b4713936134681aaef1732d46c6ab2 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:23.581063 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet e9eb24dd5a6243c49a3389320a3f78a0 (table default.Table_091b5cecb0b04277a78dc078b4f800a6 [id=1b9cb57cf2b445098305391cc7935b69]) successfully deleted
I20250903 20:05:23.581480 27687 tablet_replica.cc:331] T 46e3a65118b744298b5536f88c22f1db P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:23.582111 27693 tablet_bootstrap.cc:654] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:23.582365 27687 raft_consensus.cc:2241] T 46e3a65118b744298b5536f88c22f1db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:23.583421 27687 raft_consensus.cc:2270] T 46e3a65118b744298b5536f88c22f1db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:23.584048 27687 ts_tablet_manager.cc:1905] T 46e3a65118b744298b5536f88c22f1db P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:23.594930 27693 tablet_bootstrap.cc:492] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:23.595216 27693 ts_tablet_manager.cc:1397] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.001s
I20250903 20:05:23.596040 27693 raft_consensus.cc:357] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.596343 27693 raft_consensus.cc:383] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:23.596462 27693 raft_consensus.cc:738] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:23.596791 27693 consensus_queue.cc:260] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.599960 27695 tablet_bootstrap.cc:654] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:23.601244 27693 ts_tablet_manager.cc:1428] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250903 20:05:23.611332 27693 tablet_bootstrap.cc:492] T cd92e3dafca6419f8eba7a4ef51a911e P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:23.618542 28245 raft_consensus.cc:491] T e78e6d212daf4589a6b4b65711e356cb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:23.618728 28245 raft_consensus.cc:513] T e78e6d212daf4589a6b4b65711e356cb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.618914 27058 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_397211088a974a09bd7f031d5c7ca1cc [id=5cfc8827e4ca4f4a94424c48d6f71eb5] alter complete (version 5)
I20250903 20:05:23.619441 28245 leader_election.cc:290] T e78e6d212daf4589a6b4b65711e356cb P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:23.620213 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ff22ae81dcc340ca897c8e8667aa83d0" } new_table_name: "default.Table_3f1f99f4ab084617b3a72447d6e84209" modify_external_catalogs: true
I20250903 20:05:23.620710 27382 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e78e6d212daf4589a6b4b65711e356cb" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:23.621272 27184 leader_election.cc:343] T e78e6d212daf4589a6b4b65711e356cb P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:23.623215 27693 tablet_bootstrap.cc:654] T cd92e3dafca6419f8eba7a4ef51a911e P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:23.625322 27687 ts_tablet_manager.cc:1918] T 46e3a65118b744298b5536f88c22f1db P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 20:05:23.625541 27687 log.cc:1199] T 46e3a65118b744298b5536f88c22f1db P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/46e3a65118b744298b5536f88c22f1db
I20250903 20:05:23.626025 27687 ts_tablet_manager.cc:1939] T 46e3a65118b744298b5536f88c22f1db P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:23.626907 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 625) took 2741 ms. Trace:
I20250903 20:05:23.627064 27687 rpcz_store.cc:276] 0903 20:05:20.905741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.905881 (+ 140us) service_pool.cc:225] Handling call
0903 20:05:23.581413 (+2675532us) ts_tablet_manager.cc:1145] Deleting tablet 46e3a65118b744298b5536f88c22f1db
0903 20:05:23.581424 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.590968 (+ 9544us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.622704 (+ 31736us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.626852 (+ 4148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2650974}
I20250903 20:05:23.628299 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 46e3a65118b744298b5536f88c22f1db (table default.Table_4258f924ddb3449bb5f49fb7a88a5073 [id=decb9fcf2ee64971ba8bcd8326736852]) successfully deleted
I20250903 20:05:23.628681 27695 tablet_bootstrap.cc:492] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:23.628804 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 626) took 2694 ms. Trace:
I20250903 20:05:23.628894 27695 ts_tablet_manager.cc:1397] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.000s
I20250903 20:05:23.628983 27687 rpcz_store.cc:276] 0903 20:05:20.933991 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.934113 (+ 122us) service_pool.cc:225] Handling call
0903 20:05:23.628638 (+2694525us) ts_tablet_manager.cc:1145] Deleting tablet 0906165a50b047c8a776b30832149964
0903 20:05:23.628648 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.628750 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2686006}
I20250903 20:05:23.629794 27695 raft_consensus.cc:357] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.630095 27695 raft_consensus.cc:383] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:23.630196 27695 raft_consensus.cc:738] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:23.630523 27695 consensus_queue.cc:260] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:23.630782 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 0906165a50b047c8a776b30832149964 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0906165a50b047c8a776b30832149964 already in progress: creating tablet
I20250903 20:05:23.631461 27695 ts_tablet_manager.cc:1428] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250903 20:05:23.632107 27695 tablet_bootstrap.cc:492] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:23.634469 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 6820c9a583af4bf68f9e63ae06e6ec64
I20250903 20:05:23.635286 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e58baad01a28404886961ac38943a233"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:23.635923 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e58baad01a28404886961ac38943a233 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:23.635921 27227 tablet_service.cc:1515] Processing DeleteTablet for tablet 6820c9a583af4bf68f9e63ae06e6ec64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:23 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:23.636395 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 6820c9a583af4bf68f9e63ae06e6ec64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:23 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:23.636811 27296 heartbeater.cc:344] Connected to a master server at 127.26.65.125:44059
I20250903 20:05:23.637214 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet f36938c69f6a497f93e9458bdc9c40a0
I20250903 20:05:23.637938 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_43e145c502a34104b1388d584d5c7355"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:23.638530 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43e145c502a34104b1388d584d5c7355 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:23.640136 27433 heartbeater.cc:344] Connected to a master server at 127.26.65.125:44059
I20250903 20:05:23.640599 27227 tablet_service.cc:1515] Processing DeleteTablet for tablet f36938c69f6a497f93e9458bdc9c40a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:23 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:23.641119 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet f36938c69f6a497f93e9458bdc9c40a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:23 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:23.643100 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 02b3e262dd7841dfbb0520377eb94743
I20250903 20:05:23.643640 27062 catalog_manager.cc:6652] ReplaceTablet: tablet 02b3e262dd7841dfbb0520377eb94743 deleted and replaced by tablet fa544dfdfff04a1e97676e562aeb0f40
I20250903 20:05:23.643989 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_0c63411efaae4fdc900ae09cdd4eed96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:23.644564 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c63411efaae4fdc900ae09cdd4eed96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:23.649761 27227 tablet_service.cc:1515] Processing DeleteTablet for tablet 02b3e262dd7841dfbb0520377eb94743 with delete_type TABLET_DATA_DELETED (replaced by tablet fa544dfdfff04a1e97676e562aeb0f40) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:23.650101 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 02b3e262dd7841dfbb0520377eb94743 with delete_type TABLET_DATA_DELETED (replaced by tablet fa544dfdfff04a1e97676e562aeb0f40) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:23.650702 27433 heartbeater.cc:344] Connected to a master server at 127.26.65.125:44059
W20250903 20:05:23.653246 27053 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 4fa886f5be0f489290921e61dcaa4916 (table default.Table_0d67f3bd51344527af6248e556039647 [id=b7afa73399c94853a86ca78575d42726], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4fa886f5be0f489290921e61dcaa4916 consensus state has no leader
I20250903 20:05:23.653828 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 634) took 2651 ms. Trace:
I20250903 20:05:23.653997 27687 rpcz_store.cc:276] 0903 20:05:21.002666 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.117879 (+115213us) service_pool.cc:225] Handling call
0903 20:05:23.648736 (+2530857us) ts_tablet_manager.cc:1145] Deleting tablet 4af8c03c9d834405b4e25d35441db961
0903 20:05:23.648758 (+ 22us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.653772 (+ 5014us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2507444}
W20250903 20:05:23.654726 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 4af8c03c9d834405b4e25d35441db961 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4af8c03c9d834405b4e25d35441db961 already in progress: creating tablet
I20250903 20:05:23.655280 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 643) took 2554 ms. Trace:
I20250903 20:05:23.657081 27687 rpcz_store.cc:276] 0903 20:05:21.101136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.155506 (+ 54370us) service_pool.cc:225] Handling call
0903 20:05:23.655124 (+2499618us) ts_tablet_manager.cc:1145] Deleting tablet a72b85dfb34047528b2b28ff0d40f828
0903 20:05:23.655135 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.655219 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2481959}
W20250903 20:05:23.657845 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet a72b85dfb34047528b2b28ff0d40f828 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a72b85dfb34047528b2b28ff0d40f828 already in progress: creating tablet
I20250903 20:05:23.658540 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 640) took 2613 ms. Trace:
I20250903 20:05:23.658946 27687 rpcz_store.cc:276] 0903 20:05:21.045004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.150019 (+105015us) service_pool.cc:225] Handling call
0903 20:05:23.658372 (+2508353us) ts_tablet_manager.cc:1145] Deleting tablet d4c703dafbca4e669dbb18946ffac699
0903 20:05:23.658383 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.658486 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2478430}
I20250903 20:05:23.659212 27695 tablet_bootstrap.cc:654] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:23.659929 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet d4c703dafbca4e669dbb18946ffac699 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4c703dafbca4e669dbb18946ffac699 already in progress: creating tablet
I20250903 20:05:23.660430 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 646) took 2016 ms. Trace:
I20250903 20:05:23.661612 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 46f49fd718be4a2996d781ab31268217
I20250903 20:05:23.661763 27687 rpcz_store.cc:276] 0903 20:05:21.644324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.644390 (+ 66us) service_pool.cc:225] Handling call
0903 20:05:23.660285 (+2015895us) ts_tablet_manager.cc:1145] Deleting tablet 41056c5c5003415993bee74d7621bd14
0903 20:05:23.660294 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.660377 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1980619}
I20250903 20:05:23.662397 27060 master_service.cc:948] ReplaceTablet: received request to replace tablet c9e90abef9814338b70a3550ab5686e0 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:23.662846 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 46f49fd718be4a2996d781ab31268217 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:23 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:23.663893 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 41056c5c5003415993bee74d7621bd14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41056c5c5003415993bee74d7621bd14 already in progress: creating tablet
I20250903 20:05:23.664494 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 647) took 2020 ms. Trace:
I20250903 20:05:23.663017 27227 tablet_service.cc:1515] Processing DeleteTablet for tablet 46f49fd718be4a2996d781ab31268217 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:23 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:23.666268 27687 rpcz_store.cc:276] 0903 20:05:21.644470 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.644837 (+ 367us) service_pool.cc:225] Handling call
0903 20:05:23.664289 (+2019452us) ts_tablet_manager.cc:1145] Deleting tablet 2c772ab7c50c4994aacde5a05bb0f08e
0903 20:05:23.664302 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.664422 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1984554}
I20250903 20:05:23.665624 27693 tablet_bootstrap.cc:492] T cd92e3dafca6419f8eba7a4ef51a911e P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:23.666733 27693 ts_tablet_manager.cc:1397] T cd92e3dafca6419f8eba7a4ef51a911e P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.003s
I20250903 20:05:23.667833 27693 raft_consensus.cc:357] T cd92e3dafca6419f8eba7a4ef51a911e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.668171 27693 raft_consensus.cc:383] T cd92e3dafca6419f8eba7a4ef51a911e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:23.668324 27693 raft_consensus.cc:738] T cd92e3dafca6419f8eba7a4ef51a911e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:23.668715 27693 consensus_queue.cc:260] T cd92e3dafca6419f8eba7a4ef51a911e P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.673375 27693 ts_tablet_manager.cc:1428] T cd92e3dafca6419f8eba7a4ef51a911e P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250903 20:05:23.679224 27693 tablet_bootstrap.cc:492] T 8605bc329627438998812ed028327f53 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:23.681115 27693 tablet_bootstrap.cc:654] T 8605bc329627438998812ed028327f53 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:23.681566 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 2c772ab7c50c4994aacde5a05bb0f08e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c772ab7c50c4994aacde5a05bb0f08e already in progress: creating tablet
I20250903 20:05:23.686451 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d89675ff457643349b686394094cc04a" } new_table_name: "default.Table_904b3386c52f4e39b9e237131e88c9cf" modify_external_catalogs: true
I20250903 20:05:23.687402 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 648) took 2042 ms. Trace:
I20250903 20:05:23.687955 27687 rpcz_store.cc:276] 0903 20:05:21.644625 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.644989 (+ 364us) service_pool.cc:225] Handling call
0903 20:05:23.687246 (+2042257us) ts_tablet_manager.cc:1145] Deleting tablet a253e3399622409e92299069b19d3698
0903 20:05:23.687256 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.687348 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2007474}
W20250903 20:05:23.688417 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet a253e3399622409e92299069b19d3698 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a253e3399622409e92299069b19d3698 already in progress: creating tablet
I20250903 20:05:23.688843 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 649) took 2044 ms. Trace:
I20250903 20:05:23.688978 27687 rpcz_store.cc:276] 0903 20:05:21.644719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.649447 (+ 4728us) service_pool.cc:225] Handling call
0903 20:05:23.688704 (+2039257us) ts_tablet_manager.cc:1145] Deleting tablet b7a6d5d8a97a4769a81ec7224d591f8c
0903 20:05:23.688714 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.688791 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2008882}
W20250903 20:05:23.689527 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet b7a6d5d8a97a4769a81ec7224d591f8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7a6d5d8a97a4769a81ec7224d591f8c already in progress: creating tablet
I20250903 20:05:23.689965 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 650) took 2037 ms. Trace:
I20250903 20:05:23.690105 27687 rpcz_store.cc:276] 0903 20:05:21.652704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.652763 (+ 59us) service_pool.cc:225] Handling call
0903 20:05:23.689821 (+2037058us) ts_tablet_manager.cc:1145] Deleting tablet 5b26ed39f2144610b786f49e62e6f0a3
0903 20:05:23.689831 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.689912 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2009951}
I20250903 20:05:23.690510 27695 tablet_bootstrap.cc:492] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
W20250903 20:05:23.690585 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 5b26ed39f2144610b786f49e62e6f0a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b26ed39f2144610b786f49e62e6f0a3 already in progress: creating tablet
I20250903 20:05:23.690675 27695 ts_tablet_manager.cc:1397] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.002s
I20250903 20:05:23.690999 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 651) took 1849 ms. Trace:
I20250903 20:05:23.691138 27687 rpcz_store.cc:276] 0903 20:05:21.841104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.841244 (+ 140us) service_pool.cc:225] Handling call
0903 20:05:23.690871 (+1849627us) ts_tablet_manager.cc:1145] Deleting tablet dbfdab4332234b2a93fd3c901a9cb645
0903 20:05:23.690879 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.690949 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1849398}
I20250903 20:05:23.691567 27695 raft_consensus.cc:357] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.691804 27695 raft_consensus.cc:383] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:23.691875 27695 raft_consensus.cc:738] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:23.692159 27695 consensus_queue.cc:260] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:23.692349 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet dbfdab4332234b2a93fd3c901a9cb645 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dbfdab4332234b2a93fd3c901a9cb645 already in progress: creating tablet
I20250903 20:05:23.692759 27695 ts_tablet_manager.cc:1428] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:23.693441 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 662) took 1578 ms. Trace:
I20250903 20:05:23.693575 27687 rpcz_store.cc:276] 0903 20:05:22.114660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.114713 (+ 53us) service_pool.cc:225] Handling call
0903 20:05:23.693313 (+1578600us) ts_tablet_manager.cc:1145] Deleting tablet 4157633525034a019fd6a80185892cec
0903 20:05:23.693322 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.693391 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1578260}
W20250903 20:05:23.694051 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 4157633525034a019fd6a80185892cec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4157633525034a019fd6a80185892cec already in progress: creating tablet
I20250903 20:05:23.694468 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 663) took 1572 ms. Trace:
I20250903 20:05:23.694598 27687 rpcz_store.cc:276] 0903 20:05:22.126898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.126957 (+ 59us) service_pool.cc:225] Handling call
0903 20:05:23.694339 (+1567382us) ts_tablet_manager.cc:1145] Deleting tablet 36e1061915574407b6e7695be8b1e55a
0903 20:05:23.694347 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.694416 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1565027}
W20250903 20:05:23.695051 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 36e1061915574407b6e7695be8b1e55a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36e1061915574407b6e7695be8b1e55a already in progress: creating tablet
I20250903 20:05:23.695452 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 664) took 1562 ms. Trace:
I20250903 20:05:23.695576 27687 rpcz_store.cc:276] 0903 20:05:22.132576 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.132625 (+ 49us) service_pool.cc:225] Handling call
0903 20:05:23.695324 (+1562699us) ts_tablet_manager.cc:1145] Deleting tablet 17e7deb74389448db37289563b86c253
0903 20:05:23.695333 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.695401 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1562497}
W20250903 20:05:23.697613 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 17e7deb74389448db37289563b86c253 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17e7deb74389448db37289563b86c253 already in progress: creating tablet
I20250903 20:05:23.697969 27687 tablet_replica.cc:331] T cd92e3dafca6419f8eba7a4ef51a911e P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:23.698473 27695 tablet_bootstrap.cc:492] T 4157633525034a019fd6a80185892cec P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:23.698197 27687 raft_consensus.cc:2241] T cd92e3dafca6419f8eba7a4ef51a911e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:23.699241 27687 raft_consensus.cc:2270] T cd92e3dafca6419f8eba7a4ef51a911e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:23.700042 27687 ts_tablet_manager.cc:1905] T cd92e3dafca6419f8eba7a4ef51a911e P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:23.700894 27695 tablet_bootstrap.cc:654] T 4157633525034a019fd6a80185892cec P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:23.702867 27685 ts_tablet_manager.cc:1918] T e9eb24dd5a6243c49a3389320a3f78a0 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:23.703024 27685 log.cc:1199] T e9eb24dd5a6243c49a3389320a3f78a0 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/e9eb24dd5a6243c49a3389320a3f78a0
I20250903 20:05:23.703418 27685 ts_tablet_manager.cc:1939] T e9eb24dd5a6243c49a3389320a3f78a0 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:23.704399 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_307490b0006e4ed493ee8b2e23b7877a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:23.705047 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_307490b0006e4ed493ee8b2e23b7877a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:23.718345 27687 ts_tablet_manager.cc:1918] T cd92e3dafca6419f8eba7a4ef51a911e P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:23.718526 27687 log.cc:1199] T cd92e3dafca6419f8eba7a4ef51a911e P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/cd92e3dafca6419f8eba7a4ef51a911e
I20250903 20:05:23.719782 27687 ts_tablet_manager.cc:1939] T cd92e3dafca6419f8eba7a4ef51a911e P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:23.723191 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 665) took 1590 ms. Trace:
I20250903 20:05:23.723336 27687 rpcz_store.cc:276] 0903 20:05:22.132961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.133136 (+ 175us) service_pool.cc:225] Handling call
0903 20:05:23.697888 (+1564752us) ts_tablet_manager.cc:1145] Deleting tablet cd92e3dafca6419f8eba7a4ef51a911e
0903 20:05:23.697897 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.705255 (+ 7358us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.716482 (+ 11227us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.723149 (+ 6667us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1563609}
I20250903 20:05:23.723870 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet cd92e3dafca6419f8eba7a4ef51a911e (table default.Table_c8d55d18c6bb4511ab10f35d2db6d2f9 [id=b15b2b0164104454a23d766feea4e0de]) successfully deleted
I20250903 20:05:23.732040 27695 tablet_bootstrap.cc:492] T 4157633525034a019fd6a80185892cec P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:23.732266 27695 ts_tablet_manager.cc:1397] T 4157633525034a019fd6a80185892cec P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.002s
I20250903 20:05:23.733412 27695 raft_consensus.cc:357] T 4157633525034a019fd6a80185892cec P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.733656 27695 raft_consensus.cc:383] T 4157633525034a019fd6a80185892cec P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:23.733721 27695 raft_consensus.cc:738] T 4157633525034a019fd6a80185892cec P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:23.734084 27695 consensus_queue.cc:260] T 4157633525034a019fd6a80185892cec P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.734791 27695 ts_tablet_manager.cc:1428] T 4157633525034a019fd6a80185892cec P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250903 20:05:23.737313 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 666) took 1602 ms. Trace:
I20250903 20:05:23.737993 27687 rpcz_store.cc:276] 0903 20:05:22.135245 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.135287 (+ 42us) service_pool.cc:225] Handling call
0903 20:05:23.737091 (+1601804us) ts_tablet_manager.cc:1145] Deleting tablet e9126e4a18c546c28927f22c9f16426e
0903 20:05:23.737107 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.737259 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1601616}
W20250903 20:05:23.739202 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet e9126e4a18c546c28927f22c9f16426e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9126e4a18c546c28927f22c9f16426e already in progress: creating tablet
I20250903 20:05:23.741432 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 667) took 1605 ms. Trace:
I20250903 20:05:23.743446 27687 rpcz_store.cc:276] 0903 20:05:22.135595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.135631 (+ 36us) service_pool.cc:225] Handling call
0903 20:05:23.741210 (+1605579us) ts_tablet_manager.cc:1145] Deleting tablet c49767d618be4fd18f30e5124ce0cb84
0903 20:05:23.741230 (+ 20us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.741377 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1598980}
W20250903 20:05:23.744341 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet c49767d618be4fd18f30e5124ce0cb84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c49767d618be4fd18f30e5124ce0cb84 already in progress: creating tablet
I20250903 20:05:23.744870 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 679) took 1503 ms. Trace:
I20250903 20:05:23.746163 27687 rpcz_store.cc:276] 0903 20:05:22.241684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.264524 (+ 22840us) service_pool.cc:225] Handling call
0903 20:05:23.744730 (+1480206us) ts_tablet_manager.cc:1145] Deleting tablet 16601b34e4764a8aa14ab95ef8778202
0903 20:05:23.744743 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.744839 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1469672}
W20250903 20:05:23.747220 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 16601b34e4764a8aa14ab95ef8778202 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16601b34e4764a8aa14ab95ef8778202 already in progress: creating tablet
I20250903 20:05:23.747367 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_c1f1c15403024868bdc2079821d61fdc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:23.747954 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 677) took 1506 ms. Trace:
W20250903 20:05:23.748108 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1f1c15403024868bdc2079821d61fdc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:23.749958 27687 rpcz_store.cc:276] 0903 20:05:22.241398 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.260444 (+ 19046us) service_pool.cc:225] Handling call
0903 20:05:23.747812 (+1487368us) ts_tablet_manager.cc:1145] Deleting tablet 441bb228c5c54aebb6aba2bc5761bd00
0903 20:05:23.747824 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.747918 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1466507}
W20250903 20:05:23.751320 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 441bb228c5c54aebb6aba2bc5761bd00 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 441bb228c5c54aebb6aba2bc5761bd00 already in progress: creating tablet
I20250903 20:05:23.752617 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 695) took 1327 ms. Trace:
I20250903 20:05:23.753481 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 576) took 3419 ms. Trace:
I20250903 20:05:23.753576 27685 rpcz_store.cc:276] 0903 20:05:20.333780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.338402 (+ 4622us) service_pool.cc:225] Handling call
0903 20:05:23.480590 (+3142188us) ts_tablet_manager.cc:1145] Deleting tablet e9eb24dd5a6243c49a3389320a3f78a0
0903 20:05:23.480601 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.577241 (+ 96640us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.702730 (+125489us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.753462 (+ 50732us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3141921}
I20250903 20:05:23.753949 27695 tablet_bootstrap.cc:492] T 0ad64c699b294679b6fec37b5ad23528 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:23.754487 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 578) took 3306 ms. Trace:
I20250903 20:05:23.754489 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet e9eb24dd5a6243c49a3389320a3f78a0 (table default.Table_091b5cecb0b04277a78dc078b4f800a6 [id=1b9cb57cf2b445098305391cc7935b69]) successfully deleted
I20250903 20:05:23.754570 27685 rpcz_store.cc:276] 0903 20:05:20.447646 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.448708 (+ 1062us) service_pool.cc:225] Handling call
0903 20:05:23.754355 (+3305647us) ts_tablet_manager.cc:1145] Deleting tablet 847d0a0a3d2b45938cec9058c77f6231
0903 20:05:23.754364 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.754467 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3305271}
W20250903 20:05:23.754995 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 847d0a0a3d2b45938cec9058c77f6231 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 847d0a0a3d2b45938cec9058c77f6231 already in progress: creating tablet
I20250903 20:05:23.755746 27695 tablet_bootstrap.cc:654] T 0ad64c699b294679b6fec37b5ad23528 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:23.759037 27687 rpcz_store.cc:276] 0903 20:05:22.424759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.435305 (+ 10546us) service_pool.cc:225] Handling call
0903 20:05:23.751877 (+1316572us) ts_tablet_manager.cc:1145] Deleting tablet 4e99665f920f4de590bd6d9e918c7182
0903 20:05:23.751889 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.751991 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1316374}
I20250903 20:05:23.760987 27296 heartbeater.cc:344] Connected to a master server at 127.26.65.125:44059
W20250903 20:05:23.761685 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 4e99665f920f4de590bd6d9e918c7182 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e99665f920f4de590bd6d9e918c7182 already in progress: creating tablet
I20250903 20:05:23.762409 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 693) took 1337 ms. Trace:
I20250903 20:05:23.763459 27687 rpcz_store.cc:276] 0903 20:05:22.424537 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.435178 (+ 10641us) service_pool.cc:225] Handling call
0903 20:05:23.762194 (+1327016us) ts_tablet_manager.cc:1145] Deleting tablet 8093e36d79894ed3aa8e92c0bca4f93d
0903 20:05:23.762206 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.762367 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1324686}
I20250903 20:05:23.763677 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 579) took 3253 ms. Trace:
I20250903 20:05:23.763767 27685 rpcz_store.cc:276] 0903 20:05:20.510147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.510403 (+ 256us) service_pool.cc:225] Handling call
0903 20:05:23.763369 (+3252966us) ts_tablet_manager.cc:1145] Deleting tablet f2629f46704e4a0fa7922533928ca2ed
0903 20:05:23.763539 (+ 170us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.763651 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3252716}
W20250903 20:05:23.764214 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet f2629f46704e4a0fa7922533928ca2ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2629f46704e4a0fa7922533928ca2ed already in progress: creating tablet
W20250903 20:05:23.766695 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 8093e36d79894ed3aa8e92c0bca4f93d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8093e36d79894ed3aa8e92c0bca4f93d already in progress: creating tablet
I20250903 20:05:23.767231 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 692) took 1342 ms. Trace:
I20250903 20:05:23.767724 27687 rpcz_store.cc:276] 0903 20:05:22.424396 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.435156 (+ 10760us) service_pool.cc:225] Handling call
0903 20:05:23.767075 (+1331919us) ts_tablet_manager.cc:1145] Deleting tablet 9a71290c27ac4a22bd9aa8e14e5834f5
0903 20:05:23.767090 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.767191 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1329393}
W20250903 20:05:23.768440 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 9a71290c27ac4a22bd9aa8e14e5834f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a71290c27ac4a22bd9aa8e14e5834f5 already in progress: creating tablet
I20250903 20:05:23.768908 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 694) took 1344 ms. Trace:
I20250903 20:05:23.769315 27687 rpcz_store.cc:276] 0903 20:05:22.424667 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.435204 (+ 10537us) service_pool.cc:225] Handling call
0903 20:05:23.768777 (+1333573us) ts_tablet_manager.cc:1145] Deleting tablet 53881ad30bc843839008ace4012a3f70
0903 20:05:23.768788 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.768879 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1331050}
W20250903 20:05:23.769974 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 53881ad30bc843839008ace4012a3f70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53881ad30bc843839008ace4012a3f70 already in progress: creating tablet
I20250903 20:05:23.770481 27687 tablet_replica.cc:331] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:23.771703 27693 tablet_bootstrap.cc:492] T 8605bc329627438998812ed028327f53 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:23.771946 27693 ts_tablet_manager.cc:1397] T 8605bc329627438998812ed028327f53 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.093s user 0.001s sys 0.015s
I20250903 20:05:23.772723 27066 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_20e9f856d5e24febbdb3b95098a17f0c [id=6952319d78524829b3bc6c3de6425a67] alter complete (version 1)
I20250903 20:05:23.772888 27693 raft_consensus.cc:357] T 8605bc329627438998812ed028327f53 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.773277 27693 raft_consensus.cc:383] T 8605bc329627438998812ed028327f53 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:23.773417 27693 raft_consensus.cc:738] T 8605bc329627438998812ed028327f53 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:23.774391 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet f71d3dd9af2543299296f21381d1d563 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:23.774462 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_794a9c24e42f44bb9d18feee62639661"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:23.774648 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 0bdcbaa0169c4ab7ba6ec038b50482fe with delete_type TABLET_DATA_DELETED (replaced by tablet 1bcb9d5f010f4297bab78c0183889238) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:23.775118 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_794a9c24e42f44bb9d18feee62639661 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:23.775293 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 4be9bdc390de4a74a396003c9c3c2b71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:23.775580 27693 consensus_queue.cc:260] T 8605bc329627438998812ed028327f53 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.775926 28245 raft_consensus.cc:491] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:23.776077 28245 raft_consensus.cc:513] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:23.776528 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3d7aac88e92d4a45ab2ea694b7eaaaac consensus state has no leader
I20250903 20:05:23.776752 28245 leader_election.cc:290] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:23.776808 27063 master_service.cc:948] ReplaceTablet: received request to replace tablet aa31f2ddb9f84a8093aa1c6f64b278c9 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:23.778174 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_5d1a53d4800a4927b3de164630cc0b7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:23.778749 27227 tablet_service.cc:1515] Processing DeleteTablet for tablet 8605bc329627438998812ed028327f53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:23.778834 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d1a53d4800a4927b3de164630cc0b7a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:23.778942 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet c6d67cd0c0304c77ba571b414878dad2 with delete_type TABLET_DATA_DELETED (replaced by tablet 886b8783aa9d4618aad498d4a1115843) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:23.779132 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet fe902f1b58924be1ac3a5585413f7789 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:23.779505 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet e78e6d212daf4589a6b4b65711e356cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:23.779668 27219 tablet_service.cc:1515] Processing DeleteTablet for tablet 45d9b6ff520348628d19671127b9426c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:23.777642 27382 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "283a3afed7414a958b2dc4176f00bb01" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:23.780895 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 580) took 3270 ms. Trace:
I20250903 20:05:23.782663 28340 raft_consensus.cc:491] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:23.782822 28340 raft_consensus.cc:513] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.783483 28340 leader_election.cc:290] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:23.789726 27693 ts_tablet_manager.cc:1428] T 8605bc329627438998812ed028327f53 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
W20250903 20:05:23.790328 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fa544dfdfff04a1e97676e562aeb0f40 has no consensus state
I20250903 20:05:23.793330 27685 rpcz_store.cc:276] 0903 20:05:20.510339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.514284 (+ 3945us) service_pool.cc:225] Handling call
0903 20:05:23.780776 (+3266492us) ts_tablet_manager.cc:1145] Deleting tablet 4e9a6a6e05464b7b97ae624f51402cf0
0903 20:05:23.780784 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.780878 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3266244}
I20250903 20:05:23.785156 27244 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f71d3dd9af2543299296f21381d1d563" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:23.793821 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 582) took 3201 ms. Trace:
I20250903 20:05:23.793892 27244 raft_consensus.cc:2466] T f71d3dd9af2543299296f21381d1d563 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:23.793921 27685 rpcz_store.cc:276] 0903 20:05:20.592302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.592357 (+ 55us) service_pool.cc:225] Handling call
0903 20:05:23.793639 (+3201282us) ts_tablet_manager.cc:1145] Deleting tablet cac38daf002c4ae698d72903a277a587
0903 20:05:23.793655 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.793795 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3201031}
I20250903 20:05:23.794338 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 581) took 3202 ms. Trace:
I20250903 20:05:23.794348 27321 leader_election.cc:304] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:23.794413 27685 rpcz_store.cc:276] 0903 20:05:20.592148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.604557 (+ 12409us) service_pool.cc:225] Handling call
0903 20:05:23.794198 (+3189641us) ts_tablet_manager.cc:1145] Deleting tablet 200c06f296b94eab94f35f312c5a443b
0903 20:05:23.794231 (+ 33us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.794322 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3189300}
I20250903 20:05:23.794704 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 587) took 3115 ms. Trace:
I20250903 20:05:23.794770 27685 rpcz_store.cc:276] 0903 20:05:20.679156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.687064 (+ 7908us) service_pool.cc:225] Handling call
0903 20:05:23.794621 (+3107557us) ts_tablet_manager.cc:1145] Deleting tablet 615f42e95bb24c8480f73d371fd63b9c
0903 20:05:23.794628 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.794692 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3107261}
I20250903 20:05:23.794956 28340 raft_consensus.cc:2802] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:23.795023 27685 tablet_replica.cc:331] T c9129a3e773a4fecb28ceee27778a294 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:23.795111 28340 raft_consensus.cc:491] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:23.795214 28340 raft_consensus.cc:3058] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:23.795240 27685 raft_consensus.cc:2241] T c9129a3e773a4fecb28ceee27778a294 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:23.795400 27685 raft_consensus.cc:2270] T c9129a3e773a4fecb28ceee27778a294 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:23.795974 27685 ts_tablet_manager.cc:1905] T c9129a3e773a4fecb28ceee27778a294 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:23.797329 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c82f96b0a1524799b48098e7a76446c9" } new_table_name: "default.Table_5e672ed0c8c142029a79b72f799f3f59" modify_external_catalogs: true
I20250903 20:05:23.797811 28340 raft_consensus.cc:513] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:23.809059 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f05ef6e9eac24f7ebebd67aacd4c5c80 consensus state has no leader
I20250903 20:05:23.809302 27060 master_service.cc:948] ReplaceTablet: received request to replace tablet 1bf69315f07a470bb2b897ad7c590729 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:23.809369 28477 sys_catalog.cc:263] Verifying existing consensus state
I20250903 20:05:23.815724 28245 raft_consensus.cc:491] T cd6d6bb514134bdf8e3bc0aaad180732 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:23.815939 28245 raft_consensus.cc:513] T cd6d6bb514134bdf8e3bc0aaad180732 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.816668 28245 leader_election.cc:290] T cd6d6bb514134bdf8e3bc0aaad180732 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
W20250903 20:05:23.785635 27184 leader_election.cc:343] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:23.790176 27695 tablet_bootstrap.cc:492] T 0ad64c699b294679b6fec37b5ad23528 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:23.818222 27695 ts_tablet_manager.cc:1397] T 0ad64c699b294679b6fec37b5ad23528 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.064s user 0.014s sys 0.008s
W20250903 20:05:23.818673 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 4e9a6a6e05464b7b97ae624f51402cf0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e9a6a6e05464b7b97ae624f51402cf0 already in progress: creating tablet
W20250903 20:05:23.818939 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet cac38daf002c4ae698d72903a277a587 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cac38daf002c4ae698d72903a277a587 already in progress: creating tablet
I20250903 20:05:23.819033 27382 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd6d6bb514134bdf8e3bc0aaad180732" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:23.819234 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 200c06f296b94eab94f35f312c5a443b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 200c06f296b94eab94f35f312c5a443b already in progress: creating tablet
I20250903 20:05:23.819329 27695 raft_consensus.cc:357] T 0ad64c699b294679b6fec37b5ad23528 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:23.819470 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 615f42e95bb24c8480f73d371fd63b9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 615f42e95bb24c8480f73d371fd63b9c already in progress: creating tablet
I20250903 20:05:23.819554 27695 raft_consensus.cc:383] T 0ad64c699b294679b6fec37b5ad23528 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:23.819618 27695 raft_consensus.cc:738] T 0ad64c699b294679b6fec37b5ad23528 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:23.819890 27219 tablet_service.cc:1515] Processing DeleteTablet for tablet 7cc6cf579e2642e99fefed723b0ee900 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:23.819867 27695 consensus_queue.cc:260] T 0ad64c699b294679b6fec37b5ad23528 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:23.820283 27184 leader_election.cc:343] T cd6d6bb514134bdf8e3bc0aaad180732 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:23.820468 27219 tablet_service.cc:1515] Processing DeleteTablet for tablet 283a3afed7414a958b2dc4176f00bb01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:23.820494 27695 ts_tablet_manager.cc:1428] T 0ad64c699b294679b6fec37b5ad23528 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:23.821194 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet 46f49fd718be4a2996d781ab31268217 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:23 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:23.825559 27695 tablet_bootstrap.cc:492] T d859c0222c3647dab3dddd105e4cf0fc P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:23.826395 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 6820c9a583af4bf68f9e63ae06e6ec64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:23 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:23.827306 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 02b3e262dd7841dfbb0520377eb94743 with delete_type TABLET_DATA_DELETED (replaced by tablet fa544dfdfff04a1e97676e562aeb0f40) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:23.828022 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f71d3dd9af2543299296f21381d1d563" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:23.828814 27695 tablet_bootstrap.cc:654] T d859c0222c3647dab3dddd105e4cf0fc P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:23.828826 27320 leader_election.cc:343] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:23.829087 27488 tablet_service.cc:1515] Processing DeleteTablet for tablet f36938c69f6a497f93e9458bdc9c40a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:23 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:23.829525 27692 tablet_bootstrap.cc:492] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:23.830453 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e78e6d212daf4589a6b4b65711e356cb" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:23.831115 27183 leader_election.cc:343] T e78e6d212daf4589a6b4b65711e356cb P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:23.831228 27183 leader_election.cc:304] T e78e6d212daf4589a6b4b65711e356cb P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:23.831758 27692 ts_tablet_manager.cc:1397] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.309s user 0.004s sys 0.000s
I20250903 20:05:23.832098 28245 raft_consensus.cc:2747] T e78e6d212daf4589a6b4b65711e356cb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:23.832391 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd6d6bb514134bdf8e3bc0aaad180732" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:23.832747 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "283a3afed7414a958b2dc4176f00bb01" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:23.833214 27183 leader_election.cc:343] T cd6d6bb514134bdf8e3bc0aaad180732 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:23.833343 27183 leader_election.cc:304] T cd6d6bb514134bdf8e3bc0aaad180732 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:23.833693 27183 leader_election.cc:343] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:23.833776 27183 leader_election.cc:304] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:23.835513 27692 raft_consensus.cc:357] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:23.835685 27692 raft_consensus.cc:383] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:23.835740 27692 raft_consensus.cc:738] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:23.835971 27692 consensus_queue.cc:260] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:23.836431 27692 ts_tablet_manager.cc:1428] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20250903 20:05:23.837054 27692 tablet_bootstrap.cc:492] T 639ecb9caa4a423899ab8eb35bb716d7 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:23.840134 28245 raft_consensus.cc:2747] T cd6d6bb514134bdf8e3bc0aaad180732 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:23.840335 28245 raft_consensus.cc:2747] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:23.840443 27244 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f71d3dd9af2543299296f21381d1d563" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:23.840596 27244 raft_consensus.cc:3058] T f71d3dd9af2543299296f21381d1d563 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:23.843912 27692 tablet_bootstrap.cc:654] T 639ecb9caa4a423899ab8eb35bb716d7 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:23.845305 27244 raft_consensus.cc:2466] T f71d3dd9af2543299296f21381d1d563 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:23.845865 27321 leader_election.cc:304] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:23.846231 27574 heartbeater.cc:344] Connected to a master server at 127.26.65.125:44059
I20250903 20:05:23.846181 28298 raft_consensus.cc:2802] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:23.846392 28298 raft_consensus.cc:695] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:23.846652 28298 consensus_queue.cc:237] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.847000 27695 tablet_bootstrap.cc:492] T d859c0222c3647dab3dddd105e4cf0fc P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:23.847123 27695 ts_tablet_manager.cc:1397] T d859c0222c3647dab3dddd105e4cf0fc P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20250903 20:05:23.847774 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f71d3dd9af2543299296f21381d1d563" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:23.847829 27695 raft_consensus.cc:357] T d859c0222c3647dab3dddd105e4cf0fc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.847973 27695 raft_consensus.cc:383] T d859c0222c3647dab3dddd105e4cf0fc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:23.848026 27695 raft_consensus.cc:738] T d859c0222c3647dab3dddd105e4cf0fc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
W20250903 20:05:23.848587 27320 leader_election.cc:343] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:23.849434 27695 consensus_queue.cc:260] T d859c0222c3647dab3dddd105e4cf0fc P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.851670 28340 leader_election.cc:290] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:23.861825 27685 ts_tablet_manager.cc:1918] T c9129a3e773a4fecb28ceee27778a294 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:23.862279 27685 log.cc:1199] T c9129a3e773a4fecb28ceee27778a294 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/c9129a3e773a4fecb28ceee27778a294
I20250903 20:05:23.862906 27685 ts_tablet_manager.cc:1939] T c9129a3e773a4fecb28ceee27778a294 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:23.863480 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 588) took 3143 ms. Trace:
I20250903 20:05:23.770821 27687 raft_consensus.cc:2241] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:23.849915 27695 ts_tablet_manager.cc:1428] T d859c0222c3647dab3dddd105e4cf0fc P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:23.869717 27687 raft_consensus.cc:2270] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:23.870106 27695 tablet_bootstrap.cc:492] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:23.870571 27693 tablet_bootstrap.cc:492] T fe902f1b58924be1ac3a5585413f7789 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:23.872897 27695 tablet_bootstrap.cc:654] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:23.874325 27687 ts_tablet_manager.cc:1905] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:23.874564 27684 ts_tablet_manager.cc:1918] T b6b4713936134681aaef1732d46c6ab2 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 20:05:23.874769 27684 log.cc:1199] T b6b4713936134681aaef1732d46c6ab2 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/b6b4713936134681aaef1732d46c6ab2
I20250903 20:05:23.869232 27685 rpcz_store.cc:276] 0903 20:05:20.719990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.720055 (+ 65us) service_pool.cc:225] Handling call
0903 20:05:23.794963 (+3074908us) ts_tablet_manager.cc:1145] Deleting tablet c9129a3e773a4fecb28ceee27778a294
0903 20:05:23.794970 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.836838 (+ 41868us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.861658 (+ 24820us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.863454 (+ 1796us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3074663}
I20250903 20:05:23.875447 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet c9129a3e773a4fecb28ceee27778a294 (table default.Table_b2824f7f5f324e6ebcacb1b0645de8ec [id=50898d980302442fbd70cb88f8f4faba]) successfully deleted
I20250903 20:05:23.875640 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 589) took 3154 ms. Trace:
I20250903 20:05:23.875764 27685 rpcz_store.cc:276] 0903 20:05:20.721409 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.721758 (+ 349us) service_pool.cc:225] Handling call
0903 20:05:23.875468 (+3153710us) ts_tablet_manager.cc:1145] Deleting tablet ebf0a9aaaea1408c8c9caf28f00dbff7
0903 20:05:23.875483 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.875619 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3153459}
I20250903 20:05:23.875844 27684 ts_tablet_manager.cc:1939] T b6b4713936134681aaef1732d46c6ab2 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
W20250903 20:05:23.876371 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet ebf0a9aaaea1408c8c9caf28f00dbff7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ebf0a9aaaea1408c8c9caf28f00dbff7 already in progress: creating tablet
I20250903 20:05:23.876828 27685 tablet_replica.cc:331] T 4be9bdc390de4a74a396003c9c3c2b71 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:23.877154 27685 raft_consensus.cc:2241] T 4be9bdc390de4a74a396003c9c3c2b71 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:23.877404 27685 raft_consensus.cc:2270] T 4be9bdc390de4a74a396003c9c3c2b71 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:23.877583 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet b6b4713936134681aaef1732d46c6ab2 (table default.Table_ac1cd4ca0afa4c3494fa9953c49c838e [id=d0a75545b91c48bba47fb581bb0d1a96]) successfully deleted
W20250903 20:05:23.877876 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 46f49fd718be4a2996d781ab31268217 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46f49fd718be4a2996d781ab31268217 already in progress: creating tablet
I20250903 20:05:23.878142 27685 ts_tablet_manager.cc:1905] T 4be9bdc390de4a74a396003c9c3c2b71 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:23.882351 27693 tablet_bootstrap.cc:654] T fe902f1b58924be1ac3a5585413f7789 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:23.887051 27692 tablet_bootstrap.cc:492] T 639ecb9caa4a423899ab8eb35bb716d7 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:23.887331 27692 ts_tablet_manager.cc:1397] T 639ecb9caa4a423899ab8eb35bb716d7 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.004s
I20250903 20:05:23.893414 28340 raft_consensus.cc:491] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:23.893621 28340 raft_consensus.cc:513] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.894508 28340 leader_election.cc:290] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:23.896795 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "639ecb9caa4a423899ab8eb35bb716d7" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:23.897509 27320 leader_election.cc:343] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:23.898108 27244 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "639ecb9caa4a423899ab8eb35bb716d7" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:23.898380 27244 raft_consensus.cc:2466] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:23.899195 27321 leader_election.cc:304] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:23.900050 28340 raft_consensus.cc:2802] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:23.900334 28340 raft_consensus.cc:491] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:23.900486 28340 raft_consensus.cc:3058] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:23.901432 28315 raft_consensus.cc:491] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:23.901573 28315 raft_consensus.cc:513] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:23.902350 28315 leader_election.cc:290] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:23.902117 27692 raft_consensus.cc:357] T 639ecb9caa4a423899ab8eb35bb716d7 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.902593 27692 raft_consensus.cc:383] T 639ecb9caa4a423899ab8eb35bb716d7 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:23.902860 27692 raft_consensus.cc:738] T 639ecb9caa4a423899ab8eb35bb716d7 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:23.903800 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de8ac9f984be46ec8f021928de12a090" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:23.903999 27518 raft_consensus.cc:2466] T de8ac9f984be46ec8f021928de12a090 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:23.903801 27692 consensus_queue.cc:260] T 639ecb9caa4a423899ab8eb35bb716d7 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.904593 27320 leader_election.cc:304] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:23.905016 28315 raft_consensus.cc:2802] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:23.905143 28315 raft_consensus.cc:491] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:23.905223 28315 raft_consensus.cc:3058] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:23.908560 28315 raft_consensus.cc:513] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:23.909778 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de8ac9f984be46ec8f021928de12a090" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:23.909878 27692 ts_tablet_manager.cc:1428] T 639ecb9caa4a423899ab8eb35bb716d7 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.022s user 0.003s sys 0.000s
I20250903 20:05:23.909978 27518 raft_consensus.cc:3058] T de8ac9f984be46ec8f021928de12a090 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:23.911409 27693 tablet_bootstrap.cc:492] T fe902f1b58924be1ac3a5585413f7789 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:23.911538 27693 ts_tablet_manager.cc:1397] T fe902f1b58924be1ac3a5585413f7789 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.001s
I20250903 20:05:23.912389 27693 raft_consensus.cc:357] T fe902f1b58924be1ac3a5585413f7789 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.912545 27693 raft_consensus.cc:383] T fe902f1b58924be1ac3a5585413f7789 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:23.912617 27693 raft_consensus.cc:738] T fe902f1b58924be1ac3a5585413f7789 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:23.912823 27693 consensus_queue.cc:260] T fe902f1b58924be1ac3a5585413f7789 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.913136 27518 raft_consensus.cc:2466] T de8ac9f984be46ec8f021928de12a090 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:23.914249 27244 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de8ac9f984be46ec8f021928de12a090" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:23.914448 27244 raft_consensus.cc:2466] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:23.917706 28340 raft_consensus.cc:513] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.917963 27692 tablet_bootstrap.cc:492] T f71d3dd9af2543299296f21381d1d563 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:23.919018 27244 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "639ecb9caa4a423899ab8eb35bb716d7" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:23.919203 27244 raft_consensus.cc:3058] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:23.921861 27244 raft_consensus.cc:2466] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:23.922513 27321 leader_election.cc:304] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:23.923071 28315 leader_election.cc:290] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:23.923848 28315 raft_consensus.cc:2802] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:23.923379 27320 leader_election.cc:304] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:23.924521 28315 raft_consensus.cc:695] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:23.925014 28315 consensus_queue.cc:237] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.925607 27687 ts_tablet_manager.cc:1918] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:23.926759 27687 log.cc:1199] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/9a76a17b64a74fbfbb4f2e92a02ae1cd
I20250903 20:05:23.927356 28500 raft_consensus.cc:2802] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:23.928555 27244 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de8ac9f984be46ec8f021928de12a090" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:23.928812 27695 tablet_bootstrap.cc:492] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:23.929051 27695 ts_tablet_manager.cc:1397] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.000s
I20250903 20:05:23.929605 28340 leader_election.cc:290] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:23.930061 27695 raft_consensus.cc:357] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.930120 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "639ecb9caa4a423899ab8eb35bb716d7" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:23.930320 27695 raft_consensus.cc:383] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:23.930399 27695 raft_consensus.cc:738] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:23.930560 27518 raft_consensus.cc:3058] T 639ecb9caa4a423899ab8eb35bb716d7 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:23.930608 27692 tablet_bootstrap.cc:654] T f71d3dd9af2543299296f21381d1d563 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:23.930713 27695 consensus_queue.cc:260] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.931371 27695 ts_tablet_manager.cc:1428] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:23.931737 27695 tablet_bootstrap.cc:492] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:23.934109 27695 tablet_bootstrap.cc:654] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:23.934088 27518 raft_consensus.cc:2466] T 639ecb9caa4a423899ab8eb35bb716d7 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:23.928829 27244 raft_consensus.cc:3058] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:23.935133 27687 ts_tablet_manager.cc:1939] T 9a76a17b64a74fbfbb4f2e92a02ae1cd P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:23.936601 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 696) took 1458 ms. Trace:
I20250903 20:05:23.938035 27244 raft_consensus.cc:2466] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:23.937106 27687 rpcz_store.cc:276] 0903 20:05:22.478520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.478568 (+ 48us) service_pool.cc:225] Handling call
0903 20:05:23.770385 (+1291817us) ts_tablet_manager.cc:1145] Deleting tablet 9a76a17b64a74fbfbb4f2e92a02ae1cd
0903 20:05:23.770399 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.880870 (+110471us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.924773 (+ 43903us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.936498 (+ 11725us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1291621}
I20250903 20:05:23.939234 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 9a76a17b64a74fbfbb4f2e92a02ae1cd (table default.Table_e822637ebd11495cbb3c65907c6a7c78 [id=297ca0b9100044d783d2c610c137f77b]) successfully deleted
I20250903 20:05:23.941792 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 698) took 1352 ms. Trace:
I20250903 20:05:23.942986 27687 rpcz_store.cc:276] 0903 20:05:22.589287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.591494 (+ 2207us) service_pool.cc:225] Handling call
0903 20:05:23.939872 (+1348378us) ts_tablet_manager.cc:1145] Deleting tablet 659a127405d341c7b074d3445854256f
0903 20:05:23.939887 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.941316 (+ 1429us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1347814}
W20250903 20:05:23.946192 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 6820c9a583af4bf68f9e63ae06e6ec64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6820c9a583af4bf68f9e63ae06e6ec64 already in progress: creating tablet
I20250903 20:05:23.946399 27687 tablet_replica.cc:331] T 369c750ff5bf449e989d95ffd89497d4 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
W20250903 20:05:23.946671 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 02b3e262dd7841dfbb0520377eb94743 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02b3e262dd7841dfbb0520377eb94743 already in progress: creating tablet
W20250903 20:05:23.946986 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet f36938c69f6a497f93e9458bdc9c40a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f36938c69f6a497f93e9458bdc9c40a0 already in progress: creating tablet
I20250903 20:05:23.948534 27687 raft_consensus.cc:2241] T 369c750ff5bf449e989d95ffd89497d4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:23.950949 27687 raft_consensus.cc:2270] T 369c750ff5bf449e989d95ffd89497d4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:23.952208 28477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296: Bootstrap starting.
I20250903 20:05:23.958087 27687 ts_tablet_manager.cc:1905] T 369c750ff5bf449e989d95ffd89497d4 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:23.959573 28500 raft_consensus.cc:695] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:23.960181 28500 consensus_queue.cc:237] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:23.969318 27685 ts_tablet_manager.cc:1918] T 4be9bdc390de4a74a396003c9c3c2b71 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:23.969595 27685 log.cc:1199] T 4be9bdc390de4a74a396003c9c3c2b71 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/4be9bdc390de4a74a396003c9c3c2b71
I20250903 20:05:23.970090 27685 ts_tablet_manager.cc:1939] T 4be9bdc390de4a74a396003c9c3c2b71 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:23.970587 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 590) took 3218 ms. Trace:
I20250903 20:05:23.974035 27692 tablet_bootstrap.cc:492] T f71d3dd9af2543299296f21381d1d563 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:23.978247 28500 raft_consensus.cc:491] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:23.978528 28500 raft_consensus.cc:513] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.977866 27692 ts_tablet_manager.cc:1397] T f71d3dd9af2543299296f21381d1d563 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.059s user 0.009s sys 0.003s
I20250903 20:05:23.980137 28500 leader_election.cc:290] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:23.980525 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "885fff47fea14b84ac6b34a57b827b4b" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:23.981242 27320 leader_election.cc:343] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:23.990418 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e036e82874544ec9bad0f3df65db6cab" } modify_external_catalogs: true
I20250903 20:05:23.990689 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e036e82874544ec9bad0f3df65db6cab" } modify_external_catalogs: true
I20250903 20:05:23.991253 27693 ts_tablet_manager.cc:1428] T fe902f1b58924be1ac3a5585413f7789 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.080s user 0.003s sys 0.010s
I20250903 20:05:23.991330 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_585c269abe9b4aa784813fd7439fa4cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:23.991926 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_585c269abe9b4aa784813fd7439fa4cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:23.994316 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_abbe893fd70a474eaccad776210d7f2f" } new_table_name: "default.Table_765e9646c7d54d3c8be7a5dd6703f47c" modify_external_catalogs: true
I20250903 20:05:23.996001 27059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3da0f03ae8114e348baf44301196b2e5" } modify_external_catalogs: true
I20250903 20:05:23.996165 27059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3da0f03ae8114e348baf44301196b2e5" } modify_external_catalogs: true
I20250903 20:05:23.997092 27693 tablet_bootstrap.cc:492] T 516830fc42e84af693b3af52caee30ee P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:24.000412 27693 tablet_bootstrap.cc:654] T 516830fc42e84af693b3af52caee30ee P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:24.003978 27692 raft_consensus.cc:357] T f71d3dd9af2543299296f21381d1d563 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.006533 27692 raft_consensus.cc:383] T f71d3dd9af2543299296f21381d1d563 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:24.007253 27692 raft_consensus.cc:738] T f71d3dd9af2543299296f21381d1d563 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:24.009776 27692 consensus_queue.cc:260] T f71d3dd9af2543299296f21381d1d563 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.016945 27066 master_service.cc:948] ReplaceTablet: received request to replace tablet de8ac9f984be46ec8f021928de12a090 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:24.020520 27692 ts_tablet_manager.cc:1428] T f71d3dd9af2543299296f21381d1d563 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.040s user 0.004s sys 0.000s
I20250903 20:05:24.021337 27064 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5187f549b3d44cce99c527429ce39a4b" } modify_external_catalogs: true
I20250903 20:05:24.021562 27064 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5187f549b3d44cce99c527429ce39a4b" } modify_external_catalogs: true
W20250903 20:05:24.022990 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 13137053365a428e9d1e8d8852a53b6a consensus state has no leader
I20250903 20:05:24.023406 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_57cd5123dc7042c1887af6e833af4d7a" } modify_external_catalogs: true
I20250903 20:05:24.023612 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_57cd5123dc7042c1887af6e833af4d7a" } modify_external_catalogs: true
I20250903 20:05:24.026824 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b838fda30d4043a5ad578ec527c1d105" } modify_external_catalogs: true
I20250903 20:05:24.027017 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b838fda30d4043a5ad578ec527c1d105" } modify_external_catalogs: true
W20250903 20:05:24.028410 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f4d3e1cfdfab4be1a9143b78bf3bc46d consensus state has no leader
I20250903 20:05:24.030577 27488 tablet_service.cc:1468] Processing CreateTablet for tablet 1ec70a427065468394e7fd9db9143847 (DEFAULT_TABLE table=default.Table_5a979a57ff9e4e9796aa94e6398ead05 [id=4a97ac417c5545acbc52fe001109e20a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.031072 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ec70a427065468394e7fd9db9143847. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.031446 27060 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet de8ac9f984be46ec8f021928de12a090 (table default.Table_eaff993b7bb7437580aae238faa74b3e [id=2650c8259ef4489695c56b6a2ba898d3]). Expected version 2 got 0
I20250903 20:05:24.035667 27504 tablet_service.cc:1468] Processing CreateTablet for tablet b2de48441b5c43c091ac43ba02bf3fd6 (DEFAULT_TABLE table=default.Table_5951c39f7b88401e867c14601b0c1ea1 [id=f45b4c86128f4626a8e55231154511a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.036110 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2de48441b5c43c091ac43ba02bf3fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.036976 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 1ad2ca3e72434b69ad48e533da72e8fd (DEFAULT_TABLE table=default.Table_a6683ce4dd5d4470a5b08bddd66ee17f [id=17a4829ed913439c9229e02bf73af142]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.037377 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ad2ca3e72434b69ad48e533da72e8fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.037508 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 1ad2ca3e72434b69ad48e533da72e8fd (DEFAULT_TABLE table=default.Table_a6683ce4dd5d4470a5b08bddd66ee17f [id=17a4829ed913439c9229e02bf73af142]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.037698 27692 tablet_bootstrap.cc:492] T 4be9bdc390de4a74a396003c9c3c2b71 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:24.039105 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_28fd9ad363e94c149f59785ca4cdeb7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:24.039832 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28fd9ad363e94c149f59785ca4cdeb7b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:24.040393 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_078beaf0d9304922afad9311ba40775a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:24.040927 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_078beaf0d9304922afad9311ba40775a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:24.041570 27499 tablet_service.cc:1468] Processing CreateTablet for tablet 91169fff8a6649828648ac97ccf65fdc (DEFAULT_TABLE table=default.Table_cb368db83ae64dc784cc454f9224c3e2 [id=24484984ae4347caa6cc3b6133abf245]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.037853 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ad2ca3e72434b69ad48e533da72e8fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.042902 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91169fff8a6649828648ac97ccf65fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.043156 27365 tablet_service.cc:1468] Processing CreateTablet for tablet b2de48441b5c43c091ac43ba02bf3fd6 (DEFAULT_TABLE table=default.Table_5951c39f7b88401e867c14601b0c1ea1 [id=f45b4c86128f4626a8e55231154511a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.043601 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2de48441b5c43c091ac43ba02bf3fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.044344 27505 tablet_service.cc:1468] Processing CreateTablet for tablet a7f25f00dff7426195ac51214a14022b (DEFAULT_TABLE table=default.Table_4bc2d6a26f9b41f59aaee304f156a6c5 [id=28e973c60f364bdd9314ff5a1abbd73d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.044693 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7f25f00dff7426195ac51214a14022b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.046659 27348 tablet_service.cc:1468] Processing CreateTablet for tablet 1ad2ca3e72434b69ad48e533da72e8fd (DEFAULT_TABLE table=default.Table_a6683ce4dd5d4470a5b08bddd66ee17f [id=17a4829ed913439c9229e02bf73af142]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.047003 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ad2ca3e72434b69ad48e533da72e8fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.050566 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 91169fff8a6649828648ac97ccf65fdc (DEFAULT_TABLE table=default.Table_cb368db83ae64dc784cc454f9224c3e2 [id=24484984ae4347caa6cc3b6133abf245]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.050944 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91169fff8a6649828648ac97ccf65fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.053203 27362 tablet_service.cc:1468] Processing CreateTablet for tablet 1ec70a427065468394e7fd9db9143847 (DEFAULT_TABLE table=default.Table_5a979a57ff9e4e9796aa94e6398ead05 [id=4a97ac417c5545acbc52fe001109e20a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.053583 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ec70a427065468394e7fd9db9143847. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.056676 27356 tablet_service.cc:1468] Processing CreateTablet for tablet a7f25f00dff7426195ac51214a14022b (DEFAULT_TABLE table=default.Table_4bc2d6a26f9b41f59aaee304f156a6c5 [id=28e973c60f364bdd9314ff5a1abbd73d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.057106 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7f25f00dff7426195ac51214a14022b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.067571 27352 tablet_service.cc:1468] Processing CreateTablet for tablet f63b9a066ae94ef3afe3badaef5bcafd (DEFAULT_TABLE table=default.Table_789d6e974e2f44ff8d04b1a39adcb6a7 [id=1d7a112662d24b5b828cb7ac4b073c43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.068056 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f63b9a066ae94ef3afe3badaef5bcafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.068326 28477 log.cc:826] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296: Log is configured to *not* fsync() on all Append() calls
I20250903 20:05:24.072865 27507 tablet_service.cc:1468] Processing CreateTablet for tablet f63b9a066ae94ef3afe3badaef5bcafd (DEFAULT_TABLE table=default.Table_789d6e974e2f44ff8d04b1a39adcb6a7 [id=1d7a112662d24b5b828cb7ac4b073c43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.073442 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f63b9a066ae94ef3afe3badaef5bcafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.079398 27693 tablet_bootstrap.cc:492] T 516830fc42e84af693b3af52caee30ee P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:24.079599 27693 ts_tablet_manager.cc:1397] T 516830fc42e84af693b3af52caee30ee P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.083s user 0.002s sys 0.002s
I20250903 20:05:24.080482 27693 raft_consensus.cc:357] T 516830fc42e84af693b3af52caee30ee P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.080711 27693 raft_consensus.cc:383] T 516830fc42e84af693b3af52caee30ee P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:24.080803 27693 raft_consensus.cc:738] T 516830fc42e84af693b3af52caee30ee P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:24.081084 27693 consensus_queue.cc:260] T 516830fc42e84af693b3af52caee30ee P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.084563 27693 ts_tablet_manager.cc:1428] T 516830fc42e84af693b3af52caee30ee P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250903 20:05:24.084921 27693 tablet_bootstrap.cc:492] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:24.086625 27693 tablet_bootstrap.cc:654] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:24.089519 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 794865d716134382a939186d7144c4bb (DEFAULT_TABLE table=default.Table_a1bcf29f05be4350b110670fdbf011d1 [id=7490584c886142799d6328d2daa07272]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.090009 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 794865d716134382a939186d7144c4bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.091008 27362 tablet_service.cc:1468] Processing CreateTablet for tablet c066691b278c4278937b4d8ef98b395f (DEFAULT_TABLE table=default.Table_193a0ff511204b3b8130c17e783b51ad [id=957e293e05cc4600aac74d6c0dfcf82a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.091532 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c066691b278c4278937b4d8ef98b395f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.099699 27488 tablet_service.cc:1468] Processing CreateTablet for tablet c066691b278c4278937b4d8ef98b395f (DEFAULT_TABLE table=default.Table_193a0ff511204b3b8130c17e783b51ad [id=957e293e05cc4600aac74d6c0dfcf82a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.100298 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c066691b278c4278937b4d8ef98b395f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:24.107683 27320 consensus_peers.cc:489] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6b4713936134681aaef1732d46c6ab2. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:24.110615 27220 tablet_service.cc:1468] Processing CreateTablet for tablet b2de48441b5c43c091ac43ba02bf3fd6 (DEFAULT_TABLE table=default.Table_5951c39f7b88401e867c14601b0c1ea1 [id=f45b4c86128f4626a8e55231154511a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.111164 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2de48441b5c43c091ac43ba02bf3fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.112182 27219 tablet_service.cc:1468] Processing CreateTablet for tablet 1ec70a427065468394e7fd9db9143847 (DEFAULT_TABLE table=default.Table_5a979a57ff9e4e9796aa94e6398ead05 [id=4a97ac417c5545acbc52fe001109e20a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.112656 27219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ec70a427065468394e7fd9db9143847. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:23.983016 27244 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "885fff47fea14b84ac6b34a57b827b4b" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:24.113832 27244 raft_consensus.cc:2466] T 885fff47fea14b84ac6b34a57b827b4b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:24.114742 28245 raft_consensus.cc:491] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:24.114902 28245 raft_consensus.cc:513] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:23.970772 27685 rpcz_store.cc:276] 0903 20:05:20.752053 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.752110 (+ 57us) service_pool.cc:225] Handling call
0903 20:05:23.876759 (+3124649us) ts_tablet_manager.cc:1145] Deleting tablet 4be9bdc390de4a74a396003c9c3c2b71
0903 20:05:23.876772 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:23.946509 (+ 69737us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.969163 (+ 22654us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:23.970563 (+ 1400us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3124390}
I20250903 20:05:24.117703 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 592) took 3355 ms. Trace:
I20250903 20:05:24.117810 27685 rpcz_store.cc:276] 0903 20:05:20.762047 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.762111 (+ 64us) service_pool.cc:225] Handling call
0903 20:05:24.117525 (+3355414us) ts_tablet_manager.cc:1145] Deleting tablet 1bf69315f07a470bb2b897ad7c590729
0903 20:05:24.117539 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:24.117680 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3355134}
I20250903 20:05:24.118259 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 593) took 3350 ms. Trace:
I20250903 20:05:24.118341 27685 rpcz_store.cc:276] 0903 20:05:20.768023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.768082 (+ 59us) service_pool.cc:225] Handling call
0903 20:05:24.118116 (+3350034us) ts_tablet_manager.cc:1145] Deleting tablet 39c2ebe4ada5426590ec3769c8662644
0903 20:05:24.118127 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:24.118238 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3349793}
I20250903 20:05:24.118654 27685 tablet_replica.cc:331] T 46e3a65118b744298b5536f88c22f1db P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:24.118916 27685 raft_consensus.cc:2241] T 46e3a65118b744298b5536f88c22f1db P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:24.120632 27321 leader_election.cc:304] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:24.121470 28499 raft_consensus.cc:2802] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:24.121600 28499 raft_consensus.cc:491] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:24.121740 28499 raft_consensus.cc:3058] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:24.126829 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 91169fff8a6649828648ac97ccf65fdc (DEFAULT_TABLE table=default.Table_cb368db83ae64dc784cc454f9224c3e2 [id=24484984ae4347caa6cc3b6133abf245]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.126977 27692 tablet_bootstrap.cc:654] T 4be9bdc390de4a74a396003c9c3c2b71 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:24.128072 28499 raft_consensus.cc:513] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.123970 27687 ts_tablet_manager.cc:1918] T 369c750ff5bf449e989d95ffd89497d4 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:24.129065 27687 log.cc:1199] T 369c750ff5bf449e989d95ffd89497d4 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/369c750ff5bf449e989d95ffd89497d4
I20250903 20:05:24.131130 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 4be9bdc390de4a74a396003c9c3c2b71 (table default.Table_452fdb47678d4520828b9a3db6ddb237 [id=d945a830c86244bca66cb2652e64966d]) successfully deleted
W20250903 20:05:24.131556 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 1bf69315f07a470bb2b897ad7c590729 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1bf69315f07a470bb2b897ad7c590729 already in progress: creating tablet
I20250903 20:05:24.133491 27687 ts_tablet_manager.cc:1939] T 369c750ff5bf449e989d95ffd89497d4 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:24.134538 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "885fff47fea14b84ac6b34a57b827b4b" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
W20250903 20:05:24.135351 27320 leader_election.cc:343] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:24.135711 28499 leader_election.cc:290] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:24.136633 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 794865d716134382a939186d7144c4bb (DEFAULT_TABLE table=default.Table_a1bcf29f05be4350b110670fdbf011d1 [id=7490584c886142799d6328d2daa07272]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.137310 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 794865d716134382a939186d7144c4bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.144013 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 700) took 1549 ms. Trace:
I20250903 20:05:24.148736 27348 tablet_service.cc:1468] Processing CreateTablet for tablet 31edb50c250f42739f82cdaaf1525266 (DEFAULT_TABLE table=default.Table_c27464eb362a44268d9c085034dd3ff4 [id=e81d896683ca4c8f9267091dd554e1b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.149214 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31edb50c250f42739f82cdaaf1525266. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.145850 27687 rpcz_store.cc:276] 0903 20:05:22.594514 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.594577 (+ 63us) service_pool.cc:225] Handling call
0903 20:05:23.945698 (+1351121us) ts_tablet_manager.cc:1145] Deleting tablet 369c750ff5bf449e989d95ffd89497d4
0903 20:05:23.945708 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:24.009302 (+ 63594us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:24.123751 (+114449us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:24.143411 (+ 19660us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1350609}
I20250903 20:05:24.151896 27687 tablet_replica.cc:331] T 07565ea2b5f34dcfb407ff3bfde17243 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:24.157584 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 369c750ff5bf449e989d95ffd89497d4 (table default.Table_b0b8582c6c054baab4688a30cfe5f767 [id=feefb00b97c34fa0a5ed8af66004830e]) successfully deleted
I20250903 20:05:24.157315 27362 tablet_service.cc:1468] Processing CreateTablet for tablet a785adcd0bd943fc97b201b6d29073f4 (DEFAULT_TABLE table=default.Table_53e3669346c9467e8e27c3a33c6e1660 [id=5f1249a1893e4f83a8fa045d14637183]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.152107 27687 raft_consensus.cc:2241] T 07565ea2b5f34dcfb407ff3bfde17243 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:24.159695 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a785adcd0bd943fc97b201b6d29073f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.159628 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 31edb50c250f42739f82cdaaf1525266 (DEFAULT_TABLE table=default.Table_c27464eb362a44268d9c085034dd3ff4 [id=e81d896683ca4c8f9267091dd554e1b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.159677 27494 tablet_service.cc:1468] Processing CreateTablet for tablet ab64604c56914379943c03129ccc2e7c (DEFAULT_TABLE table=default.Table_7532aa050a154ef1824ea9f89f029692 [id=67f3cd16ccef44c7b31741438c4205a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.160030 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31edb50c250f42739f82cdaaf1525266. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.160295 27494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab64604c56914379943c03129ccc2e7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.160301 27687 raft_consensus.cc:2270] T 07565ea2b5f34dcfb407ff3bfde17243 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:24.164623 27687 ts_tablet_manager.cc:1905] T 07565ea2b5f34dcfb407ff3bfde17243 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:24.166685 27488 tablet_service.cc:1468] Processing CreateTablet for tablet a785adcd0bd943fc97b201b6d29073f4 (DEFAULT_TABLE table=default.Table_53e3669346c9467e8e27c3a33c6e1660 [id=5f1249a1893e4f83a8fa045d14637183]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.167339 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a785adcd0bd943fc97b201b6d29073f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.158277 27348 tablet_service.cc:1468] Processing CreateTablet for tablet ab64604c56914379943c03129ccc2e7c (DEFAULT_TABLE table=default.Table_7532aa050a154ef1824ea9f89f029692 [id=67f3cd16ccef44c7b31741438c4205a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.168318 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab64604c56914379943c03129ccc2e7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.174139 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 72fcb7b0cba34313a7dc2b8c29e120fc (DEFAULT_TABLE table=default.Table_e914faf335d44d019f91bda71758c473 [id=374b9abdf6a54b38876a96d8ec4a0368]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.174233 27499 tablet_service.cc:1468] Processing CreateTablet for tablet d1a254d560904b9aa31b8ee7de65ffa5 (DEFAULT_TABLE table=default.Table_81922f27b2de428ba78422c5be13fcd9 [id=0b37a13be34b403d801b712362036a5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.174664 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72fcb7b0cba34313a7dc2b8c29e120fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.174894 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1a254d560904b9aa31b8ee7de65ffa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.185345 28499 raft_consensus.cc:491] T ac9b5d4f39db4f05a15316585040d142 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:24.185567 28499 raft_consensus.cc:513] T ac9b5d4f39db4f05a15316585040d142 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.185747 27693 tablet_bootstrap.cc:492] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:24.185863 27693 ts_tablet_manager.cc:1397] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.101s user 0.035s sys 0.035s
I20250903 20:05:24.186656 27693 raft_consensus.cc:357] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.186800 27693 raft_consensus.cc:383] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:24.186847 27693 raft_consensus.cc:738] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:24.187073 27693 consensus_queue.cc:260] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.187152 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac9b5d4f39db4f05a15316585040d142" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:24.187583 27693 ts_tablet_manager.cc:1428] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:24.187794 27693 tablet_bootstrap.cc:492] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:24.189397 27693 tablet_bootstrap.cc:654] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:24.199625 27320 leader_election.cc:343] T ac9b5d4f39db4f05a15316585040d142 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:24.158459 27356 tablet_service.cc:1468] Processing CreateTablet for tablet 72fcb7b0cba34313a7dc2b8c29e120fc (DEFAULT_TABLE table=default.Table_e914faf335d44d019f91bda71758c473 [id=374b9abdf6a54b38876a96d8ec4a0368]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.200213 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72fcb7b0cba34313a7dc2b8c29e120fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.201539 27693 tablet_bootstrap.cc:492] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:24.201663 27693 ts_tablet_manager.cc:1397] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.002s
I20250903 20:05:24.202191 28499 leader_election.cc:290] T ac9b5d4f39db4f05a15316585040d142 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:24.202682 27365 tablet_service.cc:1468] Processing CreateTablet for tablet d1a254d560904b9aa31b8ee7de65ffa5 (DEFAULT_TABLE table=default.Table_81922f27b2de428ba78422c5be13fcd9 [id=0b37a13be34b403d801b712362036a5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.202592 27693 raft_consensus.cc:357] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.203262 27693 raft_consensus.cc:383] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:24.203377 27693 raft_consensus.cc:738] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:24.203661 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1a254d560904b9aa31b8ee7de65ffa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.133414 27244 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "885fff47fea14b84ac6b34a57b827b4b" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:24.206081 27244 raft_consensus.cc:3058] T 885fff47fea14b84ac6b34a57b827b4b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:24.203619 27693 consensus_queue.cc:260] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.206362 27685 raft_consensus.cc:2270] T 46e3a65118b744298b5536f88c22f1db P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:24.206687 27693 ts_tablet_manager.cc:1428] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250903 20:05:24.206954 27693 tablet_bootstrap.cc:492] T cb4d99139516405bb65c7480ddf259f1 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:24.207034 27685 ts_tablet_manager.cc:1905] T 46e3a65118b744298b5536f88c22f1db P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:24.208710 27693 tablet_bootstrap.cc:654] T cb4d99139516405bb65c7480ddf259f1 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:24.209884 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91169fff8a6649828648ac97ccf65fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.211591 27227 tablet_service.cc:1468] Processing CreateTablet for tablet a7f25f00dff7426195ac51214a14022b (DEFAULT_TABLE table=default.Table_4bc2d6a26f9b41f59aaee304f156a6c5 [id=28e973c60f364bdd9314ff5a1abbd73d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.211951 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7f25f00dff7426195ac51214a14022b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.212453 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac9b5d4f39db4f05a15316585040d142" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:24.212641 27248 raft_consensus.cc:2466] T ac9b5d4f39db4f05a15316585040d142 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:24.213109 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet aa31f2ddb9f84a8093aa1c6f64b278c9
I20250903 20:05:24.213387 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet ebc222df5d1445fea88939e1152142b9
I20250903 20:05:24.213685 27063 catalog_manager.cc:6652] ReplaceTablet: tablet aa31f2ddb9f84a8093aa1c6f64b278c9 deleted and replaced by tablet 84bd22cd8bd6462982d388222858972a
I20250903 20:05:24.214012 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ee23ae4ae2834af98c0a3bef517f0e60" } new_table_name: "default.Table_7849ab155085483f9e0ba20afc6f34a9" modify_external_catalogs: true
I20250903 20:05:24.215179 27224 tablet_service.cc:1468] Processing CreateTablet for tablet 794865d716134382a939186d7144c4bb (DEFAULT_TABLE table=default.Table_a1bcf29f05be4350b110670fdbf011d1 [id=7490584c886142799d6328d2daa07272]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.215571 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 794865d716134382a939186d7144c4bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.216104 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet ebc222df5d1445fea88939e1152142b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:23 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:24.216928 27321 leader_election.cc:304] T ac9b5d4f39db4f05a15316585040d142 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:24.217542 28500 raft_consensus.cc:2802] T ac9b5d4f39db4f05a15316585040d142 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:24.217617 28500 raft_consensus.cc:491] T ac9b5d4f39db4f05a15316585040d142 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:24.217675 28500 raft_consensus.cc:3058] T ac9b5d4f39db4f05a15316585040d142 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:24.218374 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 5c42d84a6a5649208b0539a10da21aec
I20250903 20:05:24.219059 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0ac35f89d4d94e8fa94f9c3f853bf275" } new_table_name: "default.Table_5f0199a660844c15a254a062c1540498" modify_external_catalogs: true
W20250903 20:05:24.219754 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet ebc222df5d1445fea88939e1152142b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ebc222df5d1445fea88939e1152142b9 already in progress: creating tablet
I20250903 20:05:24.220412 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet ed7c564138cc43e5a3bb4e92fb5060b6
I20250903 20:05:24.221385 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet de8ac9f984be46ec8f021928de12a090
I20250903 20:05:24.221766 27066 catalog_manager.cc:6652] ReplaceTablet: tablet de8ac9f984be46ec8f021928de12a090 deleted and replaced by tablet 90776f6369744cec8ce380b7695c44f1
I20250903 20:05:24.221185 27687 ts_tablet_manager.cc:1918] T 07565ea2b5f34dcfb407ff3bfde17243 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:24.222379 27505 tablet_service.cc:1515] Processing DeleteTablet for tablet ed7c564138cc43e5a3bb4e92fb5060b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:24.222258 27687 log.cc:1199] T 07565ea2b5f34dcfb407ff3bfde17243 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/07565ea2b5f34dcfb407ff3bfde17243
I20250903 20:05:24.222693 28500 raft_consensus.cc:513] T ac9b5d4f39db4f05a15316585040d142 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.223156 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_fae7d62089f24ca88970b4691b51f867"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:24.223367 28500 leader_election.cc:290] T ac9b5d4f39db4f05a15316585040d142 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:24.223608 28396 raft_consensus.cc:491] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250903 20:05:24.223769 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fae7d62089f24ca88970b4691b51f867 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:24.223762 28396 raft_consensus.cc:513] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:24.224412 28396 leader_election.cc:290] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:24.225232 27687 ts_tablet_manager.cc:1939] T 07565ea2b5f34dcfb407ff3bfde17243 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:24.226668 27065 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 912eee5b48ef467bbd6bbe34f8aa390c (table default.Table_f1f4f57d96f04f0caff06d9a642f5c46 [id=fdc7899c8b334d309b834ae1daa14fa0]). Expected version 4 got 0
I20250903 20:05:24.226953 27505 tablet_service.cc:1515] Processing DeleteTablet for tablet de8ac9f984be46ec8f021928de12a090 with delete_type TABLET_DATA_DELETED (replaced by tablet 90776f6369744cec8ce380b7695c44f1) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:24.227689 27684 tablet_replica.cc:331] T de8ac9f984be46ec8f021928de12a090 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:24.227900 27684 raft_consensus.cc:2241] T de8ac9f984be46ec8f021928de12a090 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:24.228111 27684 raft_consensus.cc:2270] T de8ac9f984be46ec8f021928de12a090 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:24.228719 27684 ts_tablet_manager.cc:1905] T de8ac9f984be46ec8f021928de12a090 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:24.229161 27382 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0bdcbaa0169c4ab7ba6ec038b50482fe" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:24.229343 27382 raft_consensus.cc:2466] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:24.229387 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac9b5d4f39db4f05a15316585040d142" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:24.230341 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c42d84a6a5649208b0539a10da21aec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:24.230804 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 548b50819f694fa3ad508da9f442a99c
I20250903 20:05:24.231935 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_564e21bb80a94d5888b76f1471d9b937"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:24.232513 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 701) took 1637 ms. Trace:
W20250903 20:05:24.232707 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_564e21bb80a94d5888b76f1471d9b937 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:24.233103 27687 rpcz_store.cc:276] 0903 20:05:22.595281 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.595329 (+ 48us) service_pool.cc:225] Handling call
0903 20:05:24.151819 (+1556490us) ts_tablet_manager.cc:1145] Deleting tablet 07565ea2b5f34dcfb407ff3bfde17243
0903 20:05:24.151829 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:24.184627 (+ 32798us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:24.220376 (+ 35749us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:24.232489 (+ 12113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1556238}
I20250903 20:05:24.235421 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_480743ddb101436ca551a49c01a368a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:24.236088 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_480743ddb101436ca551a49c01a368a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:24.236143 27462 leader_election.cc:304] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
W20250903 20:05:24.236686 27320 leader_election.cc:343] T ac9b5d4f39db4f05a15316585040d142 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:24.236442 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 702) took 1640 ms. Trace:
I20250903 20:05:24.236922 28396 raft_consensus.cc:2802] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:24.237013 28396 raft_consensus.cc:491] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:24.237098 28396 raft_consensus.cc:3058] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:24.237021 27687 rpcz_store.cc:276] 0903 20:05:22.596153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.596193 (+ 40us) service_pool.cc:225] Handling call
0903 20:05:24.235180 (+1638987us) ts_tablet_manager.cc:1145] Deleting tablet f9375fff5f5945078c4412aa21c06f0d
0903 20:05:24.235192 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:24.236252 (+ 1060us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":896,"tablet-delete.queue_time_us":1638610}
I20250903 20:05:24.222090 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_7039490d1b01480f9a14cdf90ce4643f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:24.238965 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7039490d1b01480f9a14cdf90ce4643f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:24.239944 28396 raft_consensus.cc:513] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:24.240923 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 548b50819f694fa3ad508da9f442a99c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:24.241324 28396 leader_election.cc:290] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:24.241798 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet ed7c564138cc43e5a3bb4e92fb5060b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed7c564138cc43e5a3bb4e92fb5060b6 already in progress: creating tablet
I20250903 20:05:24.242277 27433 heartbeater.cc:344] Connected to a master server at 127.26.65.125:44059
I20250903 20:05:24.245635 27687 tablet_replica.cc:331] T bf2545037c214bb1b5060575e3cdf81a P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:24.245832 27687 raft_consensus.cc:2241] T bf2545037c214bb1b5060575e3cdf81a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:24.245990 27687 raft_consensus.cc:2270] T bf2545037c214bb1b5060575e3cdf81a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:24.247498 27687 ts_tablet_manager.cc:1905] T bf2545037c214bb1b5060575e3cdf81a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:24.252717 27382 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0bdcbaa0169c4ab7ba6ec038b50482fe" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:24.252907 27382 raft_consensus.cc:3058] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:24.254738 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b87f7ee9a2444f9892fcadaa791f93dc" } new_table_name: "default.Table_89d23e941ec94830ade8433eb65b5e66" modify_external_catalogs: true
I20250903 20:05:24.255512 27382 raft_consensus.cc:2466] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:24.255738 27684 ts_tablet_manager.cc:1918] T de8ac9f984be46ec8f021928de12a090 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:24.255882 27684 log.cc:1199] T de8ac9f984be46ec8f021928de12a090 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/de8ac9f984be46ec8f021928de12a090
I20250903 20:05:24.257066 27462 leader_election.cc:304] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:24.257892 28500 raft_consensus.cc:491] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:24.265339 28500 raft_consensus.cc:513] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.266119 28500 leader_election.cc:290] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:24.267697 27246 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c398634d6304a12b35c0579fba4c813" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:24.267906 27246 raft_consensus.cc:2466] T 7c398634d6304a12b35c0579fba4c813 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:24.268481 27321 leader_election.cc:304] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:24.268813 27684 ts_tablet_manager.cc:1939] T de8ac9f984be46ec8f021928de12a090 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:24.269467 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c398634d6304a12b35c0579fba4c813" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:24.270079 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet aa31f2ddb9f84a8093aa1c6f64b278c9 with delete_type TABLET_DATA_DELETED (replaced by tablet 84bd22cd8bd6462982d388222858972a) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:24.270164 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet de8ac9f984be46ec8f021928de12a090 (table default.Table_eaff993b7bb7437580aae238faa74b3e [id=2650c8259ef4489695c56b6a2ba898d3]) successfully deleted
I20250903 20:05:24.270560 28500 raft_consensus.cc:2802] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:24.270653 28500 raft_consensus.cc:491] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:24.270717 28500 raft_consensus.cc:3058] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:24.273231 28500 raft_consensus.cc:513] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:24.274298 27320 leader_election.cc:343] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:24.275192 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c398634d6304a12b35c0579fba4c813" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
W20250903 20:05:24.275789 27320 leader_election.cc:343] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:24.276332 28396 raft_consensus.cc:2802] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
W20250903 20:05:24.277469 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 5c42d84a6a5649208b0539a10da21aec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c42d84a6a5649208b0539a10da21aec already in progress: creating tablet
W20250903 20:05:24.277787 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 548b50819f694fa3ad508da9f442a99c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 548b50819f694fa3ad508da9f442a99c already in progress: creating tablet
W20250903 20:05:24.278159 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet aa31f2ddb9f84a8093aa1c6f64b278c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa31f2ddb9f84a8093aa1c6f64b278c9 already in progress: creating tablet
I20250903 20:05:24.280105 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 07565ea2b5f34dcfb407ff3bfde17243 (table default.Table_4810ae9efbde43bc8c237c6e135aa5aa [id=dbeb164ccbf544cf947af9bf656559c7]) successfully deleted
I20250903 20:05:24.280673 27356 tablet_service.cc:1515] Processing DeleteTablet for tablet ebc222df5d1445fea88939e1152142b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:23 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:24.280798 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet ed7c564138cc43e5a3bb4e92fb5060b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:24.281168 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet de8ac9f984be46ec8f021928de12a090 with delete_type TABLET_DATA_DELETED (replaced by tablet 90776f6369744cec8ce380b7695c44f1) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:24.281503 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 548b50819f694fa3ad508da9f442a99c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:24.281638 27348 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c42d84a6a5649208b0539a10da21aec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:24.281673 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet aa31f2ddb9f84a8093aa1c6f64b278c9 with delete_type TABLET_DATA_DELETED (replaced by tablet 84bd22cd8bd6462982d388222858972a) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:24.282887 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet f9375fff5f5945078c4412aa21c06f0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9375fff5f5945078c4412aa21c06f0d already in progress: creating tablet
I20250903 20:05:24.283239 28500 leader_election.cc:290] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:24.283803 28245 leader_election.cc:290] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:24.284684 27246 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c398634d6304a12b35c0579fba4c813" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:24.284864 27246 raft_consensus.cc:3058] T 7c398634d6304a12b35c0579fba4c813 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:24.286011 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 54f1ce5192f547f595dea23a9eabf8e6 consensus state has no leader
I20250903 20:05:24.286545 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ced04d6dca6d46a792f30d4f704d1468"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:24.287283 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ced04d6dca6d46a792f30d4f704d1468 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:24.287786 28396 raft_consensus.cc:695] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:24.288197 28396 consensus_queue.cc:237] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:24.287426 27687 ts_tablet_manager.cc:1918] T bf2545037c214bb1b5060575e3cdf81a P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250903 20:05:24.288718 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7345305820be4d32a3ceb227f045f009 consensus state has no leader
I20250903 20:05:24.288683 27687 log.cc:1199] T bf2545037c214bb1b5060575e3cdf81a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/bf2545037c214bb1b5060575e3cdf81a
I20250903 20:05:24.289153 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_042efc3626d2434fa91ad982d346f3ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:24.289724 27687 ts_tablet_manager.cc:1939] T bf2545037c214bb1b5060575e3cdf81a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
W20250903 20:05:24.289788 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_042efc3626d2434fa91ad982d346f3ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:24.289901 27693 tablet_bootstrap.cc:492] T cb4d99139516405bb65c7480ddf259f1 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:24.290192 27693 ts_tablet_manager.cc:1397] T cb4d99139516405bb65c7480ddf259f1 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.083s user 0.003s sys 0.017s
I20250903 20:05:24.291630 27211 tablet_service.cc:1468] Processing CreateTablet for tablet 72fcb7b0cba34313a7dc2b8c29e120fc (DEFAULT_TABLE table=default.Table_e914faf335d44d019f91bda71758c473 [id=374b9abdf6a54b38876a96d8ec4a0368]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.292189 27211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72fcb7b0cba34313a7dc2b8c29e120fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.293478 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac9b5d4f39db4f05a15316585040d142" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:24.293658 27248 raft_consensus.cc:3058] T ac9b5d4f39db4f05a15316585040d142 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:24.293622 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_26e71ae071ec449d866264a856b29b89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:24.293586 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 709) took 1678 ms. Trace:
W20250903 20:05:24.294183 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26e71ae071ec449d866264a856b29b89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:24.294929 27212 tablet_service.cc:1468] Processing CreateTablet for tablet d1a254d560904b9aa31b8ee7de65ffa5 (DEFAULT_TABLE table=default.Table_81922f27b2de428ba78422c5be13fcd9 [id=0b37a13be34b403d801b712362036a5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.294116 27687 rpcz_store.cc:276] 0903 20:05:22.614758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.614810 (+ 52us) service_pool.cc:225] Handling call
0903 20:05:24.238716 (+1623906us) ts_tablet_manager.cc:1145] Deleting tablet bf2545037c214bb1b5060575e3cdf81a
0903 20:05:24.245569 (+ 6853us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:24.267451 (+ 21882us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:24.287218 (+ 19767us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:24.293447 (+ 6229us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1623523}
I20250903 20:05:24.294579 27693 raft_consensus.cc:357] T cb4d99139516405bb65c7480ddf259f1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.295799 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b68da0901fe14d3198aeb59b315c7acd" } new_table_name: "default.Table_5c8743c0e4e444fb8eac65cde6000a06" modify_external_catalogs: true
W20250903 20:05:24.295924 27462 leader_election.cc:343] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 0bdcbaa0169c4ab7ba6ec038b50482fe
I20250903 20:05:24.295712 27693 raft_consensus.cc:383] T cb4d99139516405bb65c7480ddf259f1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:24.296206 27693 raft_consensus.cc:738] T cb4d99139516405bb65c7480ddf259f1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:24.296423 27687 tablet_replica.cc:331] T fbfabb6ff1714d2e921f45256b543759 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:24.297421 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_7356ad4dcf4d4c28a78fdb0988bd6059"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:24.297598 27210 tablet_service.cc:1468] Processing CreateTablet for tablet f63b9a066ae94ef3afe3badaef5bcafd (DEFAULT_TABLE table=default.Table_789d6e974e2f44ff8d04b1a39adcb6a7 [id=1d7a112662d24b5b828cb7ac4b073c43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.297474 27687 raft_consensus.cc:2241] T fbfabb6ff1714d2e921f45256b543759 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250903 20:05:24.297905 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7356ad4dcf4d4c28a78fdb0988bd6059 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:24.297408 27693 consensus_queue.cc:260] T cb4d99139516405bb65c7480ddf259f1 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.298359 27687 raft_consensus.cc:2270] T fbfabb6ff1714d2e921f45256b543759 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:24.298902 27228 tablet_service.cc:1468] Processing CreateTablet for tablet a785adcd0bd943fc97b201b6d29073f4 (DEFAULT_TABLE table=default.Table_53e3669346c9467e8e27c3a33c6e1660 [id=5f1249a1893e4f83a8fa045d14637183]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.299666 27215 tablet_service.cc:1468] Processing CreateTablet for tablet ab64604c56914379943c03129ccc2e7c (DEFAULT_TABLE table=default.Table_7532aa050a154ef1824ea9f89f029692 [id=67f3cd16ccef44c7b31741438c4205a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.300367 27693 ts_tablet_manager.cc:1428] T cb4d99139516405bb65c7480ddf259f1 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20250903 20:05:24.301331 27213 tablet_service.cc:1468] Processing CreateTablet for tablet c066691b278c4278937b4d8ef98b395f (DEFAULT_TABLE table=default.Table_193a0ff511204b3b8130c17e783b51ad [id=957e293e05cc4600aac74d6c0dfcf82a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.301497 28512 raft_consensus.cc:491] T 6238740ce1894e899be9a40321f0ab73 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:24.301789 28512 raft_consensus.cc:513] T 6238740ce1894e899be9a40321f0ab73 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:24.303419 28512 leader_election.cc:290] T 6238740ce1894e899be9a40321f0ab73 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:24.301206 27687 ts_tablet_manager.cc:1905] T fbfabb6ff1714d2e921f45256b543759 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:24.304837 27245 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6238740ce1894e899be9a40321f0ab73" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:24.304745 27693 tablet_bootstrap.cc:492] T fa05f54ebc324b40855fa3288760e9e9 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
W20250903 20:05:24.311952 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0ab15dd783ff49868d6f9707369a5d3a consensus state has no leader
I20250903 20:05:24.312587 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_76100d98b0774693ab71b084505d007a" } new_table_name: "default.Table_ba146e717b8d4d7db2b5bec3d05a16a6" modify_external_catalogs: true
W20250903 20:05:24.313058 27462 leader_election.cc:343] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 0bdcbaa0169c4ab7ba6ec038b50482fe
I20250903 20:05:24.314472 27223 tablet_service.cc:1515] Processing DeleteTablet for tablet aa31f2ddb9f84a8093aa1c6f64b278c9 with delete_type TABLET_DATA_DELETED (replaced by tablet 84bd22cd8bd6462982d388222858972a) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:24.314713 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet ebc222df5d1445fea88939e1152142b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:23 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:24.315371 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c42d84a6a5649208b0539a10da21aec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:24.315423 27382 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cc6cf579e2642e99fefed723b0ee900" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:24.315591 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet de8ac9f984be46ec8f021928de12a090 with delete_type TABLET_DATA_DELETED (replaced by tablet 90776f6369744cec8ce380b7695c44f1) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:24.316166 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet 548b50819f694fa3ad508da9f442a99c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:24.316269 27184 leader_election.cc:343] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:24.317030 28245 raft_consensus.cc:491] T c6d67cd0c0304c77ba571b414878dad2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:24.317157 28245 raft_consensus.cc:513] T c6d67cd0c0304c77ba571b414878dad2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.317672 28245 leader_election.cc:290] T c6d67cd0c0304c77ba571b414878dad2 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:24.318537 27693 tablet_bootstrap.cc:654] T fa05f54ebc324b40855fa3288760e9e9 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:24.320017 27183 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 486296us
I20250903 20:05:24.322569 27692 tablet_bootstrap.cc:492] T 4be9bdc390de4a74a396003c9c3c2b71 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:24.322817 27692 ts_tablet_manager.cc:1397] T 4be9bdc390de4a74a396003c9c3c2b71 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.286s user 0.026s sys 0.008s
I20250903 20:05:24.323603 27692 raft_consensus.cc:357] T 4be9bdc390de4a74a396003c9c3c2b71 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.323854 27692 raft_consensus.cc:383] T 4be9bdc390de4a74a396003c9c3c2b71 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:24.323954 27692 raft_consensus.cc:738] T 4be9bdc390de4a74a396003c9c3c2b71 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:24.324230 27692 consensus_queue.cc:260] T 4be9bdc390de4a74a396003c9c3c2b71 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.317776 28245 raft_consensus.cc:491] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:24.331743 27244 raft_consensus.cc:2466] T 885fff47fea14b84ac6b34a57b827b4b P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:24.332221 27692 ts_tablet_manager.cc:1428] T 4be9bdc390de4a74a396003c9c3c2b71 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.009s user 0.001s sys 0.007s
I20250903 20:05:24.332276 27321 leader_election.cc:304] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:24.332700 27692 tablet_bootstrap.cc:492] T 912eee5b48ef467bbd6bbe34f8aa390c P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:24.332623 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cc6cf579e2642e99fefed723b0ee900" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:24.332773 28298 raft_consensus.cc:491] T 912eee5b48ef467bbd6bbe34f8aa390c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:24.332903 28298 raft_consensus.cc:513] T 912eee5b48ef467bbd6bbe34f8aa390c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:24.333003 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6d67cd0c0304c77ba571b414878dad2" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:24.333525 28298 leader_election.cc:290] T 912eee5b48ef467bbd6bbe34f8aa390c P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:24.333652 28298 raft_consensus.cc:2802] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:24.333753 28298 raft_consensus.cc:695] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:24.333981 28298 consensus_queue.cc:237] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.334159 27692 tablet_bootstrap.cc:654] T 912eee5b48ef467bbd6bbe34f8aa390c P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:24.337198 27223 tablet_service.cc:1515] Processing DeleteTablet for tablet ed7c564138cc43e5a3bb4e92fb5060b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:24.337905 27687 ts_tablet_manager.cc:1918] T fbfabb6ff1714d2e921f45256b543759 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:24.338268 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab64604c56914379943c03129ccc2e7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.338644 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c066691b278c4278937b4d8ef98b395f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.338444 27687 log.cc:1199] T fbfabb6ff1714d2e921f45256b543759 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/fbfabb6ff1714d2e921f45256b543759
I20250903 20:05:24.339097 27228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a785adcd0bd943fc97b201b6d29073f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.339334 27210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f63b9a066ae94ef3afe3badaef5bcafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.340350 27687 ts_tablet_manager.cc:1939] T fbfabb6ff1714d2e921f45256b543759 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:24.341866 27692 tablet_bootstrap.cc:492] T 912eee5b48ef467bbd6bbe34f8aa390c P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:24.342111 27692 ts_tablet_manager.cc:1397] T 912eee5b48ef467bbd6bbe34f8aa390c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.001s
I20250903 20:05:24.342950 27692 raft_consensus.cc:357] T 912eee5b48ef467bbd6bbe34f8aa390c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:24.343369 27692 raft_consensus.cc:383] T 912eee5b48ef467bbd6bbe34f8aa390c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:24.343484 27692 raft_consensus.cc:738] T 912eee5b48ef467bbd6bbe34f8aa390c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:24.343405 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6238740ce1894e899be9a40321f0ab73" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:24.343765 27692 consensus_queue.cc:260] T 912eee5b48ef467bbd6bbe34f8aa390c P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:24.343952 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "912eee5b48ef467bbd6bbe34f8aa390c" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:24.344157 27524 raft_consensus.cc:2466] T 912eee5b48ef467bbd6bbe34f8aa390c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:24.344640 27692 ts_tablet_manager.cc:1428] T 912eee5b48ef467bbd6bbe34f8aa390c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:24.345057 27692 tablet_bootstrap.cc:492] T fa05f54ebc324b40855fa3288760e9e9 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:24.346474 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 708) took 1731 ms. Trace:
I20250903 20:05:24.347199 27229 tablet_service.cc:1468] Processing CreateTablet for tablet 31edb50c250f42739f82cdaaf1525266 (DEFAULT_TABLE table=default.Table_c27464eb362a44268d9c085034dd3ff4 [id=e81d896683ca4c8f9267091dd554e1b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.347556 27692 tablet_bootstrap.cc:654] T fa05f54ebc324b40855fa3288760e9e9 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:24.347082 27687 rpcz_store.cc:276] 0903 20:05:22.614509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.618274 (+ 3765us) service_pool.cc:225] Handling call
0903 20:05:24.296271 (+1677997us) ts_tablet_manager.cc:1145] Deleting tablet fbfabb6ff1714d2e921f45256b543759
0903 20:05:24.296283 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:24.326557 (+ 30274us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:24.337517 (+ 10960us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:24.346167 (+ 8650us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":107,"tablet-delete.queue_time_us":1677389}
I20250903 20:05:24.348188 27245 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "912eee5b48ef467bbd6bbe34f8aa390c" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:24.348390 27245 raft_consensus.cc:2466] T 912eee5b48ef467bbd6bbe34f8aa390c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:24.348970 27321 leader_election.cc:304] T 912eee5b48ef467bbd6bbe34f8aa390c P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:24.349376 28315 raft_consensus.cc:2802] T 912eee5b48ef467bbd6bbe34f8aa390c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:24.349479 28315 raft_consensus.cc:491] T 912eee5b48ef467bbd6bbe34f8aa390c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:24.349579 28315 raft_consensus.cc:3058] T 912eee5b48ef467bbd6bbe34f8aa390c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:24.349778 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 721) took 1263 ms. Trace:
I20250903 20:05:24.350180 27687 rpcz_store.cc:276] 0903 20:05:23.085764 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:23.086523 (+ 759us) service_pool.cc:225] Handling call
0903 20:05:24.348840 (+1262317us) ts_tablet_manager.cc:1145] Deleting tablet 5253d82a41494ce59bb96928179dfdd9
0903 20:05:24.348905 (+ 65us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:24.349711 (+ 806us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1262074}
I20250903 20:05:24.352512 27692 tablet_bootstrap.cc:492] T fa05f54ebc324b40855fa3288760e9e9 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:24.352712 27692 ts_tablet_manager.cc:1397] T fa05f54ebc324b40855fa3288760e9e9 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20250903 20:05:24.352574 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 723) took 1266 ms. Trace:
I20250903 20:05:24.352957 27687 rpcz_store.cc:276] 0903 20:05:23.085989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:23.086760 (+ 771us) service_pool.cc:225] Handling call
0903 20:05:24.352058 (+1265298us) ts_tablet_manager.cc:1145] Deleting tablet 9767293447214a7eb68ed03d7373db7f
0903 20:05:24.352070 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:24.352550 (+ 480us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1265039}
I20250903 20:05:24.353494 27692 raft_consensus.cc:357] T fa05f54ebc324b40855fa3288760e9e9 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.353669 27692 raft_consensus.cc:383] T fa05f54ebc324b40855fa3288760e9e9 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:24.353718 27692 raft_consensus.cc:738] T fa05f54ebc324b40855fa3288760e9e9 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:24.353910 27692 consensus_queue.cc:260] T fa05f54ebc324b40855fa3288760e9e9 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.354552 27692 ts_tablet_manager.cc:1428] T fa05f54ebc324b40855fa3288760e9e9 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:24.354875 27692 tablet_bootstrap.cc:492] T 7c398634d6304a12b35c0579fba4c813 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:24.354794 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 722) took 1268 ms. Trace:
I20250903 20:05:24.355309 28315 raft_consensus.cc:513] T 912eee5b48ef467bbd6bbe34f8aa390c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:24.355142 27687 rpcz_store.cc:276] 0903 20:05:23.085867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:23.086952 (+ 1085us) service_pool.cc:225] Handling call
0903 20:05:24.354364 (+1267412us) ts_tablet_manager.cc:1145] Deleting tablet cd6d6bb514134bdf8e3bc0aaad180732
0903 20:05:24.354412 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:24.354744 (+ 332us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1267204}
I20250903 20:05:24.355999 28315 leader_election.cc:290] T 912eee5b48ef467bbd6bbe34f8aa390c P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:24.356318 27692 tablet_bootstrap.cc:654] T 7c398634d6304a12b35c0579fba4c813 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:24.356969 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 720) took 1271 ms. Trace:
I20250903 20:05:24.357362 27687 rpcz_store.cc:276] 0903 20:05:23.085612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:23.087135 (+ 1523us) service_pool.cc:225] Handling call
0903 20:05:24.356561 (+1269426us) ts_tablet_manager.cc:1145] Deleting tablet 09fa32312c9648fbbd6fd327e9738066
0903 20:05:24.356577 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:24.356949 (+ 372us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1269189}
I20250903 20:05:24.359202 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 724) took 1267 ms. Trace:
I20250903 20:05:24.359745 27687 rpcz_store.cc:276] 0903 20:05:23.091544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:23.091829 (+ 285us) service_pool.cc:225] Handling call
0903 20:05:24.358679 (+1266850us) ts_tablet_manager.cc:1145] Deleting tablet 443d77f2593b44afa3e5aae4c5b0a651
0903 20:05:24.358690 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:24.359087 (+ 397us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1266599}
I20250903 20:05:24.361510 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 725) took 1171 ms. Trace:
I20250903 20:05:24.361887 27687 rpcz_store.cc:276] 0903 20:05:23.190332 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:23.190397 (+ 65us) service_pool.cc:225] Handling call
0903 20:05:24.361088 (+1170691us) ts_tablet_manager.cc:1145] Deleting tablet cb2f50683e904dd39ee6eb128717d38c
0903 20:05:24.361099 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:24.361490 (+ 391us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1170403}
I20250903 20:05:24.363091 27462 leader_election.cc:304] T 6238740ce1894e899be9a40321f0ab73 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:24.363458 28516 raft_consensus.cc:2747] T 6238740ce1894e899be9a40321f0ab73 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:24.363528 27687 tablet_replica.cc:331] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:24.364131 27524 raft_consensus.cc:3058] T 4be9bdc390de4a74a396003c9c3c2b71 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:24.365114 27687 raft_consensus.cc:2241] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:24.356266 28245 raft_consensus.cc:513] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.367498 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "912eee5b48ef467bbd6bbe34f8aa390c" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:24.367664 27522 raft_consensus.cc:3058] T 912eee5b48ef467bbd6bbe34f8aa390c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:24.368537 27244 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "912eee5b48ef467bbd6bbe34f8aa390c" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:24.370422 27522 raft_consensus.cc:2466] T 912eee5b48ef467bbd6bbe34f8aa390c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:24.372120 27248 raft_consensus.cc:2466] T ac9b5d4f39db4f05a15316585040d142 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:24.374958 27212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1a254d560904b9aa31b8ee7de65ffa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.374637 27687 raft_consensus.cc:2270] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:24.378396 27687 ts_tablet_manager.cc:1905] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:24.378974 28501 consensus_queue.cc:1035] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:24.379842 27320 leader_election.cc:304] T 912eee5b48ef467bbd6bbe34f8aa390c P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:24.380604 28501 raft_consensus.cc:2802] T 912eee5b48ef467bbd6bbe34f8aa390c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:24.380770 28501 raft_consensus.cc:695] T 912eee5b48ef467bbd6bbe34f8aa390c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:24.381098 28501 consensus_queue.cc:237] T 912eee5b48ef467bbd6bbe34f8aa390c P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:24.390652 27244 raft_consensus.cc:3058] T 912eee5b48ef467bbd6bbe34f8aa390c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:24.391170 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet bf2545037c214bb1b5060575e3cdf81a (table default.Table_143fea9b5e3d4acbb92adc441aeede47 [id=42c554f2f532414a88b50818e2dbb458]) successfully deleted
I20250903 20:05:24.391484 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet fbfabb6ff1714d2e921f45256b543759 (table default.Table_aab58ad5394041179974552994469305 [id=1889dee4122f4e69ac2e3d22fb4126a6]) successfully deleted
I20250903 20:05:24.391564 27692 tablet_bootstrap.cc:492] T 7c398634d6304a12b35c0579fba4c813 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:24.391727 27692 ts_tablet_manager.cc:1397] T 7c398634d6304a12b35c0579fba4c813 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.037s user 0.020s sys 0.014s
W20250903 20:05:24.392174 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 5253d82a41494ce59bb96928179dfdd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5253d82a41494ce59bb96928179dfdd9 already in progress: creating tablet
W20250903 20:05:24.392486 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 9767293447214a7eb68ed03d7373db7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9767293447214a7eb68ed03d7373db7f already in progress: creating tablet
I20250903 20:05:24.392611 27692 raft_consensus.cc:357] T 7c398634d6304a12b35c0579fba4c813 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.392818 27692 raft_consensus.cc:383] T 7c398634d6304a12b35c0579fba4c813 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:24.392877 27692 raft_consensus.cc:738] T 7c398634d6304a12b35c0579fba4c813 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
W20250903 20:05:24.393059 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet cd6d6bb514134bdf8e3bc0aaad180732 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd6d6bb514134bdf8e3bc0aaad180732 already in progress: creating tablet
W20250903 20:05:24.393307 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 09fa32312c9648fbbd6fd327e9738066 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09fa32312c9648fbbd6fd327e9738066 already in progress: creating tablet
W20250903 20:05:24.393569 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 443d77f2593b44afa3e5aae4c5b0a651 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 443d77f2593b44afa3e5aae4c5b0a651 already in progress: creating tablet
W20250903 20:05:24.393844 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet cb2f50683e904dd39ee6eb128717d38c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb2f50683e904dd39ee6eb128717d38c already in progress: creating tablet
I20250903 20:05:24.398267 27229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31edb50c250f42739f82cdaaf1525266. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.399125 27057 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 0ad64c699b294679b6fec37b5ad23528 (table default.Table_24e55a7c820c413f8514bd3ad7bd3269 [id=0904dc5d0c594bc3bbf32de2508c6195]). Expected version 3 got 0
W20250903 20:05:24.399346 27184 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 59845.5us
I20250903 20:05:24.400247 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6d67cd0c0304c77ba571b414878dad2" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:24.400595 27382 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685cba2f73b412382a69dab2c7f50c1" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:24.392527 28245 leader_election.cc:290] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:24.403267 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 639ecb9caa4a423899ab8eb35bb716d7
I20250903 20:05:24.405244 27499 tablet_service.cc:1468] Processing CreateTablet for tablet 5ec4bd6da4754fd59494f2069bb901e3 (DEFAULT_TABLE table=default.Table_c1f1c15403024868bdc2079821d61fdc [id=1b87cee2db5846c2b47fcdf948543508]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.405652 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ec4bd6da4754fd59494f2069bb901e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.407011 27505 tablet_service.cc:1468] Processing CreateTablet for tablet c9d09ecd8fe347d79283fc9a3056e844 (DEFAULT_TABLE table=default.Table_0c63411efaae4fdc900ae09cdd4eed96 [id=a8d5279615374b09860a41942f9fea3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.407354 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9d09ecd8fe347d79283fc9a3056e844. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.407562 27692 consensus_queue.cc:260] T 7c398634d6304a12b35c0579fba4c813 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.408210 27692 ts_tablet_manager.cc:1428] T 7c398634d6304a12b35c0579fba4c813 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20250903 20:05:24.408543 27692 tablet_bootstrap.cc:492] T 885fff47fea14b84ac6b34a57b827b4b P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:24.410120 27692 tablet_bootstrap.cc:654] T 885fff47fea14b84ac6b34a57b827b4b P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:24.410341 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet f71d3dd9af2543299296f21381d1d563 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:24.410480 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 553443eda16c44fa84b1db22b1363808 (DEFAULT_TABLE table=default.Table_307490b0006e4ed493ee8b2e23b7877a [id=e7b6f9faf04d47e9955ffafdb333ca30]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.410580 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 639ecb9caa4a423899ab8eb35bb716d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:23 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:24.410771 27684 tablet_replica.cc:331] T f71d3dd9af2543299296f21381d1d563 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:24.410835 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 553443eda16c44fa84b1db22b1363808. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.410933 27684 raft_consensus.cc:2241] T f71d3dd9af2543299296f21381d1d563 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:24.411072 27684 raft_consensus.cc:2270] T f71d3dd9af2543299296f21381d1d563 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:24.411767 27684 ts_tablet_manager.cc:1905] T f71d3dd9af2543299296f21381d1d563 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:24.414502 27488 tablet_service.cc:1515] Processing DeleteTablet for tablet 0bdcbaa0169c4ab7ba6ec038b50482fe with delete_type TABLET_DATA_DELETED (replaced by tablet 1bcb9d5f010f4297bab78c0183889238) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:24.420092 27065 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_272defb026dc43978827f668aeedc48c" } modify_external_catalogs: true
I20250903 20:05:24.420326 27065 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_272defb026dc43978827f668aeedc48c" } modify_external_catalogs: true
I20250903 20:05:24.421720 27687 ts_tablet_manager.cc:1918] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 20:05:24.422348 27687 log.cc:1199] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/b6b4713936134681aaef1732d46c6ab2
I20250903 20:05:24.423942 27687 ts_tablet_manager.cc:1939] T b6b4713936134681aaef1732d46c6ab2 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:24.424487 27059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_86616565ede946a5b1043149012526f5" } modify_external_catalogs: true
I20250903 20:05:24.424582 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_794a9c24e42f44bb9d18feee62639661" } modify_external_catalogs: true
I20250903 20:05:24.424703 27059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_86616565ede946a5b1043149012526f5" } modify_external_catalogs: true
I20250903 20:05:24.424782 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_794a9c24e42f44bb9d18feee62639661" } modify_external_catalogs: true
I20250903 20:05:24.431427 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 989545be137341ef8d742bb1119cc51f (DEFAULT_TABLE table=default.Table_b69c590ef46648439e74d958bf9ffdf3 [id=1caafc550c7d4f478cb019f7e01b2aee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.431914 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 989545be137341ef8d742bb1119cc51f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.434899 27494 tablet_service.cc:1468] Processing CreateTablet for tablet 9ac48450b5aa448f9f040e88dc2f98bd (DEFAULT_TABLE table=default.Table_95b24ad570fd42158c2e3dc68edf37a8 [id=082a0477652f48e5b16cc5e6aac15e50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.435285 27494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ac48450b5aa448f9f040e88dc2f98bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.431427 27488 tablet_service.cc:1468] Processing CreateTablet for tablet 33c624619c17495da7e7f55fee0a49b2 (DEFAULT_TABLE table=default.Table_43e145c502a34104b1388d584d5c7355 [id=25185952ddcb4e6cab3194234abe8643]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.438706 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33c624619c17495da7e7f55fee0a49b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.439095 27502 tablet_service.cc:1468] Processing CreateTablet for tablet 0cc6d68ca0fe45a492c77879c5d9e89c (DEFAULT_TABLE table=default.Table_585c269abe9b4aa784813fd7439fa4cc [id=1efb74befa004d8daf9cb2dcce59f324]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.439441 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cc6d68ca0fe45a492c77879c5d9e89c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.439843 27496 tablet_service.cc:1468] Processing CreateTablet for tablet 0060e6c4699a4bff8faddefb4ee50add (DEFAULT_TABLE table=default.Table_794a9c24e42f44bb9d18feee62639661 [id=836b690a27db4f40895f775da5d3a6fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.440241 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0060e6c4699a4bff8faddefb4ee50add. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.444819 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_89f979a23e14485289384ecac15e6145"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:24.445407 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_89f979a23e14485289384ecac15e6145 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:24.448266 27496 tablet_service.cc:1468] Processing CreateTablet for tablet 0ffb568443854102888cb308a269ede2 (DEFAULT_TABLE table=default.Table_32ee815023e84a3dadbc1ec42d582b42 [id=828c85a9cc3f4239bad6368481ee1332]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.447820 27687 tablet_replica.cc:331] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:24.448575 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ffb568443854102888cb308a269ede2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.450034 27687 raft_consensus.cc:2241] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:24.451143 27507 tablet_service.cc:1468] Processing CreateTablet for tablet cc71346bf84f4a708bf17b5aba174582 (DEFAULT_TABLE table=default.Table_5d1a53d4800a4927b3de164630cc0b7a [id=bb6dc54b09a24dd3aec26340d6d3ea14]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.451490 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc71346bf84f4a708bf17b5aba174582. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.453449 27687 pending_rounds.cc:70] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528: Trying to abort 1 pending ops.
I20250903 20:05:24.454191 27502 tablet_service.cc:1468] Processing CreateTablet for tablet 435c002dffd24aeaa6221ab00ee5b24b (DEFAULT_TABLE table=default.Table_392e604d6bdd46bfbe7e3b6d21529019 [id=23d521bab40643c78b35149ddab89c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.454614 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 435c002dffd24aeaa6221ab00ee5b24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.454166 27687 pending_rounds.cc:77] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384968078319616 op_type: NO_OP noop_request { }
I20250903 20:05:24.455289 27687 raft_consensus.cc:2887] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:24.456351 27499 tablet_service.cc:1468] Processing CreateTablet for tablet fa544dfdfff04a1e97676e562aeb0f40 (DEFAULT_TABLE table=default.Table_3f1f99f4ab084617b3a72447d6e84209 [id=a30b474b4c8b494db74d62e576e509a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.456729 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa544dfdfff04a1e97676e562aeb0f40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.456368 27687 raft_consensus.cc:2270] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:24.463518 27494 tablet_service.cc:1468] Processing CreateTablet for tablet 908e748d08f74ed2a3071312c95ec07c (DEFAULT_TABLE table=default.Table_e58baad01a28404886961ac38943a233 [id=0c3f0f04e2064efea6f5e9efb1bfc0aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.463958 27494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 908e748d08f74ed2a3071312c95ec07c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.464988 27684 ts_tablet_manager.cc:1918] T f71d3dd9af2543299296f21381d1d563 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:24.465433 27684 log.cc:1199] T f71d3dd9af2543299296f21381d1d563 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/f71d3dd9af2543299296f21381d1d563
I20250903 20:05:24.465795 27684 ts_tablet_manager.cc:1939] T f71d3dd9af2543299296f21381d1d563 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:24.466919 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet f71d3dd9af2543299296f21381d1d563 (table default.Table_a0582ccb94a64603a5fbdd0e531a9edf [id=6f7fece7a8d54d4786f1bab10da9129c]) successfully deleted
I20250903 20:05:24.466352 27687 ts_tablet_manager.cc:1905] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:24.488837 28499 raft_consensus.cc:491] T fe902f1b58924be1ac3a5585413f7789 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:24.489075 27522 raft_consensus.cc:3058] T 6238740ce1894e899be9a40321f0ab73 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:24.489137 28499 raft_consensus.cc:513] T fe902f1b58924be1ac3a5585413f7789 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.490525 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe902f1b58924be1ac3a5585413f7789" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:24.491005 27320 leader_election.cc:343] T fe902f1b58924be1ac3a5585413f7789 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:24.491267 28499 leader_election.cc:290] T fe902f1b58924be1ac3a5585413f7789 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:24.492601 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685cba2f73b412382a69dab2c7f50c1" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:24.492615 27183 leader_election.cc:343] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:24.493077 27184 leader_election.cc:343] T c6d67cd0c0304c77ba571b414878dad2 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:24.493247 28245 raft_consensus.cc:491] T dbfdab4332234b2a93fd3c901a9cb645 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:24.499073 28500 raft_consensus.cc:491] T 8605bc329627438998812ed028327f53 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:24.499488 28500 raft_consensus.cc:513] T 8605bc329627438998812ed028327f53 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.501493 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 0060e6c4699a4bff8faddefb4ee50add
I20250903 20:05:24.501747 28500 leader_election.cc:290] T 8605bc329627438998812ed028327f53 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:24.502426 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_59c192b7b71c41f4b9a349690a5e4ea3" } modify_external_catalogs: true
I20250903 20:05:24.502678 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_59c192b7b71c41f4b9a349690a5e4ea3" } modify_external_catalogs: true
I20250903 20:05:24.485275 28514 raft_consensus.cc:491] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:24.503851 27063 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 912eee5b48ef467bbd6bbe34f8aa390c (table default.Table_f1f4f57d96f04f0caff06d9a642f5c46 [id=fdc7899c8b334d309b834ae1daa14fa0]). Expected version 4 got 0
I20250903 20:05:24.504885 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_a3ccbeb42cef4247a4f9b6b349af9371"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:24.505641 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3ccbeb42cef4247a4f9b6b349af9371 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:24.506023 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 18231d53e9db46059da678f3f8b67a52 consensus state has no leader
W20250903 20:05:24.507252 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 421dc8af40c94a89bc372e8779872a4e consensus state has no leader
I20250903 20:05:24.507615 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 849eb453d3e04b16bbb5df9017bfb2fd
I20250903 20:05:24.507807 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8605bc329627438998812ed028327f53" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:24.508684 27320 leader_election.cc:343] T 8605bc329627438998812ed028327f53 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:24.511673 27687 ts_tablet_manager.cc:1918] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:24.512622 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 97d261080b8b479ca12267b6aa2e2ab0
I20250903 20:05:24.514827 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_53bee4ebf7974f119a4a5b5bd223d51f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:24.515350 27684 tablet_replica.cc:331] T 639ecb9caa4a423899ab8eb35bb716d7 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
W20250903 20:05:24.515437 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53bee4ebf7974f119a4a5b5bd223d51f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:24.515695 27684 raft_consensus.cc:2241] T 639ecb9caa4a423899ab8eb35bb716d7 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:24.515909 27684 raft_consensus.cc:2270] T 639ecb9caa4a423899ab8eb35bb716d7 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:24.516086 27687 log.cc:1199] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/f71d3dd9af2543299296f21381d1d563
I20250903 20:05:24.516512 27687 ts_tablet_manager.cc:1939] T f71d3dd9af2543299296f21381d1d563 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:24.516839 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_5ad7dc5547cc4cbbb4570adf93f155c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:24.517006 27687 tablet_replica.cc:331] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:24.517176 27687 raft_consensus.cc:2241] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:24.517426 27684 ts_tablet_manager.cc:1905] T 639ecb9caa4a423899ab8eb35bb716d7 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:24.517455 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ad7dc5547cc4cbbb4570adf93f155c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:24.518787 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_6181f8539fd34670ad894eae5e3ac4d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:24.519340 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6181f8539fd34670ad894eae5e3ac4d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:24.520537 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_91e306fcee12497e98a9d2da6302ba2b" } new_table_name: "default.Table_423c3bfc0b8c46bc9f81d7a9013e1468" modify_external_catalogs: true
I20250903 20:05:24.521750 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_4bb1596fb7204c0894e7a5d694b16943" } new_table_name: "default.Table_e1e543b2037e4d4f81950fdfd79b6d85" modify_external_catalogs: true
I20250903 20:05:24.523324 27488 tablet_service.cc:1515] Processing DeleteTablet for tablet 97d261080b8b479ca12267b6aa2e2ab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:24.517333 27687 raft_consensus.cc:2270] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:24.524202 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e942cce7cfd5471dbbca333e31f4ba66" } new_table_name: "default.Table_aded1170b7314a75b0d6c9d5ce4bcc00" modify_external_catalogs: true
I20250903 20:05:24.524821 27687 ts_tablet_manager.cc:1905] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:24.525504 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_d4af3ffb74834cccbbbee39223164dab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:24.526113 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4af3ffb74834cccbbbee39223164dab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:24.532279 27693 tablet_bootstrap.cc:492] T fa05f54ebc324b40855fa3288760e9e9 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:24.533903 27693 ts_tablet_manager.cc:1397] T fa05f54ebc324b40855fa3288760e9e9 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.230s user 0.015s sys 0.029s
I20250903 20:05:24.536026 27063 catalog_manager.cc:5582] T 912eee5b48ef467bbd6bbe34f8aa390c P 51578fb7bceb4e119b3ab1e58c4f5528 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66). New cstate: current_term: 1 leader_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:24.536448 27063 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 912eee5b48ef467bbd6bbe34f8aa390c (table default.Table_f1f4f57d96f04f0caff06d9a642f5c46 [id=fdc7899c8b334d309b834ae1daa14fa0]). Expected version 4 got 0
I20250903 20:05:24.536942 27063 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 516830fc42e84af693b3af52caee30ee (table default.Table_4d303308b63c4ce09056e226d1a30498 [id=098bc9a9b0a24a4a9edb7859310ac9b3]). Expected version 3 got 0
I20250903 20:05:24.538409 27496 tablet_service.cc:1515] Processing DeleteTablet for tablet fa05f54ebc324b40855fa3288760e9e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:24.538481 28298 raft_consensus.cc:491] T 516830fc42e84af693b3af52caee30ee P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:24.538758 28298 raft_consensus.cc:513] T 516830fc42e84af693b3af52caee30ee P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.540601 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "516830fc42e84af693b3af52caee30ee" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:24.541108 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 4be9bdc390de4a74a396003c9c3c2b71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:24.541154 27320 leader_election.cc:343] T 516830fc42e84af693b3af52caee30ee P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:24.542933 27687 ts_tablet_manager.cc:1918] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:24.544656 27687 log.cc:1199] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/0bdcbaa0169c4ab7ba6ec038b50482fe
I20250903 20:05:24.545172 27687 ts_tablet_manager.cc:1939] T 0bdcbaa0169c4ab7ba6ec038b50482fe P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:24.546517 27687 tablet_replica.cc:331] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:24.547709 28512 raft_consensus.cc:491] T 7c398634d6304a12b35c0579fba4c813 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:24.549116 28512 raft_consensus.cc:513] T 7c398634d6304a12b35c0579fba4c813 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.550580 28512 leader_election.cc:290] T 7c398634d6304a12b35c0579fba4c813 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:24.544749 27693 raft_consensus.cc:357] T fa05f54ebc324b40855fa3288760e9e9 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.550683 27382 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c398634d6304a12b35c0579fba4c813" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:24.552243 27687 raft_consensus.cc:2241] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:24.552328 27382 raft_consensus.cc:2391] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
I20250903 20:05:24.552541 27687 raft_consensus.cc:2270] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:24.552597 27693 raft_consensus.cc:383] T fa05f54ebc324b40855fa3288760e9e9 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:24.553362 27693 raft_consensus.cc:738] T fa05f54ebc324b40855fa3288760e9e9 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:24.553597 28298 leader_election.cc:290] T 516830fc42e84af693b3af52caee30ee P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:24.554041 27687 ts_tablet_manager.cc:1905] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:24.553905 27693 consensus_queue.cc:260] T fa05f54ebc324b40855fa3288760e9e9 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.557646 27693 ts_tablet_manager.cc:1428] T fa05f54ebc324b40855fa3288760e9e9 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.023s user 0.002s sys 0.001s
I20250903 20:05:24.558458 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 15072aa10c5540a09f3ec4592e377d2a
W20250903 20:05:24.560200 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1453a8f6bd7b4b26b75d7fc08578a351 consensus state has no leader
W20250903 20:05:24.561362 27059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 30cc0d4ba8d444f1b98232f906ea0c7b consensus state has no leader
W20250903 20:05:24.562252 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b03708700ae94f27a9d4bf5634f9ab0e consensus state has no leader
I20250903 20:05:24.561935 27693 tablet_bootstrap.cc:492] T 8d2d64f103fc44d7b390559b92cbf4b4 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:24.564383 27687 ts_tablet_manager.cc:1918] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:24.567137 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 15072aa10c5540a09f3ec4592e377d2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:24.567608 27502 tablet_service.cc:1515] Processing DeleteTablet for tablet 7c398634d6304a12b35c0579fba4c813 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:24.567395 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet 849eb453d3e04b16bbb5df9017bfb2fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:24.567795 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet 0060e6c4699a4bff8faddefb4ee50add with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:24.564862 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 5ec4bd6da4754fd59494f2069bb901e3 (DEFAULT_TABLE table=default.Table_c1f1c15403024868bdc2079821d61fdc [id=1b87cee2db5846c2b47fcdf948543508]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.574208 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ec4bd6da4754fd59494f2069bb901e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.574673 27351 tablet_service.cc:1468] Processing CreateTablet for tablet 0cc6d68ca0fe45a492c77879c5d9e89c (DEFAULT_TABLE table=default.Table_585c269abe9b4aa784813fd7439fa4cc [id=1efb74befa004d8daf9cb2dcce59f324]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.574978 27351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cc6d68ca0fe45a492c77879c5d9e89c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.565563 27348 tablet_service.cc:1468] Processing CreateTablet for tablet 553443eda16c44fa84b1db22b1363808 (DEFAULT_TABLE table=default.Table_307490b0006e4ed493ee8b2e23b7877a [id=e7b6f9faf04d47e9955ffafdb333ca30]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.578701 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 553443eda16c44fa84b1db22b1363808. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.579150 27693 tablet_bootstrap.cc:654] T 8d2d64f103fc44d7b390559b92cbf4b4 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:24.566175 27362 tablet_service.cc:1468] Processing CreateTablet for tablet c9d09ecd8fe347d79283fc9a3056e844 (DEFAULT_TABLE table=default.Table_0c63411efaae4fdc900ae09cdd4eed96 [id=a8d5279615374b09860a41942f9fea3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.566787 27356 tablet_service.cc:1515] Processing DeleteTablet for tablet 639ecb9caa4a423899ab8eb35bb716d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:23 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:24.567237 27352 tablet_service.cc:1515] Processing DeleteTablet for tablet ac9b5d4f39db4f05a15316585040d142 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:24.567628 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet 8605bc329627438998812ed028327f53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:24.568006 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 885fff47fea14b84ac6b34a57b827b4b with delete_type TABLET_DATA_DELETED (replaced by tablet 5f27f525fdab407280aa23df1ae71b43) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:24.568507 27358 tablet_service.cc:1468] Processing CreateTablet for tablet 33c624619c17495da7e7f55fee0a49b2 (DEFAULT_TABLE table=default.Table_43e145c502a34104b1388d584d5c7355 [id=25185952ddcb4e6cab3194234abe8643]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.569141 27357 tablet_service.cc:1468] Processing CreateTablet for tablet 989545be137341ef8d742bb1119cc51f (DEFAULT_TABLE table=default.Table_b69c590ef46648439e74d958bf9ffdf3 [id=1caafc550c7d4f478cb019f7e01b2aee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.569813 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 9ac48450b5aa448f9f040e88dc2f98bd (DEFAULT_TABLE table=default.Table_95b24ad570fd42158c2e3dc68edf37a8 [id=082a0477652f48e5b16cc5e6aac15e50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.570688 27350 tablet_service.cc:1468] Processing CreateTablet for tablet fa544dfdfff04a1e97676e562aeb0f40 (DEFAULT_TABLE table=default.Table_3f1f99f4ab084617b3a72447d6e84209 [id=a30b474b4c8b494db74d62e576e509a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.583217 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa544dfdfff04a1e97676e562aeb0f40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.585398 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 989545be137341ef8d742bb1119cc51f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.585610 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9d09ecd8fe347d79283fc9a3056e844. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.585965 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33c624619c17495da7e7f55fee0a49b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.586129 27687 log.cc:1199] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/4be9bdc390de4a74a396003c9c3c2b71
I20250903 20:05:24.586262 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ac48450b5aa448f9f040e88dc2f98bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.586604 27687 ts_tablet_manager.cc:1939] T 4be9bdc390de4a74a396003c9c3c2b71 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:24.587263 27687 tablet_replica.cc:331] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:24.587461 27687 raft_consensus.cc:2241] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:24.587695 27687 pending_rounds.cc:70] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528: Trying to abort 1 pending ops.
I20250903 20:05:24.587735 27687 pending_rounds.cc:77] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384968545046528 op_type: NO_OP noop_request { }
I20250903 20:05:24.587829 27687 raft_consensus.cc:2887] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:24.587903 27687 raft_consensus.cc:2270] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:24.588006 27349 tablet_service.cc:1468] Processing CreateTablet for tablet cc71346bf84f4a708bf17b5aba174582 (DEFAULT_TABLE table=default.Table_5d1a53d4800a4927b3de164630cc0b7a [id=bb6dc54b09a24dd3aec26340d6d3ea14]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.588351 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc71346bf84f4a708bf17b5aba174582. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.590530 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 0ffb568443854102888cb308a269ede2 (DEFAULT_TABLE table=default.Table_32ee815023e84a3dadbc1ec42d582b42 [id=828c85a9cc3f4239bad6368481ee1332]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.590816 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ffb568443854102888cb308a269ede2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.591073 27687 ts_tablet_manager.cc:1905] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:24.594563 27352 tablet_service.cc:1468] Processing CreateTablet for tablet 435c002dffd24aeaa6221ab00ee5b24b (DEFAULT_TABLE table=default.Table_392e604d6bdd46bfbe7e3b6d21529019 [id=23d521bab40643c78b35149ddab89c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.594864 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 435c002dffd24aeaa6221ab00ee5b24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.598477 27347 tablet_service.cc:1515] Processing DeleteTablet for tablet 849eb453d3e04b16bbb5df9017bfb2fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:24.598781 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 97d261080b8b479ca12267b6aa2e2ab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:24.598961 27354 tablet_service.cc:1515] Processing DeleteTablet for tablet 15072aa10c5540a09f3ec4592e377d2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:24.599337 27356 tablet_service.cc:1468] Processing CreateTablet for tablet 908e748d08f74ed2a3071312c95ec07c (DEFAULT_TABLE table=default.Table_e58baad01a28404886961ac38943a233 [id=0c3f0f04e2064efea6f5e9efb1bfc0aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.599647 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 908e748d08f74ed2a3071312c95ec07c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.605784 27351 tablet_service.cc:1515] Processing DeleteTablet for tablet 0060e6c4699a4bff8faddefb4ee50add with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:24.630812 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 0060e6c4699a4bff8faddefb4ee50add (DEFAULT_TABLE table=default.Table_794a9c24e42f44bb9d18feee62639661 [id=836b690a27db4f40895f775da5d3a6fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.631511 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0060e6c4699a4bff8faddefb4ee50add. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.639142 27687 ts_tablet_manager.cc:1918] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:24.639901 27687 log.cc:1199] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/de8ac9f984be46ec8f021928de12a090
I20250903 20:05:24.642833 27687 ts_tablet_manager.cc:1939] T de8ac9f984be46ec8f021928de12a090 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:24.648757 27693 tablet_bootstrap.cc:492] T 8d2d64f103fc44d7b390559b92cbf4b4 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:24.648926 27693 ts_tablet_manager.cc:1397] T 8d2d64f103fc44d7b390559b92cbf4b4 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.088s user 0.000s sys 0.007s
W20250903 20:05:24.649842 27184 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 91675.7us
I20250903 20:05:24.649848 27693 raft_consensus.cc:357] T 8d2d64f103fc44d7b390559b92cbf4b4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.650033 27693 raft_consensus.cc:383] T 8d2d64f103fc44d7b390559b92cbf4b4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:24.650085 27693 raft_consensus.cc:738] T 8d2d64f103fc44d7b390559b92cbf4b4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
W20250903 20:05:24.650110 27184 leader_election.cc:343] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:24.650341 27693 consensus_queue.cc:260] T 8d2d64f103fc44d7b390559b92cbf4b4 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.650863 27693 ts_tablet_manager.cc:1428] T 8d2d64f103fc44d7b390559b92cbf4b4 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250903 20:05:24.651387 27693 tablet_bootstrap.cc:492] T c6d67cd0c0304c77ba571b414878dad2 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:24.651677 27216 tablet_service.cc:1468] Processing CreateTablet for tablet fa544dfdfff04a1e97676e562aeb0f40 (DEFAULT_TABLE table=default.Table_3f1f99f4ab084617b3a72447d6e84209 [id=a30b474b4c8b494db74d62e576e509a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.652132 27216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa544dfdfff04a1e97676e562aeb0f40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.652588 27223 tablet_service.cc:1468] Processing CreateTablet for tablet c9d09ecd8fe347d79283fc9a3056e844 (DEFAULT_TABLE table=default.Table_0c63411efaae4fdc900ae09cdd4eed96 [id=a8d5279615374b09860a41942f9fea3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.652952 27223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9d09ecd8fe347d79283fc9a3056e844. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.653285 27687 tablet_replica.cc:331] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:24.654475 27226 tablet_service.cc:1468] Processing CreateTablet for tablet 0cc6d68ca0fe45a492c77879c5d9e89c (DEFAULT_TABLE table=default.Table_585c269abe9b4aa784813fd7439fa4cc [id=1efb74befa004d8daf9cb2dcce59f324]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.654814 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cc6d68ca0fe45a492c77879c5d9e89c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.654606 27687 raft_consensus.cc:2241] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:24.656546 27687 pending_rounds.cc:70] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528: Trying to abort 1 pending ops.
I20250903 20:05:24.656766 27687 pending_rounds.cc:77] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384968408555520 op_type: NO_OP noop_request { }
I20250903 20:05:24.657358 27693 tablet_bootstrap.cc:654] T c6d67cd0c0304c77ba571b414878dad2 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:24.485427 28494 raft_consensus.cc:491] T 45d9b6ff520348628d19671127b9426c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:24.657408 27687 raft_consensus.cc:2887] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:24.658286 27687 raft_consensus.cc:2270] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:24.660399 27053 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 18231d53e9db46059da678f3f8b67a52 (table default.Table_5c8743c0e4e444fb8eac65cde6000a06 [id=fe24472bb8e74884b9bf90ccdefd6d89], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 18231d53e9db46059da678f3f8b67a52 consensus state has no leader
I20250903 20:05:24.661700 27224 tablet_service.cc:1468] Processing CreateTablet for tablet 5ec4bd6da4754fd59494f2069bb901e3 (DEFAULT_TABLE table=default.Table_c1f1c15403024868bdc2079821d61fdc [id=1b87cee2db5846c2b47fcdf948543508]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.663654 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet 639ecb9caa4a423899ab8eb35bb716d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:23 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:24.664049 27687 ts_tablet_manager.cc:1905] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:24.666080 27246 raft_consensus.cc:2466] T 7c398634d6304a12b35c0579fba4c813 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
W20250903 20:05:24.666323 28519 log.cc:927] Time spent T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528: Append to log took a long time: real 0.140s user 0.000s sys 0.000s
I20250903 20:05:24.667425 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet b6b4713936134681aaef1732d46c6ab2 (table default.Table_ac1cd4ca0afa4c3494fa9953c49c838e [id=d0a75545b91c48bba47fb581bb0d1a96]) successfully deleted
W20250903 20:05:24.667846 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 6820c9a583af4bf68f9e63ae06e6ec64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6820c9a583af4bf68f9e63ae06e6ec64 already in progress: creating tablet
W20250903 20:05:24.668180 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet f36938c69f6a497f93e9458bdc9c40a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f36938c69f6a497f93e9458bdc9c40a0 already in progress: creating tablet
W20250903 20:05:24.668581 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 02b3e262dd7841dfbb0520377eb94743 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02b3e262dd7841dfbb0520377eb94743 already in progress: creating tablet
W20250903 20:05:24.668852 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 46f49fd718be4a2996d781ab31268217 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46f49fd718be4a2996d781ab31268217 already in progress: creating tablet
I20250903 20:05:24.669107 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet f71d3dd9af2543299296f21381d1d563 (table default.Table_a0582ccb94a64603a5fbdd0e531a9edf [id=6f7fece7a8d54d4786f1bab10da9129c]) successfully deleted
I20250903 20:05:24.669386 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 0bdcbaa0169c4ab7ba6ec038b50482fe (table default.Table_05d58d894486468a8a15aaa4b4c0cabb [id=e69321e60a3a4e71a463a0011c0c6d65]) successfully deleted
I20250903 20:05:24.669823 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 4be9bdc390de4a74a396003c9c3c2b71 (table default.Table_452fdb47678d4520828b9a3db6ddb237 [id=d945a830c86244bca66cb2652e64966d]) successfully deleted
W20250903 20:05:24.670161 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet ebc222df5d1445fea88939e1152142b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ebc222df5d1445fea88939e1152142b9 already in progress: creating tablet
W20250903 20:05:24.670528 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet ed7c564138cc43e5a3bb4e92fb5060b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed7c564138cc43e5a3bb4e92fb5060b6 already in progress: creating tablet
I20250903 20:05:24.671787 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet de8ac9f984be46ec8f021928de12a090 (table default.Table_eaff993b7bb7437580aae238faa74b3e [id=2650c8259ef4489695c56b6a2ba898d3]) successfully deleted
W20250903 20:05:24.672785 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 548b50819f694fa3ad508da9f442a99c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 548b50819f694fa3ad508da9f442a99c already in progress: creating tablet
I20250903 20:05:24.675488 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 989545be137341ef8d742bb1119cc51f (DEFAULT_TABLE table=default.Table_b69c590ef46648439e74d958bf9ffdf3 [id=1caafc550c7d4f478cb019f7e01b2aee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.675774 28514 raft_consensus.cc:513] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.676236 27684 ts_tablet_manager.cc:1918] T 639ecb9caa4a423899ab8eb35bb716d7 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:24.676343 27684 log.cc:1199] T 639ecb9caa4a423899ab8eb35bb716d7 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/639ecb9caa4a423899ab8eb35bb716d7
I20250903 20:05:24.676558 28514 leader_election.cc:290] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:24.677098 27382 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8605bc329627438998812ed028327f53" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:24.677196 27684 ts_tablet_manager.cc:1939] T 639ecb9caa4a423899ab8eb35bb716d7 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:24.677311 27382 raft_consensus.cc:2466] T 8605bc329627438998812ed028327f53 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:24.677947 27692 tablet_bootstrap.cc:492] T 885fff47fea14b84ac6b34a57b827b4b P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:24.679109 27692 ts_tablet_manager.cc:1397] T 885fff47fea14b84ac6b34a57b827b4b P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.271s user 0.021s sys 0.020s
I20250903 20:05:24.659339 27183 leader_election.cc:304] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:24.682981 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c398634d6304a12b35c0579fba4c813" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:24.682709 27692 raft_consensus.cc:357] T 885fff47fea14b84ac6b34a57b827b4b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.683254 27692 raft_consensus.cc:383] T 885fff47fea14b84ac6b34a57b827b4b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:24.683533 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 9ac48450b5aa448f9f040e88dc2f98bd (DEFAULT_TABLE table=default.Table_95b24ad570fd42158c2e3dc68edf37a8 [id=082a0477652f48e5b16cc5e6aac15e50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.683550 27692 raft_consensus.cc:738] T 885fff47fea14b84ac6b34a57b827b4b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:24.683926 27227 tablet_service.cc:1468] Processing CreateTablet for tablet 553443eda16c44fa84b1db22b1363808 (DEFAULT_TABLE table=default.Table_307490b0006e4ed493ee8b2e23b7877a [id=e7b6f9faf04d47e9955ffafdb333ca30]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.657444 28245 raft_consensus.cc:513] T dbfdab4332234b2a93fd3c901a9cb645 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.684818 27693 tablet_bootstrap.cc:492] T c6d67cd0c0304c77ba571b414878dad2 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:24.685082 27693 ts_tablet_manager.cc:1397] T c6d67cd0c0304c77ba571b414878dad2 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.006s
I20250903 20:05:24.685511 27244 raft_consensus.cc:2466] T 912eee5b48ef467bbd6bbe34f8aa390c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:24.683372 28517 raft_consensus.cc:491] T 4157633525034a019fd6a80185892cec P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:24.686457 27692 consensus_queue.cc:260] T 885fff47fea14b84ac6b34a57b827b4b P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.687134 27221 tablet_service.cc:1468] Processing CreateTablet for tablet 0060e6c4699a4bff8faddefb4ee50add (DEFAULT_TABLE table=default.Table_794a9c24e42f44bb9d18feee62639661 [id=836b690a27db4f40895f775da5d3a6fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.688330 27222 tablet_service.cc:1468] Processing CreateTablet for tablet cc71346bf84f4a708bf17b5aba174582 (DEFAULT_TABLE table=default.Table_5d1a53d4800a4927b3de164630cc0b7a [id=bb6dc54b09a24dd3aec26340d6d3ea14]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.687907 27693 raft_consensus.cc:357] T c6d67cd0c0304c77ba571b414878dad2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.689199 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ec4bd6da4754fd59494f2069bb901e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.688903 27693 raft_consensus.cc:383] T c6d67cd0c0304c77ba571b414878dad2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:24.689307 27693 raft_consensus.cc:738] T c6d67cd0c0304c77ba571b414878dad2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:24.690325 27692 ts_tablet_manager.cc:1428] T 885fff47fea14b84ac6b34a57b827b4b P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20250903 20:05:24.690928 28494 raft_consensus.cc:513] T 45d9b6ff520348628d19671127b9426c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.691533 28494 leader_election.cc:290] T 45d9b6ff520348628d19671127b9426c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:24.690820 27693 consensus_queue.cc:260] T c6d67cd0c0304c77ba571b414878dad2 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.691905 27184 leader_election.cc:304] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:24.692802 28514 raft_consensus.cc:2747] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:24.693558 27692 tablet_bootstrap.cc:492] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
W20250903 20:05:24.698745 27183 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 188854us
W20250903 20:05:24.699018 27183 leader_election.cc:343] T c6d67cd0c0304c77ba571b414878dad2 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:24.699133 27183 leader_election.cc:304] T c6d67cd0c0304c77ba571b414878dad2 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:24.700017 27685 ts_tablet_manager.cc:1918] T 46e3a65118b744298b5536f88c22f1db P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 20:05:24.700306 28514 raft_consensus.cc:2747] T c6d67cd0c0304c77ba571b414878dad2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:24.700278 27382 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45d9b6ff520348628d19671127b9426c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:24.701628 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbfdab4332234b2a93fd3c901a9cb645" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:24.702785 27183 leader_election.cc:343] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:24.702898 27183 leader_election.cc:304] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:24.703603 28514 raft_consensus.cc:2747] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:24.706116 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 5c42d84a6a5649208b0539a10da21aec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c42d84a6a5649208b0539a10da21aec already in progress: creating tablet
W20250903 20:05:24.706423 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet aa31f2ddb9f84a8093aa1c6f64b278c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa31f2ddb9f84a8093aa1c6f64b278c9 already in progress: creating tablet
I20250903 20:05:24.706570 27248 raft_consensus.cc:2391] T 7c398634d6304a12b35c0579fba4c813 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
I20250903 20:05:24.706935 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f0cba70240f4f38a4fa5ccbcd99b3b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3af0c66a8c234711a43816f248e5be03) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:24.707461 28245 leader_election.cc:290] T dbfdab4332234b2a93fd3c901a9cb645 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:24.707476 27462 leader_election.cc:304] T 7c398634d6304a12b35c0579fba4c813 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:24.673211 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7179a3a012834e34b703156b9c834722" } modify_external_catalogs: true
I20250903 20:05:24.702344 27060 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 6238740ce1894e899be9a40321f0ab73 (table default.Table_0fef1d019eff4323b7ba56368614aadc [id=53074cd97dec40c7b0e1444183d6707a]). Expected version 2 got 0
I20250903 20:05:24.707716 27693 ts_tablet_manager.cc:1428] T c6d67cd0c0304c77ba571b414878dad2 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.022s user 0.002s sys 0.001s
I20250903 20:05:24.707787 28534 tablet.cc:1722] T 6238740ce1894e899be9a40321f0ab73 P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:24.709707 28245 raft_consensus.cc:491] T 0ad64c699b294679b6fec37b5ad23528 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:24.710060 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8605bc329627438998812ed028327f53" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:24.710633 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45d9b6ff520348628d19671127b9426c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:24.710932 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbfdab4332234b2a93fd3c901a9cb645" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:24.711176 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7179a3a012834e34b703156b9c834722" } modify_external_catalogs: true
I20250903 20:05:24.709465 27693 tablet_bootstrap.cc:492] T dbfdab4332234b2a93fd3c901a9cb645 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:24.707862 28512 raft_consensus.cc:3058] T 7c398634d6304a12b35c0579fba4c813 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:24.715684 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_cb71c02ad33b47768df2ee87bc601760"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:24.715952 27692 tablet_bootstrap.cc:654] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:24.716313 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb71c02ad33b47768df2ee87bc601760 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:24.717152 28512 raft_consensus.cc:2747] T 7c398634d6304a12b35c0579fba4c813 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:24.717653 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b8435bf217c34217bbea1bad9cb30edb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:24.718410 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b57aafd78ceb4611b5b6f4dd3b2bad30" } new_table_name: "default.Table_e1a05f991b98479ca07e53c4c3c85254" modify_external_catalogs: true
I20250903 20:05:24.719770 27059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_77c890250aaf48f4b658f0f126941bc7" } modify_external_catalogs: true
I20250903 20:05:24.719941 27059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_77c890250aaf48f4b658f0f126941bc7" } modify_external_catalogs: true
I20250903 20:05:24.720118 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_c4b81b1196c847c09dcf2d9281c75b26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:24.720495 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_9e9e434350e54dc4a9397cee96d16383"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:24.720734 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4b81b1196c847c09dcf2d9281c75b26 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:24.720991 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e9e434350e54dc4a9397cee96d16383 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:24.722011 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_18a6310ff569456f91088bbafcfd50f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:24.722477 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18a6310ff569456f91088bbafcfd50f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:24.723563 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_56af93fd7df844a99722a1b88e0c54c6" } modify_external_catalogs: true
I20250903 20:05:24.723702 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_56af93fd7df844a99722a1b88e0c54c6" } modify_external_catalogs: true
I20250903 20:05:24.724594 27687 ts_tablet_manager.cc:1918] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:24.724704 27687 log.cc:1199] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/639ecb9caa4a423899ab8eb35bb716d7
I20250903 20:05:24.725057 27687 ts_tablet_manager.cc:1939] T 639ecb9caa4a423899ab8eb35bb716d7 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:24.726634 27693 tablet_bootstrap.cc:654] T dbfdab4332234b2a93fd3c901a9cb645 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:24.727087 27684 tablet_replica.cc:331] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:24.727304 27684 raft_consensus.cc:2241] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:24.727593 27684 pending_rounds.cc:70] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f: Trying to abort 1 pending ops.
I20250903 20:05:24.727644 27684 pending_rounds.cc:77] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384969888124928 op_type: NO_OP noop_request { }
I20250903 20:05:24.727728 27684 raft_consensus.cc:2887] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:24.727808 27684 raft_consensus.cc:2270] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:24.728065 27687 tablet_replica.cc:331] T ac9b5d4f39db4f05a15316585040d142 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:24.728259 27687 raft_consensus.cc:2241] T ac9b5d4f39db4f05a15316585040d142 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:24.728400 27687 raft_consensus.cc:2270] T ac9b5d4f39db4f05a15316585040d142 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:24.728909 27687 ts_tablet_manager.cc:1905] T ac9b5d4f39db4f05a15316585040d142 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:24.729242 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8435bf217c34217bbea1bad9cb30edb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:24.729754 27684 ts_tablet_manager.cc:1905] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:24.731775 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 7c398634d6304a12b35c0579fba4c813 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:24.732313 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet cb4d99139516405bb65c7480ddf259f1 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d4b8d4bf29e408cbdf9557c8b0d8a4b) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:24.732734 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet fa05f54ebc324b40855fa3288760e9e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:24.733134 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d2d64f103fc44d7b390559b92cbf4b4 with delete_type TABLET_DATA_DELETED (replaced by tablet a38f65e992884635aae9942681c0dc4f) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:24.733688 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet c6d67cd0c0304c77ba571b414878dad2 with delete_type TABLET_DATA_DELETED (replaced by tablet 886b8783aa9d4618aad498d4a1115843) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:24.734318 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet fe902f1b58924be1ac3a5585413f7789 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:24.735714 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92c0365f08ec47b581b1bfed736524b2 consensus state has no leader
I20250903 20:05:24.737637 27687 ts_tablet_manager.cc:1918] T ac9b5d4f39db4f05a15316585040d142 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:24.737746 27687 log.cc:1199] T ac9b5d4f39db4f05a15316585040d142 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/ac9b5d4f39db4f05a15316585040d142
I20250903 20:05:24.738046 27687 ts_tablet_manager.cc:1939] T ac9b5d4f39db4f05a15316585040d142 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:24.741612 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 639ecb9caa4a423899ab8eb35bb716d7 (table default.Table_3da0f03ae8114e348baf44301196b2e5 [id=ad8b3a9bedef4c188b89640ab5e2473a]) successfully deleted
I20250903 20:05:24.742620 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet 885fff47fea14b84ac6b34a57b827b4b with delete_type TABLET_DATA_DELETED (replaced by tablet 5f27f525fdab407280aa23df1ae71b43) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:24.742949 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 639ecb9caa4a423899ab8eb35bb716d7 (table default.Table_3da0f03ae8114e348baf44301196b2e5 [id=ad8b3a9bedef4c188b89640ab5e2473a]) successfully deleted
I20250903 20:05:24.750380 27687 tablet_replica.cc:331] T 8605bc329627438998812ed028327f53 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:24.750607 27687 raft_consensus.cc:2241] T 8605bc329627438998812ed028327f53 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:24.750778 27687 raft_consensus.cc:2270] T 8605bc329627438998812ed028327f53 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:24.750905 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet ac9b5d4f39db4f05a15316585040d142 (table default.Table_f6b26b9f6ef340989d2250b8d83647c2 [id=74b402d4851f48c2bfd495e82917bb38]) successfully deleted
I20250903 20:05:24.751320 27687 ts_tablet_manager.cc:1905] T 8605bc329627438998812ed028327f53 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:24.751412 27693 tablet_bootstrap.cc:492] T dbfdab4332234b2a93fd3c901a9cb645 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:24.751642 27693 ts_tablet_manager.cc:1397] T dbfdab4332234b2a93fd3c901a9cb645 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.043s user 0.000s sys 0.005s
I20250903 20:05:24.752367 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5ad7dc5547cc4cbbb4570adf93f155c9" } new_table_name: "default.Table_5d167e8474a043f09801a985626f69e4" modify_external_catalogs: true
I20250903 20:05:24.752684 27693 raft_consensus.cc:357] T dbfdab4332234b2a93fd3c901a9cb645 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.752852 27693 raft_consensus.cc:383] T dbfdab4332234b2a93fd3c901a9cb645 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:24.752914 27693 raft_consensus.cc:738] T dbfdab4332234b2a93fd3c901a9cb645 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:24.753135 27693 consensus_queue.cc:260] T dbfdab4332234b2a93fd3c901a9cb645 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.754184 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_9b23868e884f40f0b6b2ffdb5e0c99d2" } modify_external_catalogs: true
I20250903 20:05:24.754436 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_9b23868e884f40f0b6b2ffdb5e0c99d2" } modify_external_catalogs: true
I20250903 20:05:24.754487 27693 ts_tablet_manager.cc:1428] T dbfdab4332234b2a93fd3c901a9cb645 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20250903 20:05:24.755935 27692 tablet_bootstrap.cc:492] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:24.756153 27692 ts_tablet_manager.cc:1397] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.065s user 0.006s sys 0.005s
I20250903 20:05:24.757009 27692 raft_consensus.cc:357] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.757256 27692 raft_consensus.cc:383] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:24.757349 27692 raft_consensus.cc:738] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:24.757648 27692 consensus_queue.cc:260] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.758175 27692 ts_tablet_manager.cc:1428] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:24.763094 27574 heartbeater.cc:344] Connected to a master server at 127.26.65.125:44059
I20250903 20:05:24.764873 27692 tablet_bootstrap.cc:492] T 516830fc42e84af693b3af52caee30ee P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:24.766691 27692 tablet_bootstrap.cc:654] T 516830fc42e84af693b3af52caee30ee P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:24.772713 27687 ts_tablet_manager.cc:1918] T 8605bc329627438998812ed028327f53 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:24.774202 27687 log.cc:1199] T 8605bc329627438998812ed028327f53 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/8605bc329627438998812ed028327f53
I20250903 20:05:24.774508 27693 tablet_bootstrap.cc:492] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:24.774866 27687 ts_tablet_manager.cc:1939] T 8605bc329627438998812ed028327f53 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:24.775799 27687 tablet_replica.cc:331] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:24.776401 27693 tablet_bootstrap.cc:654] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:24.780884 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ec8cea2d64b642689d0bc11295745ef0 has no consensus state
I20250903 20:05:24.782470 27684 ts_tablet_manager.cc:1918] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:24.783951 27687 raft_consensus.cc:2241] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:24.784219 27687 pending_rounds.cc:70] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528: Trying to abort 1 pending ops.
I20250903 20:05:24.784276 27687 pending_rounds.cc:77] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384970074533888 op_type: NO_OP noop_request { }
I20250903 20:05:24.784381 27687 raft_consensus.cc:2887] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:24.784469 27687 raft_consensus.cc:2270] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:24.786952 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_be46ef5a0c524f32b001a4fd218d497f" } new_table_name: "default.Table_7ced6983993143f48ffb33fe2b19e03c" modify_external_catalogs: true
I20250903 20:05:24.787460 27687 ts_tablet_manager.cc:1905] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:24.788787 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_312cd54d2d6f4ae5bbcd5fb11ab7700e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:24.789109 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_8dd7940c3a2c406c96060423df335b7d" } new_table_name: "default.Table_b13e911191fc47e1a83d725043ba1f30" modify_external_catalogs: true
W20250903 20:05:24.789443 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_312cd54d2d6f4ae5bbcd5fb11ab7700e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:24.791006 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_63be5f347200408ba649d6672260ac6e" } new_table_name: "default.Table_2e9e07bea789491685a849ec5dea7d73" modify_external_catalogs: true
I20250903 20:05:24.793133 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ce7f1e2809ba4ae097811cd75676be66" } modify_external_catalogs: true
I20250903 20:05:24.793532 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ce7f1e2809ba4ae097811cd75676be66" } modify_external_catalogs: true
I20250903 20:05:24.794126 27693 tablet_bootstrap.cc:492] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:24.794369 27693 ts_tablet_manager.cc:1397] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.007s
I20250903 20:05:24.795189 27693 raft_consensus.cc:357] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.795418 27693 raft_consensus.cc:383] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:24.795516 27693 raft_consensus.cc:738] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:24.795778 27693 consensus_queue.cc:260] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.796433 27693 ts_tablet_manager.cc:1428] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:24.796749 27693 tablet_bootstrap.cc:492] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:24.797170 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 989545be137341ef8d742bb1119cc51f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.798465 27693 tablet_bootstrap.cc:654] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:24.786953 27692 tablet_bootstrap.cc:492] T 516830fc42e84af693b3af52caee30ee P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:24.799907 27684 log.cc:1199] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/0bdcbaa0169c4ab7ba6ec038b50482fe
I20250903 20:05:24.799975 27692 ts_tablet_manager.cc:1397] T 516830fc42e84af693b3af52caee30ee P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.035s user 0.007s sys 0.003s
I20250903 20:05:24.800379 27684 ts_tablet_manager.cc:1939] T 0bdcbaa0169c4ab7ba6ec038b50482fe P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:24.800812 27692 raft_consensus.cc:357] T 516830fc42e84af693b3af52caee30ee P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.800973 27692 raft_consensus.cc:383] T 516830fc42e84af693b3af52caee30ee P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:24.801033 27692 raft_consensus.cc:738] T 516830fc42e84af693b3af52caee30ee P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:24.801236 27692 consensus_queue.cc:260] T 516830fc42e84af693b3af52caee30ee P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.801224 28517 raft_consensus.cc:513] T 4157633525034a019fd6a80185892cec P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.801923 28517 leader_election.cc:290] T 4157633525034a019fd6a80185892cec P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:24.802080 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 33c624619c17495da7e7f55fee0a49b2 (DEFAULT_TABLE table=default.Table_43e145c502a34104b1388d584d5c7355 [id=25185952ddcb4e6cab3194234abe8643]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.802497 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33c624619c17495da7e7f55fee0a49b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.803164 27692 ts_tablet_manager.cc:1428] T 516830fc42e84af693b3af52caee30ee P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:24.803442 27692 tablet_bootstrap.cc:492] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:24.805330 27687 ts_tablet_manager.cc:1918] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:24.805442 27687 log.cc:1199] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/885fff47fea14b84ac6b34a57b827b4b
I20250903 20:05:24.805780 27687 ts_tablet_manager.cc:1939] T 885fff47fea14b84ac6b34a57b827b4b P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:24.809273 27687 tablet_replica.cc:331] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:24.809473 27687 raft_consensus.cc:2241] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:24.809619 27687 raft_consensus.cc:2270] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:24.810138 27687 ts_tablet_manager.cc:1905] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:24.813761 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0060e6c4699a4bff8faddefb4ee50add. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.814659 27228 tablet_service.cc:1468] Processing CreateTablet for tablet 0ffb568443854102888cb308a269ede2 (DEFAULT_TABLE table=default.Table_32ee815023e84a3dadbc1ec42d582b42 [id=828c85a9cc3f4239bad6368481ee1332]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.816772 28494 raft_consensus.cc:2802] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:24.816879 28494 raft_consensus.cc:491] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:24.816944 28494 raft_consensus.cc:3058] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:24.709831 28245 raft_consensus.cc:513] T 0ad64c699b294679b6fec37b5ad23528 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:24.817601 27184 leader_election.cc:343] T 45d9b6ff520348628d19671127b9426c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:24.819600 27684 tablet_replica.cc:331] T fa05f54ebc324b40855fa3288760e9e9 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:24.819914 27684 raft_consensus.cc:2241] T fa05f54ebc324b40855fa3288760e9e9 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:24.821928 27220 tablet_service.cc:1468] Processing CreateTablet for tablet 435c002dffd24aeaa6221ab00ee5b24b (DEFAULT_TABLE table=default.Table_392e604d6bdd46bfbe7e3b6d21529019 [id=23d521bab40643c78b35149ddab89c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.820593 27692 tablet_bootstrap.cc:654] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:24.823365 27684 raft_consensus.cc:2270] T fa05f54ebc324b40855fa3288760e9e9 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:24.824868 27693 tablet_bootstrap.cc:492] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:24.825007 27693 ts_tablet_manager.cc:1397] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.004s
I20250903 20:05:24.825795 27693 raft_consensus.cc:357] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.825966 27693 raft_consensus.cc:383] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:24.826081 27693 raft_consensus.cc:738] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:24.826349 27693 consensus_queue.cc:260] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.827917 27693 ts_tablet_manager.cc:1428] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:24.828651 27693 tablet_bootstrap.cc:492] T 283a3afed7414a958b2dc4176f00bb01 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:24.829288 28494 raft_consensus.cc:513] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.829892 28494 leader_election.cc:290] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
W20250903 20:05:24.830340 27184 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 116962us
I20250903 20:05:24.831593 27693 tablet_bootstrap.cc:654] T 283a3afed7414a958b2dc4176f00bb01 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:24.831825 27184 leader_election.cc:343] T dbfdab4332234b2a93fd3c901a9cb645 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:24.832351 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc71346bf84f4a708bf17b5aba174582. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.832298 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4157633525034a019fd6a80185892cec" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:24.823233 27687 ts_tablet_manager.cc:1918] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:24.837218 27687 log.cc:1199] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/6f0cba70240f4f38a4fa5ccbcd99b3b1
I20250903 20:05:24.837548 27687 ts_tablet_manager.cc:1939] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:24.838956 28245 leader_election.cc:290] T 0ad64c699b294679b6fec37b5ad23528 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:24.840981 27687 tablet_replica.cc:331] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:24.841192 27687 raft_consensus.cc:2241] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:24.841372 27687 raft_consensus.cc:2270] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:24.841902 27687 ts_tablet_manager.cc:1905] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:24.848934 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0ad64c699b294679b6fec37b5ad23528" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:24.852900 27693 tablet_bootstrap.cc:492] T 283a3afed7414a958b2dc4176f00bb01 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:24.853693 27692 tablet_bootstrap.cc:492] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:24.853891 27692 ts_tablet_manager.cc:1397] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.008s
I20250903 20:05:24.854598 27685 log.cc:1199] T 46e3a65118b744298b5536f88c22f1db P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/46e3a65118b744298b5536f88c22f1db
I20250903 20:05:24.854835 27692 raft_consensus.cc:357] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.855125 27692 raft_consensus.cc:383] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:24.855228 27692 raft_consensus.cc:738] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:24.855492 27692 consensus_queue.cc:260] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.857329 27692 ts_tablet_manager.cc:1428] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250903 20:05:24.857642 27692 tablet_bootstrap.cc:492] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:24.858408 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4157633525034a019fd6a80185892cec" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:24.859017 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0ad64c699b294679b6fec37b5ad23528" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:24.859361 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8605bc329627438998812ed028327f53" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:24.859787 27692 tablet_bootstrap.cc:654] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:24.860002 27693 ts_tablet_manager.cc:1397] T 283a3afed7414a958b2dc4176f00bb01 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.000s
I20250903 20:05:24.860813 27693 raft_consensus.cc:357] T 283a3afed7414a958b2dc4176f00bb01 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.862457 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ac48450b5aa448f9f040e88dc2f98bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.863513 27693 raft_consensus.cc:383] T 283a3afed7414a958b2dc4176f00bb01 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:24.863648 27693 raft_consensus.cc:738] T 283a3afed7414a958b2dc4176f00bb01 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:24.863915 27693 consensus_queue.cc:260] T 283a3afed7414a958b2dc4176f00bb01 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.864588 27693 ts_tablet_manager.cc:1428] T 283a3afed7414a958b2dc4176f00bb01 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250903 20:05:24.864912 27693 tablet_bootstrap.cc:492] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:24.867136 28298 raft_consensus.cc:491] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:24.867280 28298 raft_consensus.cc:513] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.867883 28298 leader_election.cc:290] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:24.869056 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd6d6bb514134bdf8e3bc0aaad180732" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:24.869222 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 553443eda16c44fa84b1db22b1363808. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:24.869664 27320 leader_election.cc:343] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:24.870589 27228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ffb568443854102888cb308a269ede2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.871140 27693 tablet_bootstrap.cc:654] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:24.874393 27693 tablet_bootstrap.cc:492] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:24.875514 27692 tablet_bootstrap.cc:492] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:24.875710 27692 ts_tablet_manager.cc:1397] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20250903 20:05:24.876456 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 0bdcbaa0169c4ab7ba6ec038b50482fe (table default.Table_05d58d894486468a8a15aaa4b4c0cabb [id=e69321e60a3a4e71a463a0011c0c6d65]) successfully deleted
I20250903 20:05:24.876577 27692 raft_consensus.cc:357] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.876854 27692 raft_consensus.cc:383] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:24.876955 27692 raft_consensus.cc:738] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:24.877269 27692 consensus_queue.cc:260] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.881747 27692 ts_tablet_manager.cc:1428] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250903 20:05:24.882175 27692 tablet_bootstrap.cc:492] T 8605bc329627438998812ed028327f53 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:24.883783 27693 ts_tablet_manager.cc:1397] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.002s
I20250903 20:05:24.885757 27693 raft_consensus.cc:357] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.887243 27693 raft_consensus.cc:383] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:24.887393 27693 raft_consensus.cc:738] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:24.887681 27693 consensus_queue.cc:260] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.888883 27692 tablet_bootstrap.cc:654] T 8605bc329627438998812ed028327f53 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:24.889429 27184 leader_election.cc:343] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 8605bc329627438998812ed028327f53
W20250903 20:05:24.889696 27184 leader_election.cc:343] T 4157633525034a019fd6a80185892cec P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:24.889730 27693 ts_tablet_manager.cc:1428] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
W20250903 20:05:24.889873 27184 leader_election.cc:343] T 0ad64c699b294679b6fec37b5ad23528 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:24.890269 27693 tablet_bootstrap.cc:492] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:24.892103 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 908e748d08f74ed2a3071312c95ec07c (DEFAULT_TABLE table=default.Table_e58baad01a28404886961ac38943a233 [id=0c3f0f04e2064efea6f5e9efb1bfc0aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:24.891464 27433 heartbeater.cc:344] Connected to a master server at 127.26.65.125:44059
I20250903 20:05:24.893111 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 849eb453d3e04b16bbb5df9017bfb2fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:24.893847 27693 tablet_bootstrap.cc:654] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:24.896625 27685 ts_tablet_manager.cc:1939] T 46e3a65118b744298b5536f88c22f1db P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
W20250903 20:05:24.903111 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b32c82c3d8c4420cb6e3a95129550076 consensus state has no leader
I20250903 20:05:24.904078 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_f073b600fed2421f9a25aca9c22751e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:24.904729 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f073b600fed2421f9a25aca9c22751e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:24.905905 27693 tablet_bootstrap.cc:492] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
W20250903 20:05:24.903466 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6853839e73e343a0a54aa400a23db35c consensus state has no leader
I20250903 20:05:24.906188 27693 ts_tablet_manager.cc:1397] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20250903 20:05:24.906332 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_257827f9b2804d37aa0556a8affcf93e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:24.906908 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_257827f9b2804d37aa0556a8affcf93e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:24.907258 27693 raft_consensus.cc:357] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.907474 27693 raft_consensus.cc:383] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:24.907574 27693 raft_consensus.cc:738] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:24.907835 27693 consensus_queue.cc:260] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.908572 27693 ts_tablet_manager.cc:1428] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:24.908885 27693 tablet_bootstrap.cc:492] T 0ad64c699b294679b6fec37b5ad23528 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:24.910342 27693 tablet_bootstrap.cc:654] T 0ad64c699b294679b6fec37b5ad23528 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:24.913179 27183 leader_election.cc:343] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:24.913808 27213 tablet_service.cc:1515] Processing DeleteTablet for tablet 0060e6c4699a4bff8faddefb4ee50add with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:24.914027 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 435c002dffd24aeaa6221ab00ee5b24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:24.915050 27687 ts_tablet_manager.cc:1918] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:24.915535 27687 log.cc:1199] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/7c398634d6304a12b35c0579fba4c813
I20250903 20:05:24.915980 27687 ts_tablet_manager.cc:1939] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:24.916965 27687 tablet_replica.cc:331] T cb4d99139516405bb65c7480ddf259f1 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:24.917846 27687 raft_consensus.cc:2241] T cb4d99139516405bb65c7480ddf259f1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:24.918304 27687 raft_consensus.cc:2270] T cb4d99139516405bb65c7480ddf259f1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:24.919117 27687 ts_tablet_manager.cc:1905] T cb4d99139516405bb65c7480ddf259f1 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:24.922596 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 97d261080b8b479ca12267b6aa2e2ab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:24.923928 27693 tablet_bootstrap.cc:492] T 0ad64c699b294679b6fec37b5ad23528 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:24.924154 27693 ts_tablet_manager.cc:1397] T 0ad64c699b294679b6fec37b5ad23528 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.002s
I20250903 20:05:24.924979 27693 raft_consensus.cc:357] T 0ad64c699b294679b6fec37b5ad23528 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.925231 27693 raft_consensus.cc:383] T 0ad64c699b294679b6fec37b5ad23528 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:24.925338 27693 raft_consensus.cc:738] T 0ad64c699b294679b6fec37b5ad23528 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:24.925487 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 606) took 4056 ms. Trace:
I20250903 20:05:24.925693 27693 consensus_queue.cc:260] T 0ad64c699b294679b6fec37b5ad23528 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.926162 28512 raft_consensus.cc:491] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:24.926330 28512 raft_consensus.cc:513] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.926952 28512 leader_election.cc:290] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:24.928030 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe902f1b58924be1ac3a5585413f7789" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:24.928303 27380 raft_consensus.cc:2466] T fe902f1b58924be1ac3a5585413f7789 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:24.928771 27462 leader_election.cc:304] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:24.929180 28512 raft_consensus.cc:2802] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:24.929263 28512 raft_consensus.cc:491] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:24.929327 28512 raft_consensus.cc:3058] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:24.929556 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet dbfdab4332234b2a93fd3c901a9cb645 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:24.931981 28512 raft_consensus.cc:513] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.933012 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe902f1b58924be1ac3a5585413f7789" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:24.933197 27380 raft_consensus.cc:3058] T fe902f1b58924be1ac3a5585413f7789 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:24.933408 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 8605bc329627438998812ed028327f53 (table default.Table_8b0346420a89417b9ae2005be49e9f3a [id=b0b2e2d5798346708995a6b8dc028918]) successfully deleted
W20250903 20:05:24.933887 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 97d261080b8b479ca12267b6aa2e2ab0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97d261080b8b479ca12267b6aa2e2ab0 already in progress: creating tablet
I20250903 20:05:24.935894 27380 raft_consensus.cc:2466] T fe902f1b58924be1ac3a5585413f7789 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:24.936668 27462 leader_election.cc:304] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:24.936980 28396 raft_consensus.cc:2802] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:24.938056 27244 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe902f1b58924be1ac3a5585413f7789" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:24.938324 27244 raft_consensus.cc:3058] T fe902f1b58924be1ac3a5585413f7789 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:24.940975 28512 leader_election.cc:290] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:24.942135 27693 ts_tablet_manager.cc:1428] T 0ad64c699b294679b6fec37b5ad23528 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20250903 20:05:24.942514 27693 tablet_bootstrap.cc:492] T e78e6d212daf4589a6b4b65711e356cb P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:24.942653 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe902f1b58924be1ac3a5585413f7789" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:24.944360 27693 tablet_bootstrap.cc:654] T e78e6d212daf4589a6b4b65711e356cb P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:24.947949 27229 tablet_service.cc:1515] Processing DeleteTablet for tablet 15072aa10c5540a09f3ec4592e377d2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:24.948272 28298 raft_consensus.cc:491] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:24.948410 28298 raft_consensus.cc:513] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.949087 28298 leader_election.cc:290] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:24.949172 27685 rpcz_store.cc:276] 0903 20:05:20.842763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.842831 (+ 68us) service_pool.cc:225] Handling call
0903 20:05:24.118598 (+3275767us) ts_tablet_manager.cc:1145] Deleting tablet 46e3a65118b744298b5536f88c22f1db
0903 20:05:24.118609 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:24.251697 (+133088us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:24.680834 (+429137us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:24.899578 (+218744us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3274858}
I20250903 20:05:24.949904 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 607) took 4106 ms. Trace:
I20250903 20:05:24.950094 27685 rpcz_store.cc:276] 0903 20:05:20.843054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:20.843149 (+ 95us) service_pool.cc:225] Handling call
0903 20:05:24.949714 (+4106565us) ts_tablet_manager.cc:1145] Deleting tablet 0906165a50b047c8a776b30832149964
0903 20:05:24.949728 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:24.949856 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4100986}
I20250903 20:05:24.957012 27687 ts_tablet_manager.cc:1918] T cb4d99139516405bb65c7480ddf259f1 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:24.957170 27687 log.cc:1199] T cb4d99139516405bb65c7480ddf259f1 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/cb4d99139516405bb65c7480ddf259f1
I20250903 20:05:24.957489 27687 ts_tablet_manager.cc:1939] T cb4d99139516405bb65c7480ddf259f1 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:24.962805 27687 tablet_replica.cc:331] T fa05f54ebc324b40855fa3288760e9e9 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:24.963999 27693 tablet_bootstrap.cc:492] T e78e6d212daf4589a6b4b65711e356cb P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:24.964154 27693 ts_tablet_manager.cc:1397] T e78e6d212daf4589a6b4b65711e356cb P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.002s
I20250903 20:05:24.964980 27693 raft_consensus.cc:357] T e78e6d212daf4589a6b4b65711e356cb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.965176 27693 raft_consensus.cc:383] T e78e6d212daf4589a6b4b65711e356cb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:24.965238 27693 raft_consensus.cc:738] T e78e6d212daf4589a6b4b65711e356cb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:24.965401 27687 raft_consensus.cc:2241] T fa05f54ebc324b40855fa3288760e9e9 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:24.965557 27687 raft_consensus.cc:2270] T fa05f54ebc324b40855fa3288760e9e9 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:24.966360 27687 ts_tablet_manager.cc:1905] T fa05f54ebc324b40855fa3288760e9e9 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:24.966051 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_de5aee21a7e341a89dc0e5af838dfbcc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:24.967574 27693 consensus_queue.cc:260] T e78e6d212daf4589a6b4b65711e356cb P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:24.968297 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de5aee21a7e341a89dc0e5af838dfbcc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:24.971181 27693 ts_tablet_manager.cc:1428] T e78e6d212daf4589a6b4b65711e356cb P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250903 20:05:24.971551 27693 tablet_bootstrap.cc:492] T d859c0222c3647dab3dddd105e4cf0fc P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:24.973321 27693 tablet_bootstrap.cc:654] T d859c0222c3647dab3dddd105e4cf0fc P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:24.974793 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8330ab3e4b6f4560b9e41a9df280c211 consensus state has no leader
I20250903 20:05:24.975323 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_a41978d6a03a4bfe8f2b4cbd5f615ffb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:24.976143 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a41978d6a03a4bfe8f2b4cbd5f615ffb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:24.987344 27687 ts_tablet_manager.cc:1918] T fa05f54ebc324b40855fa3288760e9e9 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:24.987573 27687 log.cc:1199] T fa05f54ebc324b40855fa3288760e9e9 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/fa05f54ebc324b40855fa3288760e9e9
I20250903 20:05:24.987638 27684 ts_tablet_manager.cc:1905] T fa05f54ebc324b40855fa3288760e9e9 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:24.988560 27687 ts_tablet_manager.cc:1939] T fa05f54ebc324b40855fa3288760e9e9 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:24.989868 27687 tablet_replica.cc:331] T 8d2d64f103fc44d7b390559b92cbf4b4 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:24.990175 27687 raft_consensus.cc:2241] T 8d2d64f103fc44d7b390559b92cbf4b4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:24.990465 27687 raft_consensus.cc:2270] T 8d2d64f103fc44d7b390559b92cbf4b4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:24.991592 27687 ts_tablet_manager.cc:1905] T 8d2d64f103fc44d7b390559b92cbf4b4 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:24.991600 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 621) took 3976 ms. Trace:
I20250903 20:05:24.991744 27685 rpcz_store.cc:276] 0903 20:05:21.015071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.015634 (+ 563us) service_pool.cc:225] Handling call
0903 20:05:24.953671 (+3938037us) ts_tablet_manager.cc:1145] Deleting tablet d4c703dafbca4e669dbb18946ffac699
0903 20:05:24.953686 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:24.991542 (+ 37856us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3930499}
I20250903 20:05:24.992257 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 615) took 3977 ms. Trace:
I20250903 20:05:24.992364 27685 rpcz_store.cc:276] 0903 20:05:21.014527 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.015589 (+ 1062us) service_pool.cc:225] Handling call
0903 20:05:24.992091 (+3976502us) ts_tablet_manager.cc:1145] Deleting tablet 4af8c03c9d834405b4e25d35441db961
0903 20:05:24.992106 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:24.992212 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3953695}
I20250903 20:05:24.992743 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 622) took 3689 ms. Trace:
I20250903 20:05:24.992841 27685 rpcz_store.cc:276] 0903 20:05:21.303232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.303327 (+ 95us) service_pool.cc:225] Handling call
0903 20:05:24.992608 (+3689281us) ts_tablet_manager.cc:1145] Deleting tablet a72b85dfb34047528b2b28ff0d40f828
0903 20:05:24.992618 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:24.992702 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3681825}
I20250903 20:05:24.993214 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 626) took 3346 ms. Trace:
I20250903 20:05:24.993319 27685 rpcz_store.cc:276] 0903 20:05:21.646762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.650194 (+ 3432us) service_pool.cc:225] Handling call
0903 20:05:24.993085 (+3342891us) ts_tablet_manager.cc:1145] Deleting tablet 41056c5c5003415993bee74d7621bd14
0903 20:05:24.993094 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:24.993175 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3342673}
I20250903 20:05:24.993703 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 627) took 3346 ms. Trace:
I20250903 20:05:24.993799 27685 rpcz_store.cc:276] 0903 20:05:21.646863 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.650448 (+ 3585us) service_pool.cc:225] Handling call
0903 20:05:24.993545 (+3343097us) ts_tablet_manager.cc:1145] Deleting tablet a253e3399622409e92299069b19d3698
0903 20:05:24.993556 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:24.993656 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3342954}
I20250903 20:05:24.994161 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 628) took 3347 ms. Trace:
W20250903 20:05:24.995359 27183 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 54883.8us
I20250903 20:05:24.996495 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4157633525034a019fd6a80185892cec" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:24.997584 27320 leader_election.cc:343] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:24.997903 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet cb4d99139516405bb65c7480ddf259f1 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d4b8d4bf29e408cbdf9557c8b0d8a4b) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:24.998950 28396 raft_consensus.cc:695] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:24.999711 27692 tablet_bootstrap.cc:492] T 8605bc329627438998812ed028327f53 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:24.999540 28396 consensus_queue.cc:237] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:24.999836 27692 ts_tablet_manager.cc:1397] T 8605bc329627438998812ed028327f53 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.118s user 0.004s sys 0.000s
I20250903 20:05:25.000576 27692 raft_consensus.cc:357] T 8605bc329627438998812ed028327f53 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.000730 27692 raft_consensus.cc:383] T 8605bc329627438998812ed028327f53 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:25.000783 27692 raft_consensus.cc:738] T 8605bc329627438998812ed028327f53 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:25.000957 27692 consensus_queue.cc:260] T 8605bc329627438998812ed028327f53 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.001492 27692 ts_tablet_manager.cc:1428] T 8605bc329627438998812ed028327f53 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:25.001760 27692 tablet_bootstrap.cc:492] T ac9b5d4f39db4f05a15316585040d142 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:25.004771 27692 tablet_bootstrap.cc:654] T ac9b5d4f39db4f05a15316585040d142 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:25.013305 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 885fff47fea14b84ac6b34a57b827b4b (table default.Table_1c679659094d4666aaaf68f76f667890 [id=61a25739b03042f7bd5bb693b68656d5]) successfully deleted
I20250903 20:05:25.014241 27692 tablet_bootstrap.cc:492] T ac9b5d4f39db4f05a15316585040d142 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
W20250903 20:05:25.014119 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 849eb453d3e04b16bbb5df9017bfb2fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 849eb453d3e04b16bbb5df9017bfb2fd already in progress: creating tablet
I20250903 20:05:25.014560 27692 ts_tablet_manager.cc:1397] T ac9b5d4f39db4f05a15316585040d142 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
W20250903 20:05:25.014660 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 97d261080b8b479ca12267b6aa2e2ab0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97d261080b8b479ca12267b6aa2e2ab0 already in progress: creating tablet
W20250903 20:05:25.015259 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 15072aa10c5540a09f3ec4592e377d2a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15072aa10c5540a09f3ec4592e377d2a already in progress: creating tablet
I20250903 20:05:25.015444 27692 raft_consensus.cc:357] T ac9b5d4f39db4f05a15316585040d142 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.015684 27692 raft_consensus.cc:383] T ac9b5d4f39db4f05a15316585040d142 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:25.015784 27692 raft_consensus.cc:738] T ac9b5d4f39db4f05a15316585040d142 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
W20250903 20:05:25.015937 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 0060e6c4699a4bff8faddefb4ee50add because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0060e6c4699a4bff8faddefb4ee50add already in progress: creating tablet
I20250903 20:05:25.016093 27692 consensus_queue.cc:260] T ac9b5d4f39db4f05a15316585040d142 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.016044 27058 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet d859c0222c3647dab3dddd105e4cf0fc (table default.Table_dc24dd98fc4146eda8660576fd53a29d [id=7add4b636f1b4bffae663fdf432de05f]). Expected version 4 got 0
I20250903 20:05:25.016464 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 6f0cba70240f4f38a4fa5ccbcd99b3b1 (table default.Table_5ca5478ef07f4acdbbedb24af75f9e04 [id=7fb235b85bba4f7ab1b77bc5d2b9b58c]) successfully deleted
I20250903 20:05:25.016786 27692 ts_tablet_manager.cc:1428] T ac9b5d4f39db4f05a15316585040d142 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:25.017020 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 7c398634d6304a12b35c0579fba4c813 (table default.Table_14af7742726b4562a02fd2c9b2895277 [id=6d72901e5102476890f466fd406a7127]) successfully deleted
I20250903 20:05:25.017138 27692 tablet_bootstrap.cc:492] T cd92e3dafca6419f8eba7a4ef51a911e P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:25.017560 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet cb4d99139516405bb65c7480ddf259f1 (table default.Table_a2fef2e8171c466e86ce74826ba6e281 [id=cbcfc7067aca4796b39e3eb195f38903]) successfully deleted
I20250903 20:05:25.018891 27692 tablet_bootstrap.cc:654] T cd92e3dafca6419f8eba7a4ef51a911e P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:25.023099 27494 tablet_service.cc:1468] Processing CreateTablet for tablet 4fbd2782bdef4fc3bb40abb04472553d (DEFAULT_TABLE table=default.Table_7039490d1b01480f9a14cdf90ce4643f [id=340796f42eff4352a3fbee711f7f12c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:25.023554 27494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fbd2782bdef4fc3bb40abb04472553d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:25.024804 27693 tablet_bootstrap.cc:492] T d859c0222c3647dab3dddd105e4cf0fc P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:25.024936 27693 ts_tablet_manager.cc:1397] T d859c0222c3647dab3dddd105e4cf0fc P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.010s
I20250903 20:05:25.025740 27693 raft_consensus.cc:357] T d859c0222c3647dab3dddd105e4cf0fc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.025911 27693 raft_consensus.cc:383] T d859c0222c3647dab3dddd105e4cf0fc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:25.025962 27693 raft_consensus.cc:738] T d859c0222c3647dab3dddd105e4cf0fc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:25.026172 27693 consensus_queue.cc:260] T d859c0222c3647dab3dddd105e4cf0fc P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.027971 27693 ts_tablet_manager.cc:1428] T d859c0222c3647dab3dddd105e4cf0fc P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:25.029611 27687 ts_tablet_manager.cc:1918] T 8d2d64f103fc44d7b390559b92cbf4b4 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:25.029780 27687 log.cc:1199] T 8d2d64f103fc44d7b390559b92cbf4b4 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/8d2d64f103fc44d7b390559b92cbf4b4
I20250903 20:05:25.030158 27687 ts_tablet_manager.cc:1939] T 8d2d64f103fc44d7b390559b92cbf4b4 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:25.031788 27244 raft_consensus.cc:2466] T fe902f1b58924be1ac3a5585413f7789 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:25.032445 27692 tablet_bootstrap.cc:492] T cd92e3dafca6419f8eba7a4ef51a911e P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:25.032785 27693 tablet_bootstrap.cc:492] T c9e90abef9814338b70a3550ab5686e0 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:25.033460 27685 rpcz_store.cc:276] 0903 20:05:21.646949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.650627 (+ 3678us) service_pool.cc:225] Handling call
0903 20:05:24.994031 (+3343404us) ts_tablet_manager.cc:1145] Deleting tablet b7a6d5d8a97a4769a81ec7224d591f8c
0903 20:05:24.994040 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:24.994123 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3343249}
I20250903 20:05:25.033942 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 629) took 3381 ms. Trace:
I20250903 20:05:25.034055 27685 rpcz_store.cc:276] 0903 20:05:21.652810 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.652869 (+ 59us) service_pool.cc:225] Handling call
0903 20:05:25.033776 (+3380907us) ts_tablet_manager.cc:1145] Deleting tablet 5b26ed39f2144610b786f49e62e6f0a3
0903 20:05:25.033787 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:25.033908 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3380753}
I20250903 20:05:25.034425 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 625) took 3387 ms. Trace:
I20250903 20:05:25.034514 27685 rpcz_store.cc:276] 0903 20:05:21.646652 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.670497 (+ 23845us) service_pool.cc:225] Handling call
0903 20:05:25.034294 (+3363797us) ts_tablet_manager.cc:1145] Deleting tablet 2c772ab7c50c4994aacde5a05bb0f08e
0903 20:05:25.034303 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:25.034393 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3363213}
I20250903 20:05:25.034614 27693 tablet_bootstrap.cc:654] T c9e90abef9814338b70a3550ab5686e0 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:25.034770 27685 tablet_replica.cc:331] T f71d3dd9af2543299296f21381d1d563 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:25.034981 27685 raft_consensus.cc:2241] T f71d3dd9af2543299296f21381d1d563 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:25.035166 27685 raft_consensus.cc:2270] T f71d3dd9af2543299296f21381d1d563 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:25.035589 27692 ts_tablet_manager.cc:1397] T cd92e3dafca6419f8eba7a4ef51a911e P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.002s
I20250903 20:05:25.035720 27685 ts_tablet_manager.cc:1905] T f71d3dd9af2543299296f21381d1d563 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:25.037312 27687 tablet_replica.cc:331] T c6d67cd0c0304c77ba571b414878dad2 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:25.037694 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 908e748d08f74ed2a3071312c95ec07c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:25.037608 27692 raft_consensus.cc:357] T cd92e3dafca6419f8eba7a4ef51a911e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.037781 27692 raft_consensus.cc:383] T cd92e3dafca6419f8eba7a4ef51a911e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:25.037986 27687 raft_consensus.cc:2241] T c6d67cd0c0304c77ba571b414878dad2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:25.038240 27687 raft_consensus.cc:2270] T c6d67cd0c0304c77ba571b414878dad2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:25.038362 27692 raft_consensus.cc:738] T cd92e3dafca6419f8eba7a4ef51a911e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:25.039116 27687 ts_tablet_manager.cc:1905] T c6d67cd0c0304c77ba571b414878dad2 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:25.039335 27692 consensus_queue.cc:260] T cd92e3dafca6419f8eba7a4ef51a911e P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.040977 27574 heartbeater.cc:344] Connected to a master server at 127.26.65.125:44059
I20250903 20:05:25.042495 27692 ts_tablet_manager.cc:1428] T cd92e3dafca6419f8eba7a4ef51a911e P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250903 20:05:25.042915 27692 tablet_bootstrap.cc:492] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:25.046005 27692 tablet_bootstrap.cc:654] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:25.046521 27058 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 0ad64c699b294679b6fec37b5ad23528 (table default.Table_24e55a7c820c413f8514bd3ad7bd3269 [id=0904dc5d0c594bc3bbf32de2508c6195]). Expected version 3 got 0
I20250903 20:05:25.056412 27693 tablet_bootstrap.cc:492] T c9e90abef9814338b70a3550ab5686e0 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:25.056716 27693 ts_tablet_manager.cc:1397] T c9e90abef9814338b70a3550ab5686e0 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.000s
I20250903 20:05:25.057588 27693 raft_consensus.cc:357] T c9e90abef9814338b70a3550ab5686e0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.057825 27693 raft_consensus.cc:383] T c9e90abef9814338b70a3550ab5686e0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:25.057924 27693 raft_consensus.cc:738] T c9e90abef9814338b70a3550ab5686e0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:25.058243 27693 consensus_queue.cc:260] T c9e90abef9814338b70a3550ab5686e0 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.058840 27693 ts_tablet_manager.cc:1428] T c9e90abef9814338b70a3550ab5686e0 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:25.059177 27693 tablet_bootstrap.cc:492] T a0293a69f1c246d08d414e95729b4ab4 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:25.059574 28396 raft_consensus.cc:491] T 885fff47fea14b84ac6b34a57b827b4b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:25.059724 28396 raft_consensus.cc:513] T 885fff47fea14b84ac6b34a57b827b4b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.060403 28396 leader_election.cc:290] T 885fff47fea14b84ac6b34a57b827b4b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:25.060971 27693 tablet_bootstrap.cc:654] T a0293a69f1c246d08d414e95729b4ab4 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:25.061689 27687 ts_tablet_manager.cc:1918] T c6d67cd0c0304c77ba571b414878dad2 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:25.061792 27687 log.cc:1199] T c6d67cd0c0304c77ba571b414878dad2 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/c6d67cd0c0304c77ba571b414878dad2
I20250903 20:05:25.060119 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_796a25442570476884197d82618f9ac6" } new_table_name: "default.Table_7ac760e6522849d592391d4651feff19" modify_external_catalogs: true
I20250903 20:05:25.061856 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "885fff47fea14b84ac6b34a57b827b4b" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:25.062146 27687 ts_tablet_manager.cc:1939] T c6d67cd0c0304c77ba571b414878dad2 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:25.062151 27248 raft_consensus.cc:2391] T 885fff47fea14b84ac6b34a57b827b4b P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
W20250903 20:05:25.062677 27462 leader_election.cc:343] T 885fff47fea14b84ac6b34a57b827b4b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 885fff47fea14b84ac6b34a57b827b4b
I20250903 20:05:25.063390 27462 leader_election.cc:304] T 885fff47fea14b84ac6b34a57b827b4b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:25.064018 28396 raft_consensus.cc:3058] T 885fff47fea14b84ac6b34a57b827b4b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.064249 27687 tablet_replica.cc:331] T fe902f1b58924be1ac3a5585413f7789 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:25.064428 27687 raft_consensus.cc:2241] T fe902f1b58924be1ac3a5585413f7789 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:25.064568 27687 raft_consensus.cc:2270] T fe902f1b58924be1ac3a5585413f7789 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:25.065083 27687 ts_tablet_manager.cc:1905] T fe902f1b58924be1ac3a5585413f7789 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:25.065503 28298 raft_consensus.cc:491] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:25.065609 28298 raft_consensus.cc:513] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.066174 28298 leader_election.cc:290] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:25.066854 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cc6cf579e2642e99fefed723b0ee900" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:25.067215 28396 raft_consensus.cc:2747] T 885fff47fea14b84ac6b34a57b827b4b P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:25.067536 27320 leader_election.cc:343] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:25.072006 27692 tablet_bootstrap.cc:492] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:25.072249 27692 ts_tablet_manager.cc:1397] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.003s
I20250903 20:05:25.073091 27692 raft_consensus.cc:357] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.073343 27692 raft_consensus.cc:383] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:25.073443 27692 raft_consensus.cc:738] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:25.073727 27692 consensus_queue.cc:260] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.074318 27692 ts_tablet_manager.cc:1428] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:25.074746 27692 tablet_bootstrap.cc:492] T 45d9b6ff520348628d19671127b9426c P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:25.076524 27692 tablet_bootstrap.cc:654] T 45d9b6ff520348628d19671127b9426c P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:25.080873 27685 ts_tablet_manager.cc:1918] T f71d3dd9af2543299296f21381d1d563 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:25.081159 27685 log.cc:1199] T f71d3dd9af2543299296f21381d1d563 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/f71d3dd9af2543299296f21381d1d563
I20250903 20:05:25.081651 27685 ts_tablet_manager.cc:1939] T f71d3dd9af2543299296f21381d1d563 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:25.082126 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 631) took 3243 ms. Trace:
I20250903 20:05:25.082329 27685 rpcz_store.cc:276] 0903 20:05:21.838997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.839050 (+ 53us) service_pool.cc:225] Handling call
0903 20:05:25.034710 (+3195660us) ts_tablet_manager.cc:1145] Deleting tablet f71d3dd9af2543299296f21381d1d563
0903 20:05:25.034718 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:25.070508 (+ 35790us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:25.080689 (+ 10181us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:25.082102 (+ 1413us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3195472}
I20250903 20:05:25.084172 27692 tablet_bootstrap.cc:492] T 45d9b6ff520348628d19671127b9426c P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:25.085412 27687 ts_tablet_manager.cc:1918] T fe902f1b58924be1ac3a5585413f7789 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:25.085506 27685 tablet_replica.cc:331] T dbfdab4332234b2a93fd3c901a9cb645 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:25.085655 27692 ts_tablet_manager.cc:1397] T 45d9b6ff520348628d19671127b9426c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.001s
I20250903 20:05:25.085899 27685 raft_consensus.cc:2241] T dbfdab4332234b2a93fd3c901a9cb645 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:25.086421 27685 raft_consensus.cc:2270] T dbfdab4332234b2a93fd3c901a9cb645 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:25.086534 27692 raft_consensus.cc:357] T 45d9b6ff520348628d19671127b9426c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.086867 27687 log.cc:1199] T fe902f1b58924be1ac3a5585413f7789 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/fe902f1b58924be1ac3a5585413f7789
I20250903 20:05:25.087105 27685 ts_tablet_manager.cc:1905] T dbfdab4332234b2a93fd3c901a9cb645 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:25.087260 27687 ts_tablet_manager.cc:1939] T fe902f1b58924be1ac3a5585413f7789 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:25.087389 27692 raft_consensus.cc:383] T 45d9b6ff520348628d19671127b9426c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:25.087502 27692 raft_consensus.cc:738] T 45d9b6ff520348628d19671127b9426c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:25.087924 27692 consensus_queue.cc:260] T 45d9b6ff520348628d19671127b9426c P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.088999 27693 tablet_bootstrap.cc:492] T a0293a69f1c246d08d414e95729b4ab4 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:25.089380 27693 ts_tablet_manager.cc:1397] T a0293a69f1c246d08d414e95729b4ab4 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.003s
I20250903 20:05:25.089833 27692 ts_tablet_manager.cc:1428] T 45d9b6ff520348628d19671127b9426c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:25.090067 27687 tablet_replica.cc:331] T dbfdab4332234b2a93fd3c901a9cb645 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:25.090425 27692 tablet_bootstrap.cc:492] T d859c0222c3647dab3dddd105e4cf0fc P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:25.090660 27687 raft_consensus.cc:2241] T dbfdab4332234b2a93fd3c901a9cb645 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:25.090860 27687 raft_consensus.cc:2270] T dbfdab4332234b2a93fd3c901a9cb645 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:25.092633 27692 tablet_bootstrap.cc:654] T d859c0222c3647dab3dddd105e4cf0fc P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:25.090790 27693 raft_consensus.cc:357] T a0293a69f1c246d08d414e95729b4ab4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.092928 27693 raft_consensus.cc:383] T a0293a69f1c246d08d414e95729b4ab4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:25.093046 27693 raft_consensus.cc:738] T a0293a69f1c246d08d414e95729b4ab4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:25.093320 27693 consensus_queue.cc:260] T a0293a69f1c246d08d414e95729b4ab4 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.094187 27693 ts_tablet_manager.cc:1428] T a0293a69f1c246d08d414e95729b4ab4 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250903 20:05:25.094543 27693 tablet_bootstrap.cc:492] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:25.095082 27433 heartbeater.cc:344] Connected to a master server at 127.26.65.125:44059
I20250903 20:05:25.096249 27693 tablet_bootstrap.cc:654] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:25.097661 27687 ts_tablet_manager.cc:1905] T dbfdab4332234b2a93fd3c901a9cb645 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:25.103940 27687 ts_tablet_manager.cc:1918] T dbfdab4332234b2a93fd3c901a9cb645 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:25.104081 27687 log.cc:1199] T dbfdab4332234b2a93fd3c901a9cb645 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/dbfdab4332234b2a93fd3c901a9cb645
I20250903 20:05:25.104370 27687 ts_tablet_manager.cc:1939] T dbfdab4332234b2a93fd3c901a9cb645 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:25.109128 28396 raft_consensus.cc:491] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:25.109313 28396 raft_consensus.cc:513] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.109879 27693 tablet_bootstrap.cc:492] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:25.110029 27693 ts_tablet_manager.cc:1397] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.003s
I20250903 20:05:25.110131 28396 leader_election.cc:290] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:25.110858 27693 raft_consensus.cc:357] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.111413 27693 raft_consensus.cc:383] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250903 20:05:25.111438 27462 leader_election.cc:343] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: cb4d99139516405bb65c7480ddf259f1
I20250903 20:05:25.111521 27693 raft_consensus.cc:738] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:25.111776 27693 consensus_queue.cc:260] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.115034 27693 ts_tablet_manager.cc:1428] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250903 20:05:25.115408 27693 tablet_bootstrap.cc:492] T 1bcb9d5f010f4297bab78c0183889238 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:25.117201 27693 tablet_bootstrap.cc:654] T 1bcb9d5f010f4297bab78c0183889238 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:25.119318 27692 tablet_bootstrap.cc:492] T d859c0222c3647dab3dddd105e4cf0fc P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:25.119468 27692 ts_tablet_manager.cc:1397] T d859c0222c3647dab3dddd105e4cf0fc P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.001s
I20250903 20:05:25.119587 27499 tablet_service.cc:1468] Processing CreateTablet for tablet 8bebea9fa64e459cb24639e1f2d11ac3 (DEFAULT_TABLE table=default.Table_fae7d62089f24ca88970b4691b51f867 [id=bd8f3bd183644082856d3862b55ea3f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:25.119987 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bebea9fa64e459cb24639e1f2d11ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:25.120266 27692 raft_consensus.cc:357] T d859c0222c3647dab3dddd105e4cf0fc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.120579 27692 raft_consensus.cc:383] T d859c0222c3647dab3dddd105e4cf0fc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:25.120637 27692 raft_consensus.cc:738] T d859c0222c3647dab3dddd105e4cf0fc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:25.120823 27692 consensus_queue.cc:260] T d859c0222c3647dab3dddd105e4cf0fc P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.121259 27692 ts_tablet_manager.cc:1428] T d859c0222c3647dab3dddd105e4cf0fc P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:25.121536 27692 tablet_bootstrap.cc:492] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:25.123175 27692 tablet_bootstrap.cc:654] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:25.129127 27494 tablet_service.cc:1468] Processing CreateTablet for tablet 3e930dbaac0e40ee8f175ccf87592d31 (DEFAULT_TABLE table=default.Table_480743ddb101436ca551a49c01a368a7 [id=2610a4c446be491d94886b96f8a58904]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:25.130370 27494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e930dbaac0e40ee8f175ccf87592d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:25.139494 27502 tablet_service.cc:1468] Processing CreateTablet for tablet fb9e90d6e97d4ab4b27bd5efb2ee4b55 (DEFAULT_TABLE table=default.Table_28fd9ad363e94c149f59785ca4cdeb7b [id=5b822183a0dc47db87a1b697d9146deb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:25.140761 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 84bd22cd8bd6462982d388222858972a (DEFAULT_TABLE table=default.Table_f42f6303480447ddac82d66378c1ce36 [id=30381efd3bc44a3fb1eddb710355b429]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:25.142525 27496 tablet_service.cc:1468] Processing CreateTablet for tablet d1e89ad94adb4de8b3738e30a84a8f79 (DEFAULT_TABLE table=default.Table_564e21bb80a94d5888b76f1471d9b937 [id=3941089860a34747be28eae8bdb5c8cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:25.143074 27499 tablet_service.cc:1468] Processing CreateTablet for tablet 90776f6369744cec8ce380b7695c44f1 (DEFAULT_TABLE table=default.Table_eaff993b7bb7437580aae238faa74b3e [id=2650c8259ef4489695c56b6a2ba898d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:25.145548 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb9e90d6e97d4ab4b27bd5efb2ee4b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:25.153363 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84bd22cd8bd6462982d388222858972a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:25.155710 27488 tablet_service.cc:1468] Processing CreateTablet for tablet 0deb2e04842d48c6a572bba7d3cc8a77 (DEFAULT_TABLE table=default.Table_078beaf0d9304922afad9311ba40775a [id=03858212036244b5bc3d50873db9cf5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:25.156255 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0deb2e04842d48c6a572bba7d3cc8a77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:25.157617 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1e89ad94adb4de8b3738e30a84a8f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:25.164218 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90776f6369744cec8ce380b7695c44f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:25.173755 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet fa05f54ebc324b40855fa3288760e9e9 (table default.Table_dacd8238c8334c1c9f66e745f5036a06 [id=7ec413d945394dcfa36492f7b59aaaa2]) successfully deleted
W20250903 20:05:25.174161 27052 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 155994us
I20250903 20:05:25.175612 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 4fbd2782bdef4fc3bb40abb04472553d (DEFAULT_TABLE table=default.Table_7039490d1b01480f9a14cdf90ce4643f [id=340796f42eff4352a3fbee711f7f12c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:25.176187 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fbd2782bdef4fc3bb40abb04472553d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:25.092540 27183 leader_election.cc:343] T 45d9b6ff520348628d19671127b9426c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:25.111142 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb4d99139516405bb65c7480ddf259f1" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:25.184453 27248 raft_consensus.cc:2466] T cb4d99139516405bb65c7480ddf259f1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:25.185083 27462 leader_election.cc:304] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:25.185854 28396 raft_consensus.cc:2802] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:25.186024 28396 raft_consensus.cc:491] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:25.186141 28396 raft_consensus.cc:3058] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.188246 27183 leader_election.cc:304] T 45d9b6ff520348628d19671127b9426c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:25.188673 27183 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 190905us
W20250903 20:05:25.188951 27183 leader_election.cc:343] T dbfdab4332234b2a93fd3c901a9cb645 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:25.189085 27183 leader_election.cc:304] T dbfdab4332234b2a93fd3c901a9cb645 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:25.189270 27183 raft_consensus.cc:2684] T dbfdab4332234b2a93fd3c901a9cb645 P 89f8f0ee68b940db9ce250c0604f934a: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250903 20:05:25.189594 27183 leader_election.cc:343] T 4157633525034a019fd6a80185892cec P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:25.189729 27183 leader_election.cc:304] T 4157633525034a019fd6a80185892cec P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:25.190137 27183 leader_election.cc:343] T 0ad64c699b294679b6fec37b5ad23528 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:25.190276 27183 leader_election.cc:304] T 0ad64c699b294679b6fec37b5ad23528 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:25.190682 27183 leader_election.cc:343] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:25.190807 27183 leader_election.cc:304] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:25.191823 28245 raft_consensus.cc:2747] T 45d9b6ff520348628d19671127b9426c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:25.192014 28245 raft_consensus.cc:2747] T 4157633525034a019fd6a80185892cec P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:25.192145 28245 raft_consensus.cc:2747] T 0ad64c699b294679b6fec37b5ad23528 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:25.192260 28245 raft_consensus.cc:2747] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250903 20:05:25.199450 28298 consensus_queue.cc:1035] T 912eee5b48ef467bbd6bbe34f8aa390c P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:25.217000 27062 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 516830fc42e84af693b3af52caee30ee (table default.Table_4d303308b63c4ce09056e226d1a30498 [id=098bc9a9b0a24a4a9edb7859310ac9b3]). Expected version 3 got 0
I20250903 20:05:25.219401 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_3fe8a44f17364b249736f1722bd1c9a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:25.220247 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3fe8a44f17364b249736f1722bd1c9a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:25.223172 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_def69a11d277479cbfc58718d39b63ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:25.225108 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_def69a11d277479cbfc58718d39b63ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:25.224318 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_108ee548249b426e934082d6bc507068"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:25.225760 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_108ee548249b426e934082d6bc507068 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:25.240152 27505 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d2d64f103fc44d7b390559b92cbf4b4 with delete_type TABLET_DATA_DELETED (replaced by tablet a38f65e992884635aae9942681c0dc4f) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:25.241050 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 3e930dbaac0e40ee8f175ccf87592d31 (DEFAULT_TABLE table=default.Table_480743ddb101436ca551a49c01a368a7 [id=2610a4c446be491d94886b96f8a58904]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:25.241874 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e930dbaac0e40ee8f175ccf87592d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:25.243865 27692 tablet_bootstrap.cc:492] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:25.244567 27505 tablet_service.cc:1515] Processing DeleteTablet for tablet fe902f1b58924be1ac3a5585413f7789 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:25.245544 27692 ts_tablet_manager.cc:1397] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.124s user 0.008s sys 0.030s
I20250903 20:05:25.246865 27692 raft_consensus.cc:357] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.248641 28396 raft_consensus.cc:513] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.249562 28396 leader_election.cc:290] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:25.249806 27692 raft_consensus.cc:383] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:25.250141 27692 raft_consensus.cc:738] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:25.250660 27692 consensus_queue.cc:260] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.252168 27692 ts_tablet_manager.cc:1428] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250903 20:05:25.253624 27692 tablet_bootstrap.cc:492] T cd6d6bb514134bdf8e3bc0aaad180732 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
W20250903 20:05:25.258922 27462 leader_election.cc:343] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: cb4d99139516405bb65c7480ddf259f1
I20250903 20:05:25.259125 27692 tablet_bootstrap.cc:654] T cd6d6bb514134bdf8e3bc0aaad180732 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:25.265172 27349 tablet_service.cc:1468] Processing CreateTablet for tablet 8bebea9fa64e459cb24639e1f2d11ac3 (DEFAULT_TABLE table=default.Table_fae7d62089f24ca88970b4691b51f867 [id=bd8f3bd183644082856d3862b55ea3f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:25.265740 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bebea9fa64e459cb24639e1f2d11ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:25.283412 27692 tablet_bootstrap.cc:492] T cd6d6bb514134bdf8e3bc0aaad180732 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:25.283694 27692 ts_tablet_manager.cc:1397] T cd6d6bb514134bdf8e3bc0aaad180732 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.004s
I20250903 20:05:25.284389 27349 tablet_service.cc:1468] Processing CreateTablet for tablet 90776f6369744cec8ce380b7695c44f1 (DEFAULT_TABLE table=default.Table_eaff993b7bb7437580aae238faa74b3e [id=2650c8259ef4489695c56b6a2ba898d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:25.284636 27692 raft_consensus.cc:357] T cd6d6bb514134bdf8e3bc0aaad180732 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.284999 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90776f6369744cec8ce380b7695c44f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:25.286731 27692 raft_consensus.cc:383] T cd6d6bb514134bdf8e3bc0aaad180732 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:25.287282 27692 raft_consensus.cc:738] T cd6d6bb514134bdf8e3bc0aaad180732 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:25.287855 27692 consensus_queue.cc:260] T cd6d6bb514134bdf8e3bc0aaad180732 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.288331 27228 tablet_service.cc:1468] Processing CreateTablet for tablet 3e930dbaac0e40ee8f175ccf87592d31 (DEFAULT_TABLE table=default.Table_480743ddb101436ca551a49c01a368a7 [id=2610a4c446be491d94886b96f8a58904]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:25.288822 27228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e930dbaac0e40ee8f175ccf87592d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:25.289605 27692 ts_tablet_manager.cc:1428] T cd6d6bb514134bdf8e3bc0aaad180732 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250903 20:05:25.289901 27692 tablet_bootstrap.cc:492] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:25.293457 27692 tablet_bootstrap.cc:654] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:25.294684 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb4d99139516405bb65c7480ddf259f1" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:25.294900 27248 raft_consensus.cc:3058] T cb4d99139516405bb65c7480ddf259f1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.298718 28298 raft_consensus.cc:491] T 283a3afed7414a958b2dc4176f00bb01 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:25.298882 28298 raft_consensus.cc:513] T 283a3afed7414a958b2dc4176f00bb01 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.299671 28298 leader_election.cc:290] T 283a3afed7414a958b2dc4176f00bb01 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:25.301033 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "283a3afed7414a958b2dc4176f00bb01" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:25.301801 27320 leader_election.cc:343] T 283a3afed7414a958b2dc4176f00bb01 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:25.298136 27248 raft_consensus.cc:2466] T cb4d99139516405bb65c7480ddf259f1 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:25.302695 27462 leader_election.cc:304] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:25.303236 28396 raft_consensus.cc:2802] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:25.303396 28396 raft_consensus.cc:695] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:25.303740 28396 consensus_queue.cc:237] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.307132 27692 tablet_bootstrap.cc:492] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:25.308135 27692 ts_tablet_manager.cc:1397] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.001s
I20250903 20:05:25.309430 27692 raft_consensus.cc:357] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.309762 27692 raft_consensus.cc:383] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:25.309877 27692 raft_consensus.cc:738] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:25.310250 27692 consensus_queue.cc:260] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:25.309662 27182 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.941s user 0.045s sys 0.079s
W20250903 20:05:25.310565 27182 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.942s user 0.045s sys 0.079s
I20250903 20:05:25.310912 27321 leader_election.cc:304] T ac9b5d4f39db4f05a15316585040d142 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:25.311053 28514 raft_consensus.cc:491] T d859c0222c3647dab3dddd105e4cf0fc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250903 20:05:25.311152 27321 raft_consensus.cc:2684] T ac9b5d4f39db4f05a15316585040d142 P 51578fb7bceb4e119b3ab1e58c4f5528: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250903 20:05:25.311825 27321 leader_election.cc:304] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:25.311995 27321 raft_consensus.cc:2684] T 7c398634d6304a12b35c0579fba4c813 P 51578fb7bceb4e119b3ab1e58c4f5528: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250903 20:05:25.312187 27692 ts_tablet_manager.cc:1428] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250903 20:05:25.312489 27692 tablet_bootstrap.cc:492] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:25.314283 27220 tablet_service.cc:1468] Processing CreateTablet for tablet 4fbd2782bdef4fc3bb40abb04472553d (DEFAULT_TABLE table=default.Table_7039490d1b01480f9a14cdf90ce4643f [id=340796f42eff4352a3fbee711f7f12c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:25.315068 27692 tablet_bootstrap.cc:654] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:25.316892 28298 raft_consensus.cc:491] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:25.317142 28298 raft_consensus.cc:513] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.318041 28298 leader_election.cc:290] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:25.319111 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45d9b6ff520348628d19671127b9426c" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:25.319375 27524 raft_consensus.cc:2466] T 45d9b6ff520348628d19671127b9426c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:25.319985 27320 leader_election.cc:304] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:25.321149 28298 raft_consensus.cc:2802] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:25.321322 28298 raft_consensus.cc:491] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:25.321466 28298 raft_consensus.cc:3058] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.324831 28298 raft_consensus.cc:513] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.337877 28298 leader_election.cc:290] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:25.338706 27226 tablet_service.cc:1468] Processing CreateTablet for tablet 90776f6369744cec8ce380b7695c44f1 (DEFAULT_TABLE table=default.Table_eaff993b7bb7437580aae238faa74b3e [id=2650c8259ef4489695c56b6a2ba898d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:25.339861 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe902f1b58924be1ac3a5585413f7789" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:25.340191 27248 raft_consensus.cc:2391] T fe902f1b58924be1ac3a5585413f7789 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 51578fb7bceb4e119b3ab1e58c4f5528 in current term 1: Already voted for candidate e72ef1dbcc5143cf9f04a808f101c80f in this term.
I20250903 20:05:25.339277 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 8bebea9fa64e459cb24639e1f2d11ac3 (DEFAULT_TABLE table=default.Table_fae7d62089f24ca88970b4691b51f867 [id=bd8f3bd183644082856d3862b55ea3f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:25.341442 28514 raft_consensus.cc:513] T d859c0222c3647dab3dddd105e4cf0fc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.342149 28514 leader_election.cc:290] T d859c0222c3647dab3dddd105e4cf0fc P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:25.342550 27245 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd6d6bb514134bdf8e3bc0aaad180732" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:25.342706 27245 raft_consensus.cc:2466] T cd6d6bb514134bdf8e3bc0aaad180732 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:25.342118 27247 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cc6cf579e2642e99fefed723b0ee900" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:25.343020 27247 raft_consensus.cc:2466] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:25.343732 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d859c0222c3647dab3dddd105e4cf0fc" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:25.343953 27524 raft_consensus.cc:2466] T d859c0222c3647dab3dddd105e4cf0fc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:25.345098 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d859c0222c3647dab3dddd105e4cf0fc" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:25.345314 27380 raft_consensus.cc:2466] T d859c0222c3647dab3dddd105e4cf0fc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:25.346820 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45d9b6ff520348628d19671127b9426c" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:25.346988 27239 raft_consensus.cc:2466] T 45d9b6ff520348628d19671127b9426c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:25.347239 27692 tablet_bootstrap.cc:492] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:25.347450 27692 ts_tablet_manager.cc:1397] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.010s
I20250903 20:05:25.347908 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "283a3afed7414a958b2dc4176f00bb01" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:25.348086 27240 raft_consensus.cc:2466] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:25.348568 27246 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8605bc329627438998812ed028327f53" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:25.348773 27246 raft_consensus.cc:2391] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 51578fb7bceb4e119b3ab1e58c4f5528 in current term 1: Already voted for candidate 89f8f0ee68b940db9ce250c0604f934a in this term.
I20250903 20:05:25.348629 27692 raft_consensus.cc:357] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.346159 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4157633525034a019fd6a80185892cec" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:25.349015 27249 raft_consensus.cc:2466] T 4157633525034a019fd6a80185892cec P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:25.349042 27692 raft_consensus.cc:383] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:25.349227 27692 raft_consensus.cc:738] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:25.349287 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45d9b6ff520348628d19671127b9426c" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:25.349434 27239 raft_consensus.cc:3058] T 45d9b6ff520348628d19671127b9426c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.349773 27692 consensus_queue.cc:260] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.350343 27321 leader_election.cc:304] T fe902f1b58924be1ac3a5585413f7789 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:25.350579 27321 raft_consensus.cc:2684] T fe902f1b58924be1ac3a5585413f7789 P 51578fb7bceb4e119b3ab1e58c4f5528: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250903 20:05:25.351015 27692 ts_tablet_manager.cc:1428] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250903 20:05:25.351743 27692 tablet_bootstrap.cc:492] T 0ad64c699b294679b6fec37b5ad23528 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:25.352108 27321 leader_election.cc:304] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:25.352103 28568 consensus_queue.cc:1035] T 912eee5b48ef467bbd6bbe34f8aa390c P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:25.352842 27321 leader_election.cc:304] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:25.353572 27695 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.666s user 0.037s sys 0.071s
W20250903 20:05:25.353695 27695 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.666s user 0.037s sys 0.071s
I20250903 20:05:25.353773 28570 raft_consensus.cc:2802] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:25.354225 28570 raft_consensus.cc:491] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:25.354501 27321 leader_election.cc:304] T 283a3afed7414a958b2dc4176f00bb01 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:25.354512 28570 raft_consensus.cc:3058] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.355199 27695 tablet_bootstrap.cc:492] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:25.355608 27695 ts_tablet_manager.cc:1397] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 1.424s user 0.042s sys 0.071s
I20250903 20:05:25.356402 27239 raft_consensus.cc:2466] T 45d9b6ff520348628d19671127b9426c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:25.356907 27184 leader_election.cc:304] T d859c0222c3647dab3dddd105e4cf0fc P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:25.357278 28570 raft_consensus.cc:513] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.357465 28522 raft_consensus.cc:2802] T d859c0222c3647dab3dddd105e4cf0fc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:25.353483 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fbd2782bdef4fc3bb40abb04472553d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:25.357579 28522 raft_consensus.cc:491] T d859c0222c3647dab3dddd105e4cf0fc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:25.358534 28522 raft_consensus.cc:3058] T d859c0222c3647dab3dddd105e4cf0fc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.358794 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90776f6369744cec8ce380b7695c44f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:25.360719 27321 leader_election.cc:304] T 8605bc329627438998812ed028327f53 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:25.360952 28570 leader_election.cc:290] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:25.361086 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bebea9fa64e459cb24639e1f2d11ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:25.366940 27695 raft_consensus.cc:357] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.367249 27695 raft_consensus.cc:383] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:25.367349 27695 raft_consensus.cc:738] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:25.367647 27695 consensus_queue.cc:260] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.368824 27695 ts_tablet_manager.cc:1428] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250903 20:05:25.352910 28569 raft_consensus.cc:2802] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:25.358230 28568 raft_consensus.cc:2802] T 283a3afed7414a958b2dc4176f00bb01 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250903 20:05:25.360978 27321 raft_consensus.cc:2684] T 8605bc329627438998812ed028327f53 P 51578fb7bceb4e119b3ab1e58c4f5528: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250903 20:05:25.369072 28568 raft_consensus.cc:491] T 283a3afed7414a958b2dc4176f00bb01 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:25.369206 28568 raft_consensus.cc:3058] T 283a3afed7414a958b2dc4176f00bb01 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.371913 28568 raft_consensus.cc:513] T 283a3afed7414a958b2dc4176f00bb01 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.372583 28568 leader_election.cc:290] T 283a3afed7414a958b2dc4176f00bb01 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:25.373154 27321 leader_election.cc:304] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:25.373517 28568 raft_consensus.cc:2802] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:25.373662 28568 raft_consensus.cc:491] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:25.373809 28568 raft_consensus.cc:3058] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.377266 28568 raft_consensus.cc:513] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.378167 28568 leader_election.cc:290] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:25.373754 27321 leader_election.cc:304] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:25.378479 27695 tablet_bootstrap.cc:492] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:25.373883 28569 raft_consensus.cc:491] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:25.378542 28570 raft_consensus.cc:2802] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:25.378765 28570 raft_consensus.cc:695] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:25.379634 28570 consensus_queue.cc:237] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.380837 28569 raft_consensus.cc:3058] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.382925 27241 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4157633525034a019fd6a80185892cec" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:25.383827 27241 raft_consensus.cc:3058] T 4157633525034a019fd6a80185892cec P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.383147 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cc6cf579e2642e99fefed723b0ee900" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:25.384163 27249 raft_consensus.cc:3058] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.383280 27246 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "283a3afed7414a958b2dc4176f00bb01" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:25.384388 27246 raft_consensus.cc:3058] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.386668 27692 tablet_bootstrap.cc:654] T 0ad64c699b294679b6fec37b5ad23528 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:25.389930 27695 tablet_bootstrap.cc:654] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:25.391702 27241 raft_consensus.cc:2466] T 4157633525034a019fd6a80185892cec P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:25.392773 27321 leader_election.cc:304] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:25.393148 28569 raft_consensus.cc:513] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.393404 28571 raft_consensus.cc:2802] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:25.395237 28522 raft_consensus.cc:513] T d859c0222c3647dab3dddd105e4cf0fc P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.395496 28571 raft_consensus.cc:695] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:25.395855 28568 raft_consensus.cc:491] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:25.395982 28568 raft_consensus.cc:513] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.395896 28571 consensus_queue.cc:237] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.396518 28568 leader_election.cc:290] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:25.397195 27205 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:56514 (request call id 130) took 1503 ms. Trace:
I20250903 20:05:25.397375 27205 rpcz_store.cc:276] 0903 20:05:23.893770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:23.893857 (+ 87us) service_pool.cc:225] Handling call
0903 20:05:25.397165 (+1503308us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250903 20:05:25.412932 28298 raft_consensus.cc:491] T a0293a69f1c246d08d414e95729b4ab4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:25.413251 28298 raft_consensus.cc:513] T a0293a69f1c246d08d414e95729b4ab4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.415405 27226 tablet_service.cc:1468] Processing CreateTablet for tablet fb9e90d6e97d4ab4b27bd5efb2ee4b55 (DEFAULT_TABLE table=default.Table_28fd9ad363e94c149f59785ca4cdeb7b [id=5b822183a0dc47db87a1b697d9146deb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:25.420186 27357 tablet_service.cc:1468] Processing CreateTablet for tablet fb9e90d6e97d4ab4b27bd5efb2ee4b55 (DEFAULT_TABLE table=default.Table_28fd9ad363e94c149f59785ca4cdeb7b [id=5b822183a0dc47db87a1b697d9146deb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:25.420334 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_0955d4bd84b645e3b796679b761bbba7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:25.420650 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb9e90d6e97d4ab4b27bd5efb2ee4b55. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:25.420993 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0955d4bd84b645e3b796679b761bbba7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:25.422674 28396 raft_consensus.cc:491] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:25.422839 28396 raft_consensus.cc:513] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.423520 28396 leader_election.cc:290] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:25.427517 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 84bd22cd8bd6462982d388222858972a (DEFAULT_TABLE table=default.Table_f42f6303480447ddac82d66378c1ce36 [id=30381efd3bc44a3fb1eddb710355b429]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:25.427888 27352 tablet_service.cc:1468] Processing CreateTablet for tablet 0deb2e04842d48c6a572bba7d3cc8a77 (DEFAULT_TABLE table=default.Table_078beaf0d9304922afad9311ba40775a [id=03858212036244b5bc3d50873db9cf5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:25.428076 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84bd22cd8bd6462982d388222858972a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:25.428390 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0deb2e04842d48c6a572bba7d3cc8a77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:25.428786 27358 tablet_service.cc:1468] Processing CreateTablet for tablet d1e89ad94adb4de8b3738e30a84a8f79 (DEFAULT_TABLE table=default.Table_564e21bb80a94d5888b76f1471d9b937 [id=3941089860a34747be28eae8bdb5c8cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:25.429198 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1e89ad94adb4de8b3738e30a84a8f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:25.430424 28298 leader_election.cc:290] T a0293a69f1c246d08d414e95729b4ab4 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:25.432132 27350 tablet_service.cc:1515] Processing DeleteTablet for tablet e78e6d212daf4589a6b4b65711e356cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:25.432328 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet 4157633525034a019fd6a80185892cec with delete_type TABLET_DATA_DELETED (replaced by tablet ed7c564138cc43e5a3bb4e92fb5060b6) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:25.432425 27687 tablet_replica.cc:331] T e78e6d212daf4589a6b4b65711e356cb P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:25.432586 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet 283a3afed7414a958b2dc4176f00bb01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:25.432624 27687 raft_consensus.cc:2241] T e78e6d212daf4589a6b4b65711e356cb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:25.432776 27687 raft_consensus.cc:2270] T e78e6d212daf4589a6b4b65711e356cb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:25.433600 27687 ts_tablet_manager.cc:1905] T e78e6d212daf4589a6b4b65711e356cb P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:25.440527 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 005d7bc2c7534f288caec8326fdf2416
I20250903 20:05:25.440971 27687 ts_tablet_manager.cc:1918] T e78e6d212daf4589a6b4b65711e356cb P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:25.441212 27687 log.cc:1199] T e78e6d212daf4589a6b4b65711e356cb P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/e78e6d212daf4589a6b4b65711e356cb
I20250903 20:05:25.441624 27687 ts_tablet_manager.cc:1939] T e78e6d212daf4589a6b4b65711e356cb P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:25.442270 27687 tablet_replica.cc:331] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:25.443377 27057 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet d859c0222c3647dab3dddd105e4cf0fc (table default.Table_dc24dd98fc4146eda8660576fd53a29d [id=7add4b636f1b4bffae663fdf432de05f]). Expected version 4 got 0
I20250903 20:05:25.446033 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_0e14fb71c06946bdabb837fff47ec0e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:25.446707 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e14fb71c06946bdabb837fff47ec0e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:25.457337 27488 tablet_service.cc:1515] Processing DeleteTablet for tablet 005d7bc2c7534f288caec8326fdf2416 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:25.457957 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 46e3a65118b744298b5536f88c22f1db (table default.Table_4258f924ddb3449bb5f49fb7a88a5073 [id=decb9fcf2ee64971ba8bcd8326736852]) successfully deleted
I20250903 20:05:25.458019 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 45d9b6ff520348628d19671127b9426c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:25.458269 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 0906165a50b047c8a776b30832149964 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0906165a50b047c8a776b30832149964 already in progress: creating tablet
I20250903 20:05:25.458109 27358 tablet_service.cc:1515] Processing DeleteTablet for tablet 7cc6cf579e2642e99fefed723b0ee900 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:25.458612 27320 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 138653us
I20250903 20:05:25.458290 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet cd6d6bb514134bdf8e3bc0aaad180732 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:25.459743 26890 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 52055.4us
I20250903 20:05:25.459789 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45d9b6ff520348628d19671127b9426c" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:25.460005 27524 raft_consensus.cc:3058] T 45d9b6ff520348628d19671127b9426c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.463269 27524 raft_consensus.cc:2466] T 45d9b6ff520348628d19671127b9426c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:25.464172 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cc6cf579e2642e99fefed723b0ee900" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:25.465310 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 9685cba2f73b412382a69dab2c7f50c1 with delete_type TABLET_DATA_DELETED (replaced by tablet 47c1287bff1145a5b2ab6d14dbd608c7) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:25.466027 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "283a3afed7414a958b2dc4176f00bb01" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:25.467218 28522 leader_election.cc:290] T d859c0222c3647dab3dddd105e4cf0fc P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:25.467556 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d859c0222c3647dab3dddd105e4cf0fc" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:25.467821 27524 raft_consensus.cc:3058] T d859c0222c3647dab3dddd105e4cf0fc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.468896 27687 raft_consensus.cc:2241] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:25.469429 27687 pending_rounds.cc:70] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528: Trying to abort 1 pending ops.
I20250903 20:05:25.469491 27687 pending_rounds.cc:77] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384974424666112 op_type: NO_OP noop_request { }
I20250903 20:05:25.469591 27687 raft_consensus.cc:2887] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:25.469671 27687 raft_consensus.cc:2270] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:25.471153 27524 raft_consensus.cc:2466] T d859c0222c3647dab3dddd105e4cf0fc P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:25.473150 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "516830fc42e84af693b3af52caee30ee" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:25.473863 27248 raft_consensus.cc:2466] T 516830fc42e84af693b3af52caee30ee P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:25.474704 27228 tablet_service.cc:1468] Processing CreateTablet for tablet 84bd22cd8bd6462982d388222858972a (DEFAULT_TABLE table=default.Table_f42f6303480447ddac82d66378c1ce36 [id=30381efd3bc44a3fb1eddb710355b429]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:25.474789 28569 leader_election.cc:290] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:25.476363 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2cc35f4244a42eeb4a65de749824c11" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:25.477036 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4157633525034a019fd6a80185892cec" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:25.479053 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0293a69f1c246d08d414e95729b4ab4" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:25.480069 27320 leader_election.cc:343] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:25.480412 27320 leader_election.cc:343] T 283a3afed7414a958b2dc4176f00bb01 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:25.480772 27320 leader_election.cc:343] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:25.481060 27320 leader_election.cc:343] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:25.481829 28570 raft_consensus.cc:491] T 0ad64c699b294679b6fec37b5ad23528 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250903 20:05:25.481985 27320 leader_election.cc:343] T a0293a69f1c246d08d414e95729b4ab4 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:25.482141 28570 raft_consensus.cc:513] T 0ad64c699b294679b6fec37b5ad23528 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.482985 28570 leader_election.cc:290] T 0ad64c699b294679b6fec37b5ad23528 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:25.484038 28396 raft_consensus.cc:491] T 8605bc329627438998812ed028327f53 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:25.484215 28396 raft_consensus.cc:513] T 8605bc329627438998812ed028327f53 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.484520 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd6d6bb514134bdf8e3bc0aaad180732" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:25.484735 27526 raft_consensus.cc:3058] T cd6d6bb514134bdf8e3bc0aaad180732 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.484882 28396 leader_election.cc:290] T 8605bc329627438998812ed028327f53 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:25.487649 27183 leader_election.cc:304] T d859c0222c3647dab3dddd105e4cf0fc P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:25.487794 27526 raft_consensus.cc:2466] T cd6d6bb514134bdf8e3bc0aaad180732 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:25.492280 28522 raft_consensus.cc:2802] T d859c0222c3647dab3dddd105e4cf0fc P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:25.493094 28522 raft_consensus.cc:695] T d859c0222c3647dab3dddd105e4cf0fc P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:25.493558 28522 consensus_queue.cc:237] T d859c0222c3647dab3dddd105e4cf0fc P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.495335 27217 tablet_service.cc:1468] Processing CreateTablet for tablet d1e89ad94adb4de8b3738e30a84a8f79 (DEFAULT_TABLE table=default.Table_564e21bb80a94d5888b76f1471d9b937 [id=3941089860a34747be28eae8bdb5c8cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:25.497841 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d859c0222c3647dab3dddd105e4cf0fc" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:25.498062 27380 raft_consensus.cc:3058] T d859c0222c3647dab3dddd105e4cf0fc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.500761 27380 raft_consensus.cc:2466] T d859c0222c3647dab3dddd105e4cf0fc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:25.501724 27221 tablet_service.cc:1468] Processing CreateTablet for tablet 0deb2e04842d48c6a572bba7d3cc8a77 (DEFAULT_TABLE table=default.Table_078beaf0d9304922afad9311ba40775a [id=03858212036244b5bc3d50873db9cf5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:25.520474 28396 raft_consensus.cc:491] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:25.520699 28396 raft_consensus.cc:513] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.521433 28396 leader_election.cc:290] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:25.523350 27241 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd6d6bb514134bdf8e3bc0aaad180732" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:25.523553 27241 raft_consensus.cc:3058] T cd6d6bb514134bdf8e3bc0aaad180732 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.525835 27321 leader_election.cc:304] T 516830fc42e84af693b3af52caee30ee P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:25.526968 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8605bc329627438998812ed028327f53" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:25.527395 27247 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d2d64f103fc44d7b390559b92cbf4b4" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:25.527545 27247 raft_consensus.cc:2466] T 8d2d64f103fc44d7b390559b92cbf4b4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:25.528343 27462 leader_election.cc:304] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:25.528808 28396 raft_consensus.cc:2802] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:25.528895 28396 raft_consensus.cc:491] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:25.529014 28396 raft_consensus.cc:3058] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.527235 27242 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0ad64c699b294679b6fec37b5ad23528" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:25.529953 27242 raft_consensus.cc:2466] T 0ad64c699b294679b6fec37b5ad23528 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:25.530853 27241 raft_consensus.cc:2466] T cd6d6bb514134bdf8e3bc0aaad180732 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:25.531980 27687 ts_tablet_manager.cc:1905] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:25.532773 27244 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2cc35f4244a42eeb4a65de749824c11" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:25.532946 27244 raft_consensus.cc:2466] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:25.527788 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0293a69f1c246d08d414e95729b4ab4" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:25.534394 27240 raft_consensus.cc:2391] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 89f8f0ee68b940db9ce250c0604f934a in this term.
I20250903 20:05:25.535262 27249 raft_consensus.cc:2466] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:25.535290 28396 raft_consensus.cc:513] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.536032 27246 raft_consensus.cc:2466] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:25.536028 28396 leader_election.cc:290] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:25.536684 27321 leader_election.cc:304] T 0ad64c699b294679b6fec37b5ad23528 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:25.537014 27321 leader_election.cc:304] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:25.537262 27321 leader_election.cc:304] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:25.537511 28568 raft_consensus.cc:2802] T 516830fc42e84af693b3af52caee30ee P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:25.537596 28568 raft_consensus.cc:491] T 516830fc42e84af693b3af52caee30ee P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:25.537657 28568 raft_consensus.cc:3058] T 516830fc42e84af693b3af52caee30ee P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.540261 28568 raft_consensus.cc:513] T 516830fc42e84af693b3af52caee30ee P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:25.537660 27321 leader_election.cc:343] T a0293a69f1c246d08d414e95729b4ab4 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:25.541013 27321 leader_election.cc:304] T a0293a69f1c246d08d414e95729b4ab4 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:25.541414 28568 leader_election.cc:290] T 516830fc42e84af693b3af52caee30ee P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:25.541779 28568 raft_consensus.cc:2802] T 0ad64c699b294679b6fec37b5ad23528 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:25.541855 28568 raft_consensus.cc:491] T 0ad64c699b294679b6fec37b5ad23528 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:25.541924 28568 raft_consensus.cc:3058] T 0ad64c699b294679b6fec37b5ad23528 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.543697 27687 ts_tablet_manager.cc:1918] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:25.544030 27687 log.cc:1199] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/4157633525034a019fd6a80185892cec
I20250903 20:05:25.543915 28571 raft_consensus.cc:2802] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:25.544250 28571 raft_consensus.cc:695] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:25.544559 28571 consensus_queue.cc:237] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.545449 28568 raft_consensus.cc:513] T 0ad64c699b294679b6fec37b5ad23528 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.545950 28396 raft_consensus.cc:491] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:25.546240 28396 raft_consensus.cc:513] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.547063 28396 leader_election.cc:290] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:25.547341 28568 leader_election.cc:290] T 0ad64c699b294679b6fec37b5ad23528 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
W20250903 20:05:25.548005 27462 leader_election.cc:343] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: dbfdab4332234b2a93fd3c901a9cb645
I20250903 20:05:25.548098 28568 raft_consensus.cc:2747] T a0293a69f1c246d08d414e95729b4ab4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:25.545780 28569 raft_consensus.cc:2802] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:25.548418 28569 raft_consensus.cc:491] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:25.548560 28569 raft_consensus.cc:3058] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.549203 27687 ts_tablet_manager.cc:1939] T 4157633525034a019fd6a80185892cec P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:25.550757 27246 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d2d64f103fc44d7b390559b92cbf4b4" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:25.550961 27246 raft_consensus.cc:3058] T 8d2d64f103fc44d7b390559b92cbf4b4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.551241 28569 raft_consensus.cc:513] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.551805 28569 leader_election.cc:290] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:25.552114 27687 tablet_replica.cc:331] T 283a3afed7414a958b2dc4176f00bb01 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:25.552524 27687 raft_consensus.cc:2241] T 283a3afed7414a958b2dc4176f00bb01 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:25.553104 27687 raft_consensus.cc:2270] T 283a3afed7414a958b2dc4176f00bb01 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:25.554409 27687 ts_tablet_manager.cc:1905] T 283a3afed7414a958b2dc4176f00bb01 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:25.554821 27462 leader_election.cc:343] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 6f0cba70240f4f38a4fa5ccbcd99b3b1
I20250903 20:05:25.554993 27246 raft_consensus.cc:2466] T 8d2d64f103fc44d7b390559b92cbf4b4 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:25.555693 27462 leader_election.cc:304] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:25.556233 28396 raft_consensus.cc:2802] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:25.556823 28396 raft_consensus.cc:695] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:25.557224 28396 consensus_queue.cc:237] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.562369 28547 raft_consensus.cc:491] T 516830fc42e84af693b3af52caee30ee P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:25.562561 28547 raft_consensus.cc:513] T 516830fc42e84af693b3af52caee30ee P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.563347 28547 leader_election.cc:290] T 516830fc42e84af693b3af52caee30ee P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:25.566354 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "516830fc42e84af693b3af52caee30ee" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:25.566747 27239 raft_consensus.cc:2466] T 516830fc42e84af693b3af52caee30ee P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:25.567564 27462 leader_election.cc:304] T 516830fc42e84af693b3af52caee30ee P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:25.568135 28547 raft_consensus.cc:2802] T 516830fc42e84af693b3af52caee30ee P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:25.568262 28547 raft_consensus.cc:491] T 516830fc42e84af693b3af52caee30ee P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:25.568368 28547 raft_consensus.cc:3058] T 516830fc42e84af693b3af52caee30ee P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.570931 27695 tablet_bootstrap.cc:492] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:25.571087 27695 ts_tablet_manager.cc:1397] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.193s user 0.005s sys 0.001s
I20250903 20:05:25.572086 28547 raft_consensus.cc:513] T 516830fc42e84af693b3af52caee30ee P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.572834 28547 leader_election.cc:290] T 516830fc42e84af693b3af52caee30ee P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:25.573817 27695 raft_consensus.cc:357] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.574079 27695 raft_consensus.cc:383] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:25.574190 27695 raft_consensus.cc:738] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:25.574505 27695 consensus_queue.cc:260] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.575205 27695 ts_tablet_manager.cc:1428] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:25.575523 27695 tablet_bootstrap.cc:492] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:25.575464 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "516830fc42e84af693b3af52caee30ee" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:25.575928 27239 raft_consensus.cc:3058] T 516830fc42e84af693b3af52caee30ee P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.577607 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "516830fc42e84af693b3af52caee30ee" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:25.577889 27526 raft_consensus.cc:2391] T 516830fc42e84af693b3af52caee30ee P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 51578fb7bceb4e119b3ab1e58c4f5528 in current term 1: Already voted for candidate e72ef1dbcc5143cf9f04a808f101c80f in this term.
I20250903 20:05:25.578644 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0ad64c699b294679b6fec37b5ad23528" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:25.579433 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2cc35f4244a42eeb4a65de749824c11" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:25.580374 27692 tablet_bootstrap.cc:492] T 0ad64c699b294679b6fec37b5ad23528 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:25.584028 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0ad64c699b294679b6fec37b5ad23528" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:25.606750 27692 ts_tablet_manager.cc:1397] T 0ad64c699b294679b6fec37b5ad23528 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.255s user 0.001s sys 0.009s
I20250903 20:05:25.607654 27692 raft_consensus.cc:357] T 0ad64c699b294679b6fec37b5ad23528 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.607923 27692 raft_consensus.cc:383] T 0ad64c699b294679b6fec37b5ad23528 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:25.608042 27692 raft_consensus.cc:738] T 0ad64c699b294679b6fec37b5ad23528 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:25.608494 27692 consensus_queue.cc:260] T 0ad64c699b294679b6fec37b5ad23528 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.615119 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb9e90d6e97d4ab4b27bd5efb2ee4b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:25.615798 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0deb2e04842d48c6a572bba7d3cc8a77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:25.615988 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1e89ad94adb4de8b3738e30a84a8f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:25.616123 27692 ts_tablet_manager.cc:1428] T 0ad64c699b294679b6fec37b5ad23528 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20250903 20:05:25.619319 27228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84bd22cd8bd6462982d388222858972a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:25.620414 27692 tablet_bootstrap.cc:492] T 283a3afed7414a958b2dc4176f00bb01 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:25.622439 27684 ts_tablet_manager.cc:1918] T fa05f54ebc324b40855fa3288760e9e9 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:25.624872 27684 log.cc:1199] T fa05f54ebc324b40855fa3288760e9e9 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/fa05f54ebc324b40855fa3288760e9e9
I20250903 20:05:25.625309 27684 ts_tablet_manager.cc:1939] T fa05f54ebc324b40855fa3288760e9e9 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:25.606066 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 3d875d29e66d48ed872e38d20f30d9c0
I20250903 20:05:25.626024 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 766) took 1116 ms. Trace:
I20250903 20:05:25.626235 27684 rpcz_store.cc:276] 0903 20:05:24.509719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:24.538333 (+ 28614us) service_pool.cc:225] Handling call
0903 20:05:24.819550 (+281217us) ts_tablet_manager.cc:1145] Deleting tablet fa05f54ebc324b40855fa3288760e9e9
0903 20:05:24.819558 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:25.162859 (+343301us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:25.622330 (+459471us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:25.626005 (+ 3675us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":280955}
I20250903 20:05:25.626888 27684 tablet_replica.cc:331] T 4be9bdc390de4a74a396003c9c3c2b71 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:25.612460 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 24ddae9179cd4240914cbbae31d8b851
I20250903 20:05:25.627187 27684 raft_consensus.cc:2241] T 4be9bdc390de4a74a396003c9c3c2b71 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:25.627424 27684 raft_consensus.cc:2270] T 4be9bdc390de4a74a396003c9c3c2b71 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:25.634908 27052 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 169622us
I20250903 20:05:25.635339 27684 ts_tablet_manager.cc:1905] T 4be9bdc390de4a74a396003c9c3c2b71 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:25.635342 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet d4c703dafbca4e669dbb18946ffac699 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4c703dafbca4e669dbb18946ffac699 already in progress: creating tablet
W20250903 20:05:25.635793 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 4af8c03c9d834405b4e25d35441db961 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4af8c03c9d834405b4e25d35441db961 already in progress: creating tablet
W20250903 20:05:25.636242 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet a72b85dfb34047528b2b28ff0d40f828 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a72b85dfb34047528b2b28ff0d40f828 already in progress: creating tablet
W20250903 20:05:25.636511 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 41056c5c5003415993bee74d7621bd14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41056c5c5003415993bee74d7621bd14 already in progress: creating tablet
W20250903 20:05:25.638458 27321 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 100794us
I20250903 20:05:25.638799 27321 leader_election.cc:304] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:25.639354 27321 leader_election.cc:304] T 283a3afed7414a958b2dc4176f00bb01 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:25.639740 27321 raft_consensus.cc:2684] T 283a3afed7414a958b2dc4176f00bb01 P 51578fb7bceb4e119b3ab1e58c4f5528: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250903 20:05:25.639583 28567 raft_consensus.cc:2802] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:25.639995 27066 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet d859c0222c3647dab3dddd105e4cf0fc (table default.Table_dc24dd98fc4146eda8660576fd53a29d [id=7add4b636f1b4bffae663fdf432de05f]). Expected version 4 got 0
I20250903 20:05:25.640233 28567 raft_consensus.cc:695] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:25.640774 28567 consensus_queue.cc:237] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.642093 27692 tablet_bootstrap.cc:654] T 283a3afed7414a958b2dc4176f00bb01 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:25.643780 27066 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 912eee5b48ef467bbd6bbe34f8aa390c (table default.Table_f1f4f57d96f04f0caff06d9a642f5c46 [id=fdc7899c8b334d309b834ae1daa14fa0]). Expected version 4 got 0
W20250903 20:05:25.643755 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet a253e3399622409e92299069b19d3698 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a253e3399622409e92299069b19d3698 already in progress: creating tablet
W20250903 20:05:25.646886 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet b7a6d5d8a97a4769a81ec7224d591f8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7a6d5d8a97a4769a81ec7224d591f8c already in progress: creating tablet
I20250903 20:05:25.647042 27059 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 912eee5b48ef467bbd6bbe34f8aa390c (table default.Table_f1f4f57d96f04f0caff06d9a642f5c46 [id=fdc7899c8b334d309b834ae1daa14fa0]). Expected version 4 got 0
I20250903 20:05:25.646108 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b0d2811b8e4a40f4a34d3be594183ac6" } new_table_name: "default.Table_672b60689df64b1d8c9ebd27468d68d4" modify_external_catalogs: true
I20250903 20:05:25.650957 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 45f7068bafaf48d290ded9d1272c0e46
I20250903 20:05:25.656314 27488 tablet_service.cc:1515] Processing DeleteTablet for tablet 24ddae9179cd4240914cbbae31d8b851 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:25.657727 27488 tablet_service.cc:1515] Processing DeleteTablet for tablet cd92e3dafca6419f8eba7a4ef51a911e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:25.657878 27687 ts_tablet_manager.cc:1918] T 283a3afed7414a958b2dc4176f00bb01 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:25.657989 27687 log.cc:1199] T 283a3afed7414a958b2dc4176f00bb01 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/283a3afed7414a958b2dc4176f00bb01
I20250903 20:05:25.658360 27687 ts_tablet_manager.cc:1939] T 283a3afed7414a958b2dc4176f00bb01 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:25.658372 27488 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f0cba70240f4f38a4fa5ccbcd99b3b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3af0c66a8c234711a43816f248e5be03) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:25.658540 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet 3d875d29e66d48ed872e38d20f30d9c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:25.659659 28547 raft_consensus.cc:491] T ac9b5d4f39db4f05a15316585040d142 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:25.659822 28547 raft_consensus.cc:513] T ac9b5d4f39db4f05a15316585040d142 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.660557 28547 leader_election.cc:290] T ac9b5d4f39db4f05a15316585040d142 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:25.663806 27687 tablet_replica.cc:331] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:25.665064 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet 8605bc329627438998812ed028327f53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:25.665531 27685 ts_tablet_manager.cc:1918] T dbfdab4332234b2a93fd3c901a9cb645 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:25.665798 27687 raft_consensus.cc:2241] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:25.666070 27687 pending_rounds.cc:70] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528: Trying to abort 1 pending ops.
I20250903 20:05:25.666154 27687 pending_rounds.cc:77] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384974358003712 op_type: NO_OP noop_request { }
I20250903 20:05:25.666323 27687 raft_consensus.cc:2887] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:25.666452 27687 raft_consensus.cc:2270] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:25.667526 27692 tablet_bootstrap.cc:492] T 283a3afed7414a958b2dc4176f00bb01 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:25.667716 27692 ts_tablet_manager.cc:1397] T 283a3afed7414a958b2dc4176f00bb01 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20250903 20:05:25.667948 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet c6d67cd0c0304c77ba571b414878dad2 with delete_type TABLET_DATA_DELETED (replaced by tablet 886b8783aa9d4618aad498d4a1115843) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:25.668509 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet 45d9b6ff520348628d19671127b9426c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:25.668704 27692 raft_consensus.cc:357] T 283a3afed7414a958b2dc4176f00bb01 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.668895 27692 raft_consensus.cc:383] T 283a3afed7414a958b2dc4176f00bb01 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:25.668977 27692 raft_consensus.cc:738] T 283a3afed7414a958b2dc4176f00bb01 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:25.669252 27692 consensus_queue.cc:260] T 283a3afed7414a958b2dc4176f00bb01 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.669799 27692 ts_tablet_manager.cc:1428] T 283a3afed7414a958b2dc4176f00bb01 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:25.669970 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet ac9b5d4f39db4f05a15316585040d142 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:25.670317 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "516830fc42e84af693b3af52caee30ee" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:25.670442 27687 ts_tablet_manager.cc:1905] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:25.670507 27381 raft_consensus.cc:2391] T 516830fc42e84af693b3af52caee30ee P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
I20250903 20:05:25.671319 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "516830fc42e84af693b3af52caee30ee" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:25.671491 27381 raft_consensus.cc:2391] T 516830fc42e84af693b3af52caee30ee P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
I20250903 20:05:25.672405 27692 tablet_bootstrap.cc:492] T 7cc6cf579e2642e99fefed723b0ee900 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:25.673913 27692 tablet_bootstrap.cc:654] T 7cc6cf579e2642e99fefed723b0ee900 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:25.677182 27320 leader_election.cc:343] T 0ad64c699b294679b6fec37b5ad23528 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:25.679008 27685 log.cc:1199] T dbfdab4332234b2a93fd3c901a9cb645 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/dbfdab4332234b2a93fd3c901a9cb645
I20250903 20:05:25.678872 27684 ts_tablet_manager.cc:1918] T 4be9bdc390de4a74a396003c9c3c2b71 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:25.679968 27684 log.cc:1199] T 4be9bdc390de4a74a396003c9c3c2b71 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/4be9bdc390de4a74a396003c9c3c2b71
I20250903 20:05:25.680421 27684 ts_tablet_manager.cc:1939] T 4be9bdc390de4a74a396003c9c3c2b71 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:25.680857 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 765) took 1171 ms. Trace:
I20250903 20:05:25.680989 27684 rpcz_store.cc:276] 0903 20:05:24.509611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:24.541054 (+ 31443us) service_pool.cc:225] Handling call
0903 20:05:25.626845 (+1085791us) ts_tablet_manager.cc:1145] Deleting tablet 4be9bdc390de4a74a396003c9c3c2b71
0903 20:05:25.626853 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:25.659573 (+ 32720us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:25.678767 (+ 19194us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:25.680841 (+ 2074us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1085553}
I20250903 20:05:25.681514 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 769) took 1122 ms. Trace:
I20250903 20:05:25.684144 27684 rpcz_store.cc:276] 0903 20:05:24.559522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:24.567040 (+ 7518us) service_pool.cc:225] Handling call
0903 20:05:25.681392 (+1114352us) ts_tablet_manager.cc:1145] Deleting tablet 15072aa10c5540a09f3ec4592e377d2a
0903 20:05:25.681402 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:25.681497 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1113863}
W20250903 20:05:25.683674 27462 leader_election.cc:343] T 8605bc329627438998812ed028327f53 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 8605bc329627438998812ed028327f53
I20250903 20:05:25.684631 27462 leader_election.cc:304] T 8605bc329627438998812ed028327f53 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
W20250903 20:05:25.684775 27320 leader_election.cc:343] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:25.684986 28396 raft_consensus.cc:3058] T 8605bc329627438998812ed028327f53 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.682904 27246 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac9b5d4f39db4f05a15316585040d142" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:25.687839 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 5b26ed39f2144610b786f49e62e6f0a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b26ed39f2144610b786f49e62e6f0a3 already in progress: creating tablet
W20250903 20:05:25.688176 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 2c772ab7c50c4994aacde5a05bb0f08e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c772ab7c50c4994aacde5a05bb0f08e already in progress: creating tablet
I20250903 20:05:25.688573 27685 ts_tablet_manager.cc:1939] T dbfdab4332234b2a93fd3c901a9cb645 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:25.688596 28396 raft_consensus.cc:2747] T 8605bc329627438998812ed028327f53 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:25.689018 27462 leader_election.cc:343] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 8d2d64f103fc44d7b390559b92cbf4b4
I20250903 20:05:25.688968 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 630) took 3859 ms. Trace:
I20250903 20:05:25.689128 27685 rpcz_store.cc:276] 0903 20:05:21.829763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.829824 (+ 61us) service_pool.cc:225] Handling call
0903 20:05:25.085444 (+3255620us) ts_tablet_manager.cc:1145] Deleting tablet dbfdab4332234b2a93fd3c901a9cb645
0903 20:05:25.085455 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:25.409406 (+323951us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:25.665410 (+256004us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:25.688949 (+ 23539us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3228492}
W20250903 20:05:25.689373 27462 leader_election.cc:343] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 8d2d64f103fc44d7b390559b92cbf4b4
I20250903 20:05:25.689535 27685 tablet_replica.cc:331] T cd92e3dafca6419f8eba7a4ef51a911e P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:25.689725 27685 raft_consensus.cc:2241] T cd92e3dafca6419f8eba7a4ef51a911e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
W20250903 20:05:25.689738 27462 leader_election.cc:343] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 6f0cba70240f4f38a4fa5ccbcd99b3b1
I20250903 20:05:25.689896 27685 raft_consensus.cc:2270] T cd92e3dafca6419f8eba7a4ef51a911e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:25.689908 27462 leader_election.cc:304] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:25.690479 27685 ts_tablet_manager.cc:1905] T cd92e3dafca6419f8eba7a4ef51a911e P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:25.690687 28396 raft_consensus.cc:2747] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:25.690917 27684 tablet_replica.cc:331] T 7c398634d6304a12b35c0579fba4c813 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:25.691104 27684 raft_consensus.cc:2241] T 7c398634d6304a12b35c0579fba4c813 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:25.691304 27684 raft_consensus.cc:2270] T 7c398634d6304a12b35c0579fba4c813 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:25.691807 27684 ts_tablet_manager.cc:1905] T 7c398634d6304a12b35c0579fba4c813 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:25.692166 27063 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:45936 (request call id 952) took 1069 ms. Trace:
W20250903 20:05:25.694353 27053 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 18231d53e9db46059da678f3f8b67a52 (table default.Table_5c8743c0e4e444fb8eac65cde6000a06 [id=fe24472bb8e74884b9bf90ccdefd6d89], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 18231d53e9db46059da678f3f8b67a52 consensus state has no leader
W20250903 20:05:25.696811 27320 leader_election.cc:343] T 0ad64c699b294679b6fec37b5ad23528 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:25.703510 27692 tablet_bootstrap.cc:492] T 7cc6cf579e2642e99fefed723b0ee900 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:25.703737 27692 ts_tablet_manager.cc:1397] T 7cc6cf579e2642e99fefed723b0ee900 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.002s
I20250903 20:05:25.704814 27692 raft_consensus.cc:357] T 7cc6cf579e2642e99fefed723b0ee900 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.705070 27692 raft_consensus.cc:383] T 7cc6cf579e2642e99fefed723b0ee900 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:25.705164 27692 raft_consensus.cc:738] T 7cc6cf579e2642e99fefed723b0ee900 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:25.705420 27692 consensus_queue.cc:260] T 7cc6cf579e2642e99fefed723b0ee900 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:25.706279 27462 leader_election.cc:343] T ac9b5d4f39db4f05a15316585040d142 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: ac9b5d4f39db4f05a15316585040d142
I20250903 20:05:25.708705 27692 ts_tablet_manager.cc:1428] T 7cc6cf579e2642e99fefed723b0ee900 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250903 20:05:25.709430 27687 ts_tablet_manager.cc:1918] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:25.709542 27687 log.cc:1199] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/45d9b6ff520348628d19671127b9426c
I20250903 20:05:25.709900 27687 ts_tablet_manager.cc:1939] T 45d9b6ff520348628d19671127b9426c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
W20250903 20:05:25.685268 27308 kernel_stack_watchdog.cc:198] Thread 27693 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 371ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7efcc2d19980 (unknown) at ??:0
@ 0x92cb07 __sanitizer::internal_sched_yield() at ??:?
@ 0x8921bd __interceptor_pthread_create at ??:?
@ 0x7efcbf8e8fb1 kudu::Thread::StartThread() at ??:0
@ 0xa63f97 kudu::Thread::Create() at ??:?
@ 0x7efcbf908dfb kudu::ThreadPool::CreateThread() at ??:0
@ 0x7efcbf90481d kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7efcbf909b4f kudu::ThreadPool::Submit() at ??:0
@ 0x7efccacb688d kudu::log::SegmentAllocator::AsyncAllocateSegmentUnlocked() at ??:0
@ 0x7efccacb587f kudu::log::SegmentAllocator::AllocateSegmentAndRollOver() at ??:0
@ 0x7efccacb543e kudu::log::SegmentAllocator::Init() at ??:0
@ 0x7efccacbd4b5 kudu::log::Log::Init() at ??:0
@ 0x7efccacbc811 kudu::log::Log::Open() at ??:0
@ 0x7efccc6005e1 kudu::tablet::TabletBootstrap::OpenNewLog() at ??:0
@ 0x7efccc5fc94c kudu::tablet::TabletBootstrap::RunBootstrap() at ??:0
@ 0x7efccc5fb0cf kudu::tablet::TabletBootstrap::Bootstrap() at ??:0
W20250903 20:05:25.685343 27693 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.567s user 0.158s sys 0.218s
I20250903 20:05:25.677393 28579 raft_consensus.cc:491] T c9e90abef9814338b70a3550ab5686e0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250903 20:05:25.710636 27693 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.592s user 0.158s sys 0.218s
I20250903 20:05:25.710691 28579 raft_consensus.cc:513] T c9e90abef9814338b70a3550ab5686e0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.711036 27687 tablet_replica.cc:331] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:25.711212 27687 raft_consensus.cc:2241] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:25.711354 28579 leader_election.cc:290] T c9e90abef9814338b70a3550ab5686e0 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:25.711408 27687 pending_rounds.cc:70] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528: Trying to abort 1 pending ops.
I20250903 20:05:25.711445 27687 pending_rounds.cc:77] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384975431921664 op_type: NO_OP noop_request { }
I20250903 20:05:25.711530 27693 tablet_bootstrap.cc:492] T 1bcb9d5f010f4297bab78c0183889238 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:25.711658 27693 ts_tablet_manager.cc:1397] T 1bcb9d5f010f4297bab78c0183889238 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.596s user 0.161s sys 0.219s
I20250903 20:05:25.712318 27693 raft_consensus.cc:357] T 1bcb9d5f010f4297bab78c0183889238 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.712468 27693 raft_consensus.cc:383] T 1bcb9d5f010f4297bab78c0183889238 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:25.712539 27693 raft_consensus.cc:738] T 1bcb9d5f010f4297bab78c0183889238 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:25.712754 27693 consensus_queue.cc:260] T 1bcb9d5f010f4297bab78c0183889238 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.711542 27687 raft_consensus.cc:2887] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:25.713099 27687 raft_consensus.cc:2270] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:25.713172 27693 ts_tablet_manager.cc:1428] T 1bcb9d5f010f4297bab78c0183889238 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250903 20:05:25.713835 27687 ts_tablet_manager.cc:1905] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:25.715664 27693 tablet_bootstrap.cc:492] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:25.717226 27693 tablet_bootstrap.cc:654] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:25.719558 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9e90abef9814338b70a3550ab5686e0" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:25.720252 27692 tablet_bootstrap.cc:492] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:25.721403 28579 raft_consensus.cc:491] T 1bcb9d5f010f4297bab78c0183889238 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:25.721585 27692 tablet_bootstrap.cc:654] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:25.721549 28579 raft_consensus.cc:513] T 1bcb9d5f010f4297bab78c0183889238 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.722122 28579 leader_election.cc:290] T 1bcb9d5f010f4297bab78c0183889238 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:25.722806 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bcb9d5f010f4297bab78c0183889238" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:25.723116 27320 leader_election.cc:343] T c9e90abef9814338b70a3550ab5686e0 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:25.723479 27320 leader_election.cc:343] T 1bcb9d5f010f4297bab78c0183889238 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:25.725781 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dbfdab4332234b2a93fd3c901a9cb645" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:25.726006 27246 raft_consensus.cc:2391] T ac9b5d4f39db4f05a15316585040d142 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
I20250903 20:05:25.731110 27685 ts_tablet_manager.cc:1918] T cd92e3dafca6419f8eba7a4ef51a911e P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:25.731276 27685 log.cc:1199] T cd92e3dafca6419f8eba7a4ef51a911e P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/cd92e3dafca6419f8eba7a4ef51a911e
I20250903 20:05:25.734830 27059 master_service.cc:948] ReplaceTablet: received request to replace tablet d93258c032524b71997389b81846a5be from {username='slave'} at 127.0.0.1:45936
W20250903 20:05:25.736618 27462 leader_election.cc:343] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: Tablet not RUNNING: SHUTDOWN
I20250903 20:05:25.736738 27462 leader_election.cc:304] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:25.736989 28396 raft_consensus.cc:2747] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:25.737399 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet f71d3dd9af2543299296f21381d1d563 (table default.Table_a0582ccb94a64603a5fbdd0e531a9edf [id=6f7fece7a8d54d4786f1bab10da9129c]) successfully deleted
I20250903 20:05:25.694320 27063 rpcz_store.cc:276] 0903 20:05:24.618511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:24.672988 (+ 54477us) service_pool.cc:225] Handling call
0903 20:05:24.711427 (+ 38439us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0903 20:05:24.711585 (+ 158us) catalog_manager.cc:2790] Modifying in-memory table state
0903 20:05:24.711665 (+ 80us) catalog_manager.cc:2799] Locking tablets
0903 20:05:25.010457 (+298792us) catalog_manager.cc:2813] Removing table and tablets from system table
0903 20:05:25.650575 (+640118us) catalog_manager.cc:2831] Removing table from by-name map
0903 20:05:25.650841 (+ 266us) catalog_manager.cc:2847] Committing in-memory state
0903 20:05:25.650909 (+ 68us) catalog_manager.cc:2851] Aborting table tasks
0903 20:05:25.688178 (+ 37269us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:25.641494 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:25.641633 (+ 139us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:25.641639 (+ 6us) write_op.cc:435] Acquired schema lock
0903 20:05:25.641640 (+ 1us) tablet.cc:660] Decoding operations
0903 20:05:25.641776 (+ 136us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:25.641790 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:25.641795 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0903 20:05:25.641913 (+ 118us) tablet.cc:699] Row locks acquired
0903 20:05:25.641917 (+ 4us) write_op.cc:260] PREPARE: finished
0903 20:05:25.641983 (+ 66us) write_op.cc:270] Start()
0903 20:05:25.642025 (+ 42us) write_op.cc:276] Timestamp: P: 1756929925641929 usec, L: 0
0903 20:05:25.642029 (+ 4us) op_driver.cc:348] REPLICATION: starting
0903 20:05:25.642286 (+ 257us) log.cc:844] Serialized 759 byte log entry
0903 20:05:25.649686 (+ 7400us) op_driver.cc:464] REPLICATION: finished
0903 20:05:25.649851 (+ 165us) write_op.cc:301] APPLY: starting
0903 20:05:25.649866 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:25.649911 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:25.649947 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:25.650006 (+ 59us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:25.650037 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0903 20:05:25.650044 (+ 7us) write_op.cc:312] APPLY: finished
0903 20:05:25.650110 (+ 66us) log.cc:844] Serialized 29 byte log entry
0903 20:05:25.650161 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:25.650179 (+ 18us) write_op.cc:454] Released schema lock
0903 20:05:25.650186 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0903 20:05:25.661479 (+ 11293us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:25.661488 (+ 9us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":125,"apply.run_cpu_time_us":424,"apply.run_wall_time_us":423,"num_ops":2,"prepare.queue_time_us":630441,"prepare.run_cpu_time_us":938,"prepare.run_wall_time_us":19969,"raft.queue_time_us":33,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":225,"replication_time_us":7631}]]}
I20250903 20:05:25.737418 27462 leader_election.cc:304] T ac9b5d4f39db4f05a15316585040d142 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:25.740842 28396 raft_consensus.cc:3058] T ac9b5d4f39db4f05a15316585040d142 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:25.742326 27052 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 48911.8us
I20250903 20:05:25.743355 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet dbfdab4332234b2a93fd3c901a9cb645 (table default.Table_7e582ad4ca9446c68da4850af02126ea [id=67cd1b33fab14606babd6ba4524a6966]) successfully deleted
I20250903 20:05:25.743691 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 8d2d64f103fc44d7b390559b92cbf4b4 (table default.Table_bb8c68be79034a5c8469862c43953131 [id=f0171ee670294ce58ab84483fd71ab7e]) successfully deleted
I20250903 20:05:25.743903 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet c6d67cd0c0304c77ba571b414878dad2 (table default.Table_a540eea45dfb46ec92b02a3f9a2b3aed [id=5fe055478b724744a65fcaa2f25bef0d]) successfully deleted
I20250903 20:05:25.744077 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet fe902f1b58924be1ac3a5585413f7789 (table default.Table_1840897b7e7f4f1e8052d16ca5e77a4f [id=78785eeb2c464e6d9f0cc5c8792376ec]) successfully deleted
I20250903 20:05:25.744364 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet dbfdab4332234b2a93fd3c901a9cb645 (table default.Table_7e582ad4ca9446c68da4850af02126ea [id=67cd1b33fab14606babd6ba4524a6966]) successfully deleted
I20250903 20:05:25.745476 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet e78e6d212daf4589a6b4b65711e356cb (table default.Table_7f07933eeaf8443384edb1e59396e36e [id=6b3f6e948024464fa28fe985534395ff]) successfully deleted
I20250903 20:05:25.745994 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 4157633525034a019fd6a80185892cec (table default.Table_57cd5123dc7042c1887af6e833af4d7a [id=4b42213db8534ac3b3b97318f5a38e61]) successfully deleted
I20250903 20:05:25.746281 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 283a3afed7414a958b2dc4176f00bb01 (table default.Table_37d794e122f2457294c1eb6115be9472 [id=2ed5c84cc5a3466ba8a8cfeda62b643e]) successfully deleted
I20250903 20:05:25.746939 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 45d9b6ff520348628d19671127b9426c (table default.Table_509afdab983d4eed97a5d124ff19ab3b [id=7a023e4098354d038d92f90d46fc5125]) successfully deleted
I20250903 20:05:25.747361 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "516830fc42e84af693b3af52caee30ee" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:25.747771 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet f2cc35f4244a42eeb4a65de749824c11 with delete_type TABLET_DATA_DELETED (replaced by tablet fc634116a78f45dd833034ddda90e9bf) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:25.748625 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 005d7bc2c7534f288caec8326fdf2416 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:25.748917 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet 005d7bc2c7534f288caec8326fdf2416 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:25.749482 27685 ts_tablet_manager.cc:1939] T cd92e3dafca6419f8eba7a4ef51a911e P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:25.749505 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet c9e90abef9814338b70a3550ab5686e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 607c9a9cbc5540e5bd0fa436fcba564c) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:25.749850 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet f2cc35f4244a42eeb4a65de749824c11 with delete_type TABLET_DATA_DELETED (replaced by tablet fc634116a78f45dd833034ddda90e9bf) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:25.750424 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet a0293a69f1c246d08d414e95729b4ab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:25.750988 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 3d875d29e66d48ed872e38d20f30d9c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:25.751226 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0ad64c699b294679b6fec37b5ad23528" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:25.751392 27249 raft_consensus.cc:3058] T 0ad64c699b294679b6fec37b5ad23528 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.751502 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet 3d875d29e66d48ed872e38d20f30d9c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:25.751696 27246 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2cc35f4244a42eeb4a65de749824c11" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:25.751834 27246 raft_consensus.cc:3058] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.752044 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 24ddae9179cd4240914cbbae31d8b851 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:25.752784 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 45f7068bafaf48d290ded9d1272c0e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:25.753679 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet fa05f54ebc324b40855fa3288760e9e9 (table default.Table_dacd8238c8334c1c9f66e745f5036a06 [id=7ec413d945394dcfa36492f7b59aaaa2]) successfully deleted
I20250903 20:05:25.754294 27695 tablet_bootstrap.cc:654] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:25.755227 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_78426e4139c24ecd9133e4564ed77d31" } new_table_name: "default.Table_f752a52fa3ba4ac99bc29428fa871318" modify_external_catalogs: true
I20250903 20:05:25.756615 27684 ts_tablet_manager.cc:1918] T 7c398634d6304a12b35c0579fba4c813 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:25.756721 27684 log.cc:1199] T 7c398634d6304a12b35c0579fba4c813 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/7c398634d6304a12b35c0579fba4c813
I20250903 20:05:25.757026 27684 ts_tablet_manager.cc:1939] T 7c398634d6304a12b35c0579fba4c813 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:25.757246 27239 raft_consensus.cc:2466] T 516830fc42e84af693b3af52caee30ee P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:25.760627 27687 ts_tablet_manager.cc:1918] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:25.760882 28547 raft_consensus.cc:491] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:25.760993 28547 raft_consensus.cc:513] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.761199 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet b6663ba96faa44b88fe8b6def4ff6951
I20250903 20:05:25.761857 28547 leader_election.cc:290] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:25.760748 27687 log.cc:1199] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/7cc6cf579e2642e99fefed723b0ee900
I20250903 20:05:25.762415 27687 ts_tablet_manager.cc:1939] T 7cc6cf579e2642e99fefed723b0ee900 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:25.763501 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet 24ddae9179cd4240914cbbae31d8b851 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:25.763311 27061 catalog_manager.cc:5582] T d859c0222c3647dab3dddd105e4cf0fc P 89f8f0ee68b940db9ce250c0604f934a reported cstate change: term changed from 0 to 1, leader changed from <none> to 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65). New cstate: current_term: 1 leader_uuid: "89f8f0ee68b940db9ce250c0604f934a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:25.763749 27061 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet d859c0222c3647dab3dddd105e4cf0fc (table default.Table_dc24dd98fc4146eda8660576fd53a29d [id=7add4b636f1b4bffae663fdf432de05f]). Expected version 4 got 0
I20250903 20:05:25.764139 27211 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56534 (request call id 727) took 1552 ms. Trace:
I20250903 20:05:25.764214 27211 rpcz_store.cc:276] 0903 20:05:24.211282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:24.215717 (+ 4435us) service_pool.cc:225] Handling call
0903 20:05:24.291909 (+ 76192us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0903 20:05:24.291950 (+ 41us) ts_tablet_manager.cc:676] Creating new metadata...
0903 20:05:25.704016 (+1412066us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:25.758002 (+ 53986us) tablet_replica.cc:206] Creating consensus instance
0903 20:05:25.764122 (+ 6120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250903 20:05:25.765831 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 764) took 1256 ms. Trace:
I20250903 20:05:25.765908 27684 rpcz_store.cc:276] 0903 20:05:24.509496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:24.567567 (+ 58071us) service_pool.cc:225] Handling call
0903 20:05:25.684518 (+1116951us) ts_tablet_manager.cc:1145] Deleting tablet 7c398634d6304a12b35c0579fba4c813
0903 20:05:25.690873 (+ 6355us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:25.711937 (+ 21064us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:25.756501 (+ 44564us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:25.765810 (+ 9309us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1116618}
I20250903 20:05:25.766359 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 767) took 1256 ms. Trace:
I20250903 20:05:25.766426 27684 rpcz_store.cc:276] 0903 20:05:24.509854 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:24.567344 (+ 57490us) service_pool.cc:225] Handling call
0903 20:05:25.766178 (+1198834us) ts_tablet_manager.cc:1145] Deleting tablet 849eb453d3e04b16bbb5df9017bfb2fd
0903 20:05:25.766189 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:25.766343 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1198156}
I20250903 20:05:25.766722 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 763) took 1257 ms. Trace:
I20250903 20:05:25.766783 27684 rpcz_store.cc:276] 0903 20:05:24.509362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:24.567742 (+ 58380us) service_pool.cc:225] Handling call
0903 20:05:25.766627 (+1198885us) ts_tablet_manager.cc:1145] Deleting tablet 0060e6c4699a4bff8faddefb4ee50add
0903 20:05:25.766635 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:25.766707 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1198513}
I20250903 20:05:25.766992 27684 tablet_replica.cc:331] T 885fff47fea14b84ac6b34a57b827b4b P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:25.767156 27684 raft_consensus.cc:2241] T 885fff47fea14b84ac6b34a57b827b4b P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:25.767305 27684 raft_consensus.cc:2270] T 885fff47fea14b84ac6b34a57b827b4b P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:25.767798 27684 ts_tablet_manager.cc:1905] T 885fff47fea14b84ac6b34a57b827b4b P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:25.772074 28396 raft_consensus.cc:2747] T ac9b5d4f39db4f05a15316585040d142 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:25.772938 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e78e6d212daf4589a6b4b65711e356cb" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:25.773119 27239 raft_consensus.cc:2466] T e78e6d212daf4589a6b4b65711e356cb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:25.773378 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9e90abef9814338b70a3550ab5686e0" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:25.775498 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bcb9d5f010f4297bab78c0183889238" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:25.775637 27321 leader_election.cc:304] T 516830fc42e84af693b3af52caee30ee P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:25.776091 27321 leader_election.cc:343] T c9e90abef9814338b70a3550ab5686e0 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:25.776221 27321 leader_election.cc:304] T c9e90abef9814338b70a3550ab5686e0 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:25.776420 28567 raft_consensus.cc:2747] T 516830fc42e84af693b3af52caee30ee P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20250903 20:05:25.776535 27321 leader_election.cc:343] T 1bcb9d5f010f4297bab78c0183889238 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:25.776625 28567 raft_consensus.cc:2747] T c9e90abef9814338b70a3550ab5686e0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:25.776650 27321 leader_election.cc:304] T 1bcb9d5f010f4297bab78c0183889238 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:25.776814 28567 raft_consensus.cc:2747] T 1bcb9d5f010f4297bab78c0183889238 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:25.777690 28522 raft_consensus.cc:491] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:25.777796 28522 raft_consensus.cc:513] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.780895 28522 leader_election.cc:290] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:25.781632 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "514e382479014fa7bf1f404efff11c86" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:25.782039 27382 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "514e382479014fa7bf1f404efff11c86" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:25.783263 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 4be9bdc390de4a74a396003c9c3c2b71 (table default.Table_452fdb47678d4520828b9a3db6ddb237 [id=d945a830c86244bca66cb2652e64966d]) successfully deleted
W20250903 20:05:25.783306 27183 leader_election.cc:343] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:25.783525 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 15072aa10c5540a09f3ec4592e377d2a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15072aa10c5540a09f3ec4592e377d2a already in progress: creating tablet
I20250903 20:05:25.783804 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 7c398634d6304a12b35c0579fba4c813 (table default.Table_14af7742726b4562a02fd2c9b2895277 [id=6d72901e5102476890f466fd406a7127]) successfully deleted
W20250903 20:05:25.783970 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 849eb453d3e04b16bbb5df9017bfb2fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 849eb453d3e04b16bbb5df9017bfb2fd already in progress: creating tablet
W20250903 20:05:25.784166 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 0060e6c4699a4bff8faddefb4ee50add because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0060e6c4699a4bff8faddefb4ee50add already in progress: creating tablet
I20250903 20:05:25.784991 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet 45f7068bafaf48d290ded9d1272c0e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:25.785117 27184 leader_election.cc:343] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:25.785197 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet b6663ba96faa44b88fe8b6def4ff6951 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:25.785219 27184 leader_election.cc:304] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:25.785583 28522 raft_consensus.cc:2747] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:25.790594 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d0b3804ced547f3bd482399cbaa9270 consensus state has no leader
I20250903 20:05:25.792994 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_791801575c0e4c98b42111d7a3254236" } new_table_name: "default.Table_f9dfd80fc96d427a9aba1d7f0c4cd594" modify_external_catalogs: true
W20250903 20:05:25.796149 27462 leader_election.cc:343] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: e78e6d212daf4589a6b4b65711e356cb
I20250903 20:05:25.796366 27462 leader_election.cc:304] T 516830fc42e84af693b3af52caee30ee P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:25.796769 28547 raft_consensus.cc:2802] T 516830fc42e84af693b3af52caee30ee P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:25.796878 28547 raft_consensus.cc:695] T 516830fc42e84af693b3af52caee30ee P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:25.797116 28547 consensus_queue.cc:237] T 516830fc42e84af693b3af52caee30ee P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.798596 27687 tablet_replica.cc:331] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:25.798895 27687 raft_consensus.cc:2241] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:25.799176 27687 pending_rounds.cc:70] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528: Trying to abort 1 pending ops.
I20250903 20:05:25.799279 27687 pending_rounds.cc:77] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384975033880576 op_type: NO_OP noop_request { }
I20250903 20:05:25.799398 27687 raft_consensus.cc:2887] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:25.799526 27687 raft_consensus.cc:2270] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:25.800323 27687 ts_tablet_manager.cc:1905] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:25.802760 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_5e14429fe4e340aaa9c2d5ad605c9a92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:25.803354 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e14429fe4e340aaa9c2d5ad605c9a92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:25.803591 27211 tablet_service.cc:1515] Processing DeleteTablet for tablet 45f7068bafaf48d290ded9d1272c0e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:25.806196 27693 tablet_bootstrap.cc:492] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:25.806394 27693 ts_tablet_manager.cc:1397] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.091s user 0.000s sys 0.025s
I20250903 20:05:25.807147 27693 raft_consensus.cc:357] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.807333 27693 raft_consensus.cc:383] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:25.807394 27693 raft_consensus.cc:738] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:25.810585 27693 consensus_queue.cc:260] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.811089 27693 ts_tablet_manager.cc:1428] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20250903 20:05:25.811396 27693 tablet_bootstrap.cc:492] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:25.813100 27693 tablet_bootstrap.cc:654] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:25.813997 27462 leader_election.cc:304] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:25.814325 28547 raft_consensus.cc:2802] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:25.814402 28547 raft_consensus.cc:491] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:25.814469 28547 raft_consensus.cc:3058] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.817498 28396 raft_consensus.cc:491] T cd92e3dafca6419f8eba7a4ef51a911e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:25.817629 28396 raft_consensus.cc:513] T cd92e3dafca6419f8eba7a4ef51a911e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.818173 28396 leader_election.cc:290] T cd92e3dafca6419f8eba7a4ef51a911e P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:25.819592 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd92e3dafca6419f8eba7a4ef51a911e" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:25.820094 27462 leader_election.cc:343] T cd92e3dafca6419f8eba7a4ef51a911e P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: Tablet not RUNNING: STOPPED
W20250903 20:05:25.820297 27462 leader_election.cc:343] T cd92e3dafca6419f8eba7a4ef51a911e P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: cd92e3dafca6419f8eba7a4ef51a911e
I20250903 20:05:25.820381 27462 leader_election.cc:304] T cd92e3dafca6419f8eba7a4ef51a911e P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:25.823936 28396 raft_consensus.cc:2747] T cd92e3dafca6419f8eba7a4ef51a911e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:25.827577 27693 tablet_bootstrap.cc:492] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:25.827731 27693 ts_tablet_manager.cc:1397] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.004s
I20250903 20:05:25.827847 27695 tablet_bootstrap.cc:492] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:25.827993 27695 ts_tablet_manager.cc:1397] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.253s user 0.003s sys 0.004s
I20250903 20:05:25.828552 27693 raft_consensus.cc:357] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.828723 27695 raft_consensus.cc:357] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.829033 27695 raft_consensus.cc:383] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:25.829085 27695 raft_consensus.cc:738] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:25.829310 27695 consensus_queue.cc:260] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.830076 27695 ts_tablet_manager.cc:1428] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:25.830180 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 645) took 3850 ms. Trace:
I20250903 20:05:25.830355 27695 tablet_bootstrap.cc:492] T c9e90abef9814338b70a3550ab5686e0 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:25.828745 27693 raft_consensus.cc:383] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:25.830564 27693 raft_consensus.cc:738] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:25.830794 27693 consensus_queue.cc:260] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:25.830284 27685 rpcz_store.cc:276] 0903 20:05:21.980185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.980240 (+ 55us) service_pool.cc:225] Handling call
0903 20:05:25.689486 (+3709246us) ts_tablet_manager.cc:1145] Deleting tablet cd92e3dafca6419f8eba7a4ef51a911e
0903 20:05:25.689495 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:25.694896 (+ 5401us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:25.730878 (+ 35982us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:25.830165 (+ 99287us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3708730}
I20250903 20:05:25.832211 27693 ts_tablet_manager.cc:1428] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250903 20:05:25.832419 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 647) took 3851 ms. Trace:
I20250903 20:05:25.832499 27693 tablet_bootstrap.cc:492] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:25.832492 27685 rpcz_store.cc:276] 0903 20:05:21.980455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.980496 (+ 41us) service_pool.cc:225] Handling call
0903 20:05:25.832286 (+3851790us) ts_tablet_manager.cc:1145] Deleting tablet c49767d618be4fd18f30e5124ce0cb84
0903 20:05:25.832295 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:25.832401 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3851412}
I20250903 20:05:25.832924 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 643) took 3853 ms. Trace:
I20250903 20:05:25.832990 27685 rpcz_store.cc:276] 0903 20:05:21.979655 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.979844 (+ 189us) service_pool.cc:225] Handling call
0903 20:05:25.832820 (+3852976us) ts_tablet_manager.cc:1145] Deleting tablet 36e1061915574407b6e7695be8b1e55a
0903 20:05:25.832829 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:25.832910 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3851846}
I20250903 20:05:25.833400 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 644) took 3853 ms. Trace:
I20250903 20:05:25.833469 27685 rpcz_store.cc:276] 0903 20:05:21.979772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.981611 (+ 1839us) service_pool.cc:225] Handling call
0903 20:05:25.833306 (+3851695us) ts_tablet_manager.cc:1145] Deleting tablet 17e7deb74389448db37289563b86c253
0903 20:05:25.833314 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:25.833385 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3851533}
I20250903 20:05:25.833864 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 646) took 3853 ms. Trace:
I20250903 20:05:25.833926 27685 rpcz_store.cc:276] 0903 20:05:21.980312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.981799 (+ 1487us) service_pool.cc:225] Handling call
0903 20:05:25.833768 (+3851969us) ts_tablet_manager.cc:1145] Deleting tablet e9126e4a18c546c28927f22c9f16426e
0903 20:05:25.833778 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:25.833850 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3850036}
I20250903 20:05:25.834146 27693 tablet_bootstrap.cc:654] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:25.834275 27685 tablet_replica.cc:331] T 4157633525034a019fd6a80185892cec P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:25.834429 27685 raft_consensus.cc:2241] T 4157633525034a019fd6a80185892cec P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:25.834553 27685 raft_consensus.cc:2270] T 4157633525034a019fd6a80185892cec P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:25.834991 27685 ts_tablet_manager.cc:1905] T 4157633525034a019fd6a80185892cec P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:25.846282 27249 raft_consensus.cc:2466] T 0ad64c699b294679b6fec37b5ad23528 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:25.854393 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_585c269abe9b4aa784813fd7439fa4cc" } new_table_name: "default.Table_65ca0d048e8f4941b28537ed656d05cb" modify_external_catalogs: true
I20250903 20:05:25.869838 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 7cc6cf579e2642e99fefed723b0ee900 (table default.Table_bffb6801f48c4124bfe24c49586d6e48 [id=aa020b54da65482ba59a71768c2bd03b]) successfully deleted
I20250903 20:05:25.902622 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_de11ae4b3b9b42c7b6e256023fb6a3a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:25.908298 27066 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5557dfdd953a41e9bd3a8be4cd689604 consensus state has no leader
W20250903 20:05:26.034380 27052 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 164511us
I20250903 20:05:26.034477 27066 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:45936 (request call id 978) took 1048 ms. Trace:
I20250903 20:05:25.847005 27321 leader_election.cc:304] T 0ad64c699b294679b6fec37b5ad23528 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:26.034763 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de11ae4b3b9b42c7b6e256023fb6a3a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:26.035121 27321 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 188138us
I20250903 20:05:25.855412 27209 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:56514 (request call id 133) took 1203 ms. Trace:
I20250903 20:05:26.035522 27209 rpcz_store.cc:276] 0903 20:05:24.652213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:24.656167 (+ 3954us) service_pool.cc:225] Handling call
0903 20:05:25.545237 (+889070us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x6160001a0e54. stack: 00007f589abe0dc6 00007f589abe0b80 00007f5899bc83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f58a6639700 00007f58aa3d371f 00007f58a4b04c36 00007f58a4b04965 00007f58a19910be 00007f58a198ff9e 00007f58a19961d5 00007f58a1996aff 00007f58a1996928 0000000000a774dc
0903 20:05:25.855383 (+310146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":333569280}
I20250903 20:05:25.852641 28396 consensus_queue.cc:1035] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:25.924566 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet d93258c032524b71997389b81846a5be
I20250903 20:05:26.036653 27059 catalog_manager.cc:6652] ReplaceTablet: tablet d93258c032524b71997389b81846a5be deleted and replaced by tablet 67d34a6947f94f08bad2d935892dbfde
I20250903 20:05:26.036899 27059 master_service.cc:948] ReplaceTablet: received request to replace tablet a4c582b441f6467c99ba0b090abcf4cc from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:25.868965 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet b6663ba96faa44b88fe8b6def4ff6951 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:25.869467 27693 tablet_bootstrap.cc:492] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:25.869501 27211 tablet_service.cc:1515] Processing DeleteTablet for tablet b6663ba96faa44b88fe8b6def4ff6951 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:26.038595 27693 ts_tablet_manager.cc:1397] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.206s user 0.005s sys 0.005s
I20250903 20:05:26.039326 27693 raft_consensus.cc:357] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.039500 27693 raft_consensus.cc:383] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.039556 27693 raft_consensus.cc:738] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:25.853027 28547 raft_consensus.cc:513] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.039790 27693 consensus_queue.cc:260] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.040205 28547 leader_election.cc:290] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:26.040259 27693 ts_tablet_manager.cc:1428] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:26.040632 27693 tablet_bootstrap.cc:492] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
W20250903 20:05:25.938931 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 79c0cc0c30ab42eab0be415e3db9589f consensus state has no leader
I20250903 20:05:26.041431 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e8b42e26d9e6426e96d80135d5015108"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:26.042009 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8b42e26d9e6426e96d80135d5015108 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:25.870072 27692 tablet_bootstrap.cc:492] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:25.921169 28596 raft_consensus.cc:491] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:26.042383 27692 ts_tablet_manager.cc:1397] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.322s user 0.012s sys 0.029s
I20250903 20:05:26.042347 28596 raft_consensus.cc:513] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.042899 28596 leader_election.cc:290] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:26.043102 27692 raft_consensus.cc:357] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.043251 27692 raft_consensus.cc:383] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.043300 27692 raft_consensus.cc:738] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:26.043493 27692 consensus_queue.cc:260] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.043732 27505 tablet_service.cc:1515] Processing DeleteTablet for tablet d93258c032524b71997389b81846a5be with delete_type TABLET_DATA_DELETED (replaced by tablet 67d34a6947f94f08bad2d935892dbfde) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:25.954234 27687 ts_tablet_manager.cc:1918] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:26.043946 27687 log.cc:1199] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/cd6d6bb514134bdf8e3bc0aaad180732
I20250903 20:05:26.043974 27692 ts_tablet_manager.cc:1428] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:26.044248 27692 tablet_bootstrap.cc:492] T a0293a69f1c246d08d414e95729b4ab4 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:26.045711 27692 tablet_bootstrap.cc:654] T a0293a69f1c246d08d414e95729b4ab4 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:26.045689 27057 master_service.cc:948] ReplaceTablet: received request to replace tablet 121a8ba102454cb4807483fac16e3781 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:25.875985 27695 tablet_bootstrap.cc:654] T c9e90abef9814338b70a3550ab5686e0 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:26.048504 27693 tablet_bootstrap.cc:654] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:25.953225 27065 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 912eee5b48ef467bbd6bbe34f8aa390c (table default.Table_f1f4f57d96f04f0caff06d9a642f5c46 [id=fdc7899c8b334d309b834ae1daa14fa0]). Expected version 4 got 0
I20250903 20:05:25.949200 27684 ts_tablet_manager.cc:1918] T 885fff47fea14b84ac6b34a57b827b4b P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:26.048995 27684 log.cc:1199] T 885fff47fea14b84ac6b34a57b827b4b P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/885fff47fea14b84ac6b34a57b827b4b
I20250903 20:05:26.049327 27684 ts_tablet_manager.cc:1939] T 885fff47fea14b84ac6b34a57b827b4b P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:26.050397 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 770) took 1308 ms. Trace:
I20250903 20:05:26.050491 27684 rpcz_store.cc:276] 0903 20:05:24.742304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:24.742507 (+ 203us) service_pool.cc:225] Handling call
0903 20:05:25.766953 (+1024446us) ts_tablet_manager.cc:1145] Deleting tablet 885fff47fea14b84ac6b34a57b827b4b
0903 20:05:25.766961 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:25.846396 (+ 79435us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:25.949063 (+102667us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:26.050370 (+101307us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1024174}
I20250903 20:05:26.050964 27684 tablet_replica.cc:331] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:26.051200 27684 raft_consensus.cc:2241] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:26.051492 27684 pending_rounds.cc:70] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f: Trying to abort 1 pending ops.
I20250903 20:05:26.051539 27684 pending_rounds.cc:77] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384974047244288 op_type: NO_OP noop_request { }
I20250903 20:05:26.051645 27684 raft_consensus.cc:2887] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:26.051734 27684 raft_consensus.cc:2270] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:25.951397 27246 raft_consensus.cc:2466] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:25.951965 27685 ts_tablet_manager.cc:1918] T 4157633525034a019fd6a80185892cec P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:26.052942 27685 log.cc:1199] T 4157633525034a019fd6a80185892cec P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/4157633525034a019fd6a80185892cec
I20250903 20:05:26.053328 27685 ts_tablet_manager.cc:1939] T 4157633525034a019fd6a80185892cec P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:26.053702 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 642) took 4074 ms. Trace:
I20250903 20:05:26.053788 27685 rpcz_store.cc:276] 0903 20:05:21.979505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:21.988874 (+ 9369us) service_pool.cc:225] Handling call
0903 20:05:25.834235 (+3845361us) ts_tablet_manager.cc:1145] Deleting tablet 4157633525034a019fd6a80185892cec
0903 20:05:25.834244 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:25.856307 (+ 22063us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:25.951859 (+ 95552us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:26.053684 (+101825us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3845146}
I20250903 20:05:26.059029 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e78e6d212daf4589a6b4b65711e356cb" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:26.059222 27248 raft_consensus.cc:3058] T e78e6d212daf4589a6b4b65711e356cb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:25.966535 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_7afeb3c17d814683916ce7fc2788310e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:25.978312 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 811c22a650c747a6a8db710e3cdd847c consensus state has no leader
W20250903 20:05:26.002951 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0cc6d68ca0fe45a492c77879c5d9e89c consensus state has no leader
I20250903 20:05:26.038147 28579 raft_consensus.cc:2802] T 0ad64c699b294679b6fec37b5ad23528 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:26.025103 27494 tablet_service.cc:1468] Processing CreateTablet for tablet b437eca9eb0b4c60bf797121a5a8bad8 (DEFAULT_TABLE table=default.Table_312cd54d2d6f4ae5bbcd5fb11ab7700e [id=716994dd8a9f4e0d857f8a88e95048a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.061707 28579 raft_consensus.cc:695] T 0ad64c699b294679b6fec37b5ad23528 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:26.062007 28579 consensus_queue.cc:237] T 0ad64c699b294679b6fec37b5ad23528 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.062131 27494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b437eca9eb0b4c60bf797121a5a8bad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.064545 27505 tablet_service.cc:1468] Processing CreateTablet for tablet e35574d360334150bb494e13f55d3656 (DEFAULT_TABLE table=default.Table_53bee4ebf7974f119a4a5b5bd223d51f [id=a1476e89c2fd4e0c8ec4d1b4763aec55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.064900 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e35574d360334150bb494e13f55d3656. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.026072 27499 tablet_service.cc:1468] Processing CreateTablet for tablet e15da14bcbdc4c80820795f46ef89b75 (DEFAULT_TABLE table=default.Table_b8435bf217c34217bbea1bad9cb30edb [id=b8575044e37346a294cc7289c56f3539]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.066947 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e15da14bcbdc4c80820795f46ef89b75. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:26.067090 27462 consensus_peers.cc:489] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe902f1b58924be1ac3a5585413f7789. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:26.067633 27246 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6d67cd0c0304c77ba571b414878dad2" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:26.067873 27246 raft_consensus.cc:2466] T c6d67cd0c0304c77ba571b414878dad2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:26.034729 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet cd92e3dafca6419f8eba7a4ef51a911e (table default.Table_c8d55d18c6bb4511ab10f35d2db6d2f9 [id=b15b2b0164104454a23d766feea4e0de]) successfully deleted
I20250903 20:05:26.026932 27488 tablet_service.cc:1468] Processing CreateTablet for tablet 3618bfd17de4408d870cd206b185d951 (DEFAULT_TABLE table=default.Table_18a6310ff569456f91088bbafcfd50f4 [id=b9b95fda98d640ef8345b626444e0971]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.070521 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3618bfd17de4408d870cd206b185d951. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.070711 27321 leader_election.cc:304] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:26.071523 28568 raft_consensus.cc:2802] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:26.071624 28568 raft_consensus.cc:695] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:26.071851 28568 consensus_queue.cc:237] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.027961 27496 tablet_service.cc:1468] Processing CreateTablet for tablet 31186292b25f4c3cb23597f66dcbd282 (DEFAULT_TABLE table=default.Table_9e9e434350e54dc4a9397cee96d16383 [id=19062b2beb664b52ab0c17fe7afc4aa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.074766 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31186292b25f4c3cb23597f66dcbd282. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.029104 27507 tablet_service.cc:1468] Processing CreateTablet for tablet ec8cea2d64b642689d0bc11295745ef0 (DEFAULT_TABLE table=default.Table_5d167e8474a043f09801a985626f69e4 [id=5b991c02da364bf4902498aadbe4084e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.078380 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_2623cd1b586e40818f657e9fc9f5f570"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:26.078807 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec8cea2d64b642689d0bc11295745ef0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:26.079149 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2623cd1b586e40818f657e9fc9f5f570 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:26.079298 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 885fff47fea14b84ac6b34a57b827b4b (table default.Table_1c679659094d4666aaaf68f76f667890 [id=61a25739b03042f7bd5bb693b68656d5]) successfully deleted
I20250903 20:05:26.079342 27064 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 912eee5b48ef467bbd6bbe34f8aa390c (table default.Table_f1f4f57d96f04f0caff06d9a642f5c46 [id=fdc7899c8b334d309b834ae1daa14fa0]). Expected version 4 got 0
I20250903 20:05:26.080391 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 648) took 3997 ms. Trace:
I20250903 20:05:26.080476 27685 rpcz_store.cc:276] 0903 20:05:22.083025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.083103 (+ 78us) service_pool.cc:225] Handling call
0903 20:05:26.054138 (+3971035us) ts_tablet_manager.cc:1145] Deleting tablet 57a1167457b84ea8a493be232b1d088a
0903 20:05:26.054148 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:26.080369 (+ 26221us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3970806}
W20250903 20:05:26.080667 27462 leader_election.cc:343] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: e78e6d212daf4589a6b4b65711e356cb
I20250903 20:05:26.080790 27685 tablet_replica.cc:331] T ac9b5d4f39db4f05a15316585040d142 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
W20250903 20:05:26.080854 27462 leader_election.cc:343] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: c6d67cd0c0304c77ba571b414878dad2
I20250903 20:05:26.080965 27685 raft_consensus.cc:2241] T ac9b5d4f39db4f05a15316585040d142 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:26.080993 27462 leader_election.cc:304] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:26.081117 27685 raft_consensus.cc:2270] T ac9b5d4f39db4f05a15316585040d142 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:26.081380 28547 raft_consensus.cc:2802] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:26.081450 28547 raft_consensus.cc:491] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:26.081506 28547 raft_consensus.cc:3058] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:26.081691 27685 ts_tablet_manager.cc:1905] T ac9b5d4f39db4f05a15316585040d142 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:26.033161 27502 tablet_service.cc:1468] Processing CreateTablet for tablet 57943651b6374843ade249d11d6183cf (DEFAULT_TABLE table=default.Table_6181f8539fd34670ad894eae5e3ac4d3 [id=77cc8a47fe3c4b6bb7453993aa515d3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.082867 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57943651b6374843ade249d11d6183cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.087445 27574 heartbeater.cc:344] Connected to a master server at 127.26.65.125:44059
I20250903 20:05:26.088987 27684 ts_tablet_manager.cc:1905] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:26.093339 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7afeb3c17d814683916ce7fc2788310e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:26.034602 27066 rpcz_store.cc:276] 0903 20:05:24.986196 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:25.645952 (+659756us) service_pool.cc:225] Handling call
0903 20:05:25.653969 (+ 8017us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0903 20:05:25.654269 (+ 300us) catalog_manager.cc:3683] Apply alter schema
0903 20:05:25.688121 (+ 33852us) catalog_manager.cc:3715] Acquired catalog manager lock
0903 20:05:25.692552 (+ 4431us) catalog_manager.cc:3873] Updating metadata on disk
0903 20:05:25.908026 (+215474us) catalog_manager.cc:3904] Committing alterations to in-memory state
0903 20:05:26.034427 (+126401us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:25.896997 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:25.897729 (+ 732us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:25.897735 (+ 6us) write_op.cc:435] Acquired schema lock
0903 20:05:25.897737 (+ 2us) tablet.cc:660] Decoding operations
0903 20:05:25.898305 (+ 568us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:25.898318 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:25.898324 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0903 20:05:25.898769 (+ 445us) tablet.cc:699] Row locks acquired
0903 20:05:25.898773 (+ 4us) write_op.cc:260] PREPARE: finished
0903 20:05:25.898965 (+ 192us) write_op.cc:270] Start()
0903 20:05:25.899152 (+ 187us) write_op.cc:276] Timestamp: P: 1756929925898915 usec, L: 0
0903 20:05:25.899156 (+ 4us) op_driver.cc:348] REPLICATION: starting
0903 20:05:25.901741 (+ 2585us) log.cc:844] Serialized 405 byte log entry
0903 20:05:25.903330 (+ 1589us) spinlock_profiling.cc:243] Waited 2.28 ms on lock 0x616000040f54. stack: 00007f4b9d6c2dc6 00007f4b9d6c2b80 00007f4b9c6aa3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f4ba9121d62 00007f4baa4ca2d5 00007f4baa4c8df9 00007f4baa4d1220 00007f4baa4d102d 0000000000a774dc 00007f4b9d70914f 00007f4b9d70c41b 00007f4b9d70c298 0000000000a774dc
0903 20:05:25.907295 (+ 3965us) op_driver.cc:464] REPLICATION: finished
0903 20:05:25.907544 (+ 249us) write_op.cc:301] APPLY: starting
0903 20:05:25.907565 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:25.907621 (+ 56us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:25.907625 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:25.907685 (+ 60us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:25.907729 (+ 44us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0903 20:05:25.907739 (+ 10us) write_op.cc:312] APPLY: finished
0903 20:05:25.907832 (+ 93us) log.cc:844] Serialized 23 byte log entry
0903 20:05:25.907900 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:25.907925 (+ 25us) write_op.cc:454] Released schema lock
0903 20:05:25.907934 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0903 20:05:25.911237 (+ 3303us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:25.911247 (+ 10us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":190,"apply.run_cpu_time_us":467,"apply.run_wall_time_us":466,"num_ops":1,"prepare.queue_time_us":193946,"prepare.run_cpu_time_us":1336,"prepare.run_wall_time_us":14125,"raft.queue_time_us":133,"raft.run_cpu_time_us":349,"raft.run_wall_time_us":349,"replication_time_us":8097,"spinlock_wait_cycles":5248896}]]}
I20250903 20:05:26.095749 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 167b359047ac457c9d165ec199cd9163 (DEFAULT_TABLE table=default.Table_d4af3ffb74834cccbbbee39223164dab [id=b0762c5ff3c64468abcfbc0e2f99479f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.096130 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 167b359047ac457c9d165ec199cd9163. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.099974 27490 tablet_service.cc:1468] Processing CreateTablet for tablet 8412226d9d574c938172429cf58ee210 (DEFAULT_TABLE table=default.Table_26e71ae071ec449d866264a856b29b89 [id=1a4e82adf74f4e97b85d59ad45b979ff]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:26.103026 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet c49767d618be4fd18f30e5124ce0cb84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c49767d618be4fd18f30e5124ce0cb84 already in progress: creating tablet
I20250903 20:05:26.103178 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7b1039b36f6940dd95220a8f25849711" } modify_external_catalogs: true
W20250903 20:05:26.103361 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 36e1061915574407b6e7695be8b1e55a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36e1061915574407b6e7695be8b1e55a already in progress: creating tablet
I20250903 20:05:26.104713 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7b1039b36f6940dd95220a8f25849711" } modify_external_catalogs: true
W20250903 20:05:26.105719 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 17e7deb74389448db37289563b86c253 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17e7deb74389448db37289563b86c253 already in progress: creating tablet
W20250903 20:05:26.105957 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet e9126e4a18c546c28927f22c9f16426e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9126e4a18c546c28927f22c9f16426e already in progress: creating tablet
W20250903 20:05:26.108901 27052 catalog_manager.cc:4895] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 57a1167457b84ea8a493be232b1d088a because the tablet was not found. No further retry: Not found: Tablet not found: 57a1167457b84ea8a493be232b1d088a
I20250903 20:05:26.108899 27498 tablet_service.cc:1468] Processing CreateTablet for tablet a87f5eb2868f4b28bd18636d6c2a458d (DEFAULT_TABLE table=default.Table_89f979a23e14485289384ecac15e6145 [id=6ae007b50afb45a786f0058a57d2f78e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.109975 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 45d9b6ff520348628d19671127b9426c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:26.111424 27060 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 912eee5b48ef467bbd6bbe34f8aa390c (table default.Table_f1f4f57d96f04f0caff06d9a642f5c46 [id=fdc7899c8b334d309b834ae1daa14fa0]). Expected version 4 got 0
I20250903 20:05:26.113576 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet a4c582b441f6467c99ba0b090abcf4cc
I20250903 20:05:26.114785 27500 tablet_service.cc:1515] Processing DeleteTablet for tablet a4c582b441f6467c99ba0b090abcf4cc with delete_type TABLET_DATA_DELETED (replaced by tablet a6ffdb473d5745d181a7121d10991705) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:26.115795 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c21ce98311d847f2aaae2ea2c0549856" } modify_external_catalogs: true
I20250903 20:05:26.116807 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6688c005609a44fcaf62e7da7d57df49" } modify_external_catalogs: true
I20250903 20:05:26.117872 27500 tablet_service.cc:1468] Processing CreateTablet for tablet cd14f3d496ea429c98bcb73ca2d8a47d (DEFAULT_TABLE table=default.Table_c4b81b1196c847c09dcf2d9281c75b26 [id=f2b2542fe75144bb9cb4821a55e35476]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.114178 27059 catalog_manager.cc:6652] ReplaceTablet: tablet a4c582b441f6467c99ba0b090abcf4cc deleted and replaced by tablet a6ffdb473d5745d181a7121d10991705
I20250903 20:05:26.120803 27211 tablet_service.cc:1515] Processing DeleteTablet for tablet 514e382479014fa7bf1f404efff11c86 with delete_type TABLET_DATA_DELETED (replaced by tablet 07439db2c3b8470eb2b797cc68376aa7) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:26.122711 27365 tablet_service.cc:1468] Processing CreateTablet for tablet b437eca9eb0b4c60bf797121a5a8bad8 (DEFAULT_TABLE table=default.Table_312cd54d2d6f4ae5bbcd5fb11ab7700e [id=716994dd8a9f4e0d857f8a88e95048a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.123102 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b437eca9eb0b4c60bf797121a5a8bad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.125025 27433 heartbeater.cc:344] Connected to a master server at 127.26.65.125:44059
I20250903 20:05:26.128407 27064 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_39de38c00cdb44eb85fdb76134b7139c" } modify_external_catalogs: true
I20250903 20:05:26.128659 27064 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_39de38c00cdb44eb85fdb76134b7139c" } modify_external_catalogs: true
I20250903 20:05:26.130352 27206 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:56514 (request call id 131) took 1815 ms. Trace:
I20250903 20:05:26.130453 27206 rpcz_store.cc:276] 0903 20:05:24.315249 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:24.315712 (+ 463us) service_pool.cc:225] Handling call
0903 20:05:25.561033 (+1245321us) spinlock_profiling.cc:243] Waited 88.5 ms on lock 0x6160001e4654. stack: 00007f589abe0dc6 00007f589abe0b80 00007f5899bc83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f58a6639700 00007f58aa3d371f 00007f58a4b04c36 00007f58a4b04965 00007f58a19910be 00007f58a198ff9e 00007f58a19961d5 00007f58a1996aff 00007f58a1996928 0000000000a774dc
0903 20:05:25.773352 (+212319us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x6160001e4c54. stack: 00007f589abe0dc6 00007f589abe0b80 00007f5899bc83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f58a6639700 00007f58aa3d371f 00007f58a4b04c36 00007f58a4b04965 00007f58a19910be 00007f58a198ff9e 00007f58a19961d5 00007f58a1996aff 00007f58a1996928 0000000000a774dc
0903 20:05:25.849454 (+ 76102us) spinlock_profiling.cc:243] Waited 74.1 ms on lock 0x616000225d54. stack: 00007f589abe0dc6 00007f589abe0b80 00007f5899bc83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f58a6639700 00007f58aa3d371f 00007f58a4b04c36 00007f58a4b04965 00007f58a19910be 00007f58a198ff9e 00007f58a19961d5 00007f58a1996aff 00007f58a1996928 0000000000a774dc
0903 20:05:26.055604 (+206150us) spinlock_profiling.cc:243] Waited 205 ms on lock 0x6160001c2154. stack: 00007f589abe0dc6 00007f589abe0b80 00007f5899bc83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f58a6639700 00007f58aa3d371f 00007f58a4b04c36 00007f58a4b04965 00007f58a19910be 00007f58a198ff9e 00007f58a19961d5 00007f58a1996aff 00007f58a1996928 0000000000a774dc
0903 20:05:26.130316 (+ 74712us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1170283136}
I20250903 20:05:26.135408 27687 ts_tablet_manager.cc:1939] T cd6d6bb514134bdf8e3bc0aaad180732 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:26.137641 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c21ce98311d847f2aaae2ea2c0549856" } modify_external_catalogs: true
I20250903 20:05:26.138291 27060 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 0ad64c699b294679b6fec37b5ad23528 (table default.Table_24e55a7c820c413f8514bd3ad7bd3269 [id=0904dc5d0c594bc3bbf32de2508c6195]). Expected version 3 got 0
I20250903 20:05:26.144961 27497 tablet_service.cc:1468] Processing CreateTablet for tablet 4aea507c5e214a57a80bd4de1a6ccd67 (DEFAULT_TABLE table=default.Table_a3ccbeb42cef4247a4f9b6b349af9371 [id=cc99c3326ca04692af63ce855bbb1caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.145435 27489 tablet_service.cc:1515] Processing DeleteTablet for tablet cd6d6bb514134bdf8e3bc0aaad180732 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:26.145658 27503 tablet_service.cc:1515] Processing DeleteTablet for tablet e78e6d212daf4589a6b4b65711e356cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:26.145830 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 283a3afed7414a958b2dc4176f00bb01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:26.146011 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet dbfdab4332234b2a93fd3c901a9cb645 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:26.149402 27687 tablet_replica.cc:331] T c9e90abef9814338b70a3550ab5686e0 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:26.149626 27687 raft_consensus.cc:2241] T c9e90abef9814338b70a3550ab5686e0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:26.149777 27687 raft_consensus.cc:2270] T c9e90abef9814338b70a3550ab5686e0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:26.150334 27687 ts_tablet_manager.cc:1905] T c9e90abef9814338b70a3550ab5686e0 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:26.150939 27490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8412226d9d574c938172429cf58ee210. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.154661 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aea507c5e214a57a80bd4de1a6ccd67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.155310 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd14f3d496ea429c98bcb73ca2d8a47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.158726 27498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a87f5eb2868f4b28bd18636d6c2a458d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.158155 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6688c005609a44fcaf62e7da7d57df49" } modify_external_catalogs: true
I20250903 20:05:26.161103 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 121a8ba102454cb4807483fac16e3781
I20250903 20:05:26.162796 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet 121a8ba102454cb4807483fac16e3781 with delete_type TABLET_DATA_DELETED (replaced by tablet e64c4a9179b6491ba92df75e0cc1bc50) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:26.164599 27493 tablet_service.cc:1468] Processing CreateTablet for tablet 6b87e04ec9dd47bb9bba5dcb0ed39a5f (DEFAULT_TABLE table=default.Table_cb71c02ad33b47768df2ee87bc601760 [id=8e4226a54ada4ee8a83a6f9e1168ab1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.165042 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b87e04ec9dd47bb9bba5dcb0ed39a5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.168578 28396 raft_consensus.cc:491] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:26.168744 28396 raft_consensus.cc:513] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.169435 28396 leader_election.cc:290] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:26.171005 27462 leader_election.cc:343] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 4157633525034a019fd6a80185892cec
I20250903 20:05:26.173921 27248 raft_consensus.cc:2466] T e78e6d212daf4589a6b4b65711e356cb P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
W20250903 20:05:26.173988 27462 leader_election.cc:343] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 4157633525034a019fd6a80185892cec
I20250903 20:05:26.174108 27462 leader_election.cc:304] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:26.174535 27462 leader_election.cc:304] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:26.174762 28598 raft_consensus.cc:2747] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:26.174947 28598 raft_consensus.cc:2802] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:26.175052 28598 raft_consensus.cc:695] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:26.175314 28598 consensus_queue.cc:237] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.177714 27057 catalog_manager.cc:6652] ReplaceTablet: tablet 121a8ba102454cb4807483fac16e3781 deleted and replaced by tablet e64c4a9179b6491ba92df75e0cc1bc50
I20250903 20:05:26.178676 27059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7532aa050a154ef1824ea9f89f029692" } modify_external_catalogs: true
I20250903 20:05:26.178918 27059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7532aa050a154ef1824ea9f89f029692" } modify_external_catalogs: true
I20250903 20:05:26.179132 27695 tablet_bootstrap.cc:492] T c9e90abef9814338b70a3550ab5686e0 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:26.179286 27695 ts_tablet_manager.cc:1397] T c9e90abef9814338b70a3550ab5686e0 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.349s user 0.005s sys 0.000s
I20250903 20:05:26.180277 27695 raft_consensus.cc:357] T c9e90abef9814338b70a3550ab5686e0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.180474 27695 raft_consensus.cc:383] T c9e90abef9814338b70a3550ab5686e0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.180534 27695 raft_consensus.cc:738] T c9e90abef9814338b70a3550ab5686e0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:26.180790 27695 consensus_queue.cc:260] T c9e90abef9814338b70a3550ab5686e0 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.181368 27695 ts_tablet_manager.cc:1428] T c9e90abef9814338b70a3550ab5686e0 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:26.181627 27695 tablet_bootstrap.cc:492] T 961ade36f7d14eba91bb16d5eae2e254 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:26.182703 27211 tablet_service.cc:1468] Processing CreateTablet for tablet b437eca9eb0b4c60bf797121a5a8bad8 (DEFAULT_TABLE table=default.Table_312cd54d2d6f4ae5bbcd5fb11ab7700e [id=716994dd8a9f4e0d857f8a88e95048a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.174952 28547 raft_consensus.cc:513] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.183581 27349 tablet_service.cc:1468] Processing CreateTablet for tablet e15da14bcbdc4c80820795f46ef89b75 (DEFAULT_TABLE table=default.Table_b8435bf217c34217bbea1bad9cb30edb [id=b8575044e37346a294cc7289c56f3539]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.183987 28547 leader_election.cc:290] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:26.184444 27357 tablet_service.cc:1468] Processing CreateTablet for tablet 3618bfd17de4408d870cd206b185d951 (DEFAULT_TABLE table=default.Table_18a6310ff569456f91088bbafcfd50f4 [id=b9b95fda98d640ef8345b626444e0971]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.185169 27358 tablet_service.cc:1468] Processing CreateTablet for tablet 31186292b25f4c3cb23597f66dcbd282 (DEFAULT_TABLE table=default.Table_9e9e434350e54dc4a9397cee96d16383 [id=19062b2beb664b52ab0c17fe7afc4aa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.185874 27361 tablet_service.cc:1468] Processing CreateTablet for tablet ec8cea2d64b642689d0bc11295745ef0 (DEFAULT_TABLE table=default.Table_5d167e8474a043f09801a985626f69e4 [id=5b991c02da364bf4902498aadbe4084e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.186631 27352 tablet_service.cc:1468] Processing CreateTablet for tablet 57943651b6374843ade249d11d6183cf (DEFAULT_TABLE table=default.Table_6181f8539fd34670ad894eae5e3ac4d3 [id=77cc8a47fe3c4b6bb7453993aa515d3d]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:26.186920 27462 leader_election.cc:343] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: c6d67cd0c0304c77ba571b414878dad2
I20250903 20:05:26.187278 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet d93258c032524b71997389b81846a5be with delete_type TABLET_DATA_DELETED (replaced by tablet 67d34a6947f94f08bad2d935892dbfde) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:26.188164 27360 tablet_service.cc:1468] Processing CreateTablet for tablet e35574d360334150bb494e13f55d3656 (DEFAULT_TABLE table=default.Table_53bee4ebf7974f119a4a5b5bd223d51f [id=a1476e89c2fd4e0c8ec4d1b4763aec55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.189340 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet d93258c032524b71997389b81846a5be with delete_type TABLET_DATA_DELETED (replaced by tablet 67d34a6947f94f08bad2d935892dbfde) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:26.189616 27220 tablet_service.cc:1468] Processing CreateTablet for tablet 57943651b6374843ade249d11d6183cf (DEFAULT_TABLE table=default.Table_6181f8539fd34670ad894eae5e3ac4d3 [id=77cc8a47fe3c4b6bb7453993aa515d3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.189729 27221 tablet_service.cc:1468] Processing CreateTablet for tablet e15da14bcbdc4c80820795f46ef89b75 (DEFAULT_TABLE table=default.Table_b8435bf217c34217bbea1bad9cb30edb [id=b8575044e37346a294cc7289c56f3539]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.190037 27227 tablet_service.cc:1468] Processing CreateTablet for tablet e35574d360334150bb494e13f55d3656 (DEFAULT_TABLE table=default.Table_53bee4ebf7974f119a4a5b5bd223d51f [id=a1476e89c2fd4e0c8ec4d1b4763aec55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.190245 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 31186292b25f4c3cb23597f66dcbd282 (DEFAULT_TABLE table=default.Table_9e9e434350e54dc4a9397cee96d16383 [id=19062b2beb664b52ab0c17fe7afc4aa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.190706 27228 tablet_service.cc:1468] Processing CreateTablet for tablet ec8cea2d64b642689d0bc11295745ef0 (DEFAULT_TABLE table=default.Table_5d167e8474a043f09801a985626f69e4 [id=5b991c02da364bf4902498aadbe4084e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.192353 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6d67cd0c0304c77ba571b414878dad2" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:26.192523 27248 raft_consensus.cc:3058] T c6d67cd0c0304c77ba571b414878dad2 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:26.190037 27226 tablet_service.cc:1468] Processing CreateTablet for tablet 3618bfd17de4408d870cd206b185d951 (DEFAULT_TABLE table=default.Table_18a6310ff569456f91088bbafcfd50f4 [id=b9b95fda98d640ef8345b626444e0971]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.197130 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 68201d1d224646d3954cb2aa58a3c37e (DEFAULT_TABLE table=default.Table_7356ad4dcf4d4c28a78fdb0988bd6059 [id=794e6b016eda4eb08189d9bd4832ee77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.198340 27350 tablet_service.cc:1515] Processing DeleteTablet for tablet 514e382479014fa7bf1f404efff11c86 with delete_type TABLET_DATA_DELETED (replaced by tablet 07439db2c3b8470eb2b797cc68376aa7) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:26.199252 27503 tablet_service.cc:1468] Processing CreateTablet for tablet c6583a9e2c2748e5bf450ae5d09bf272 (DEFAULT_TABLE table=default.Table_ced04d6dca6d46a792f30d4f704d1468 [id=174224f17e2e4749b6a901d5df850c10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.200155 27489 tablet_service.cc:1468] Processing CreateTablet for tablet 28afd941430b45ed95d5f9bb49a2220a (DEFAULT_TABLE table=default.Table_042efc3626d2434fa91ad982d346f3ed [id=2b09f16232be41c5be9226a06f94ec3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.200989 27350 tablet_service.cc:1515] Processing DeleteTablet for tablet 6788b1683f984949816edfc5b68de8db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:26.201682 27350 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bcb9d5f010f4297bab78c0183889238 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a9e55f0811345f7afa71645f38ebdaf) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:26.202525 27350 tablet_service.cc:1468] Processing CreateTablet for tablet 167b359047ac457c9d165ec199cd9163 (DEFAULT_TABLE table=default.Table_d4af3ffb74834cccbbbee39223164dab [id=b0762c5ff3c64468abcfbc0e2f99479f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.203543 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 167b359047ac457c9d165ec199cd9163 (DEFAULT_TABLE table=default.Table_d4af3ffb74834cccbbbee39223164dab [id=b0762c5ff3c64468abcfbc0e2f99479f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.204468 27356 tablet_service.cc:1468] Processing CreateTablet for tablet 8412226d9d574c938172429cf58ee210 (DEFAULT_TABLE table=default.Table_26e71ae071ec449d866264a856b29b89 [id=1a4e82adf74f4e97b85d59ad45b979ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.205401 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 8412226d9d574c938172429cf58ee210 (DEFAULT_TABLE table=default.Table_26e71ae071ec449d866264a856b29b89 [id=1a4e82adf74f4e97b85d59ad45b979ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.205331 27057 catalog_manager.cc:5582] T 0ad64c699b294679b6fec37b5ad23528 P 51578fb7bceb4e119b3ab1e58c4f5528 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66). New cstate: current_term: 1 leader_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:26.205672 27362 tablet_service.cc:1468] Processing CreateTablet for tablet a87f5eb2868f4b28bd18636d6c2a458d (DEFAULT_TABLE table=default.Table_89f979a23e14485289384ecac15e6145 [id=6ae007b50afb45a786f0058a57d2f78e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.205853 27057 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 0ad64c699b294679b6fec37b5ad23528 (table default.Table_24e55a7c820c413f8514bd3ad7bd3269 [id=0904dc5d0c594bc3bbf32de2508c6195]). Expected version 3 got 0
I20250903 20:05:26.207156 27224 tablet_service.cc:1468] Processing CreateTablet for tablet a87f5eb2868f4b28bd18636d6c2a458d (DEFAULT_TABLE table=default.Table_89f979a23e14485289384ecac15e6145 [id=6ae007b50afb45a786f0058a57d2f78e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.207932 27216 tablet_service.cc:1515] Processing DeleteTablet for tablet a4c582b441f6467c99ba0b090abcf4cc with delete_type TABLET_DATA_DELETED (replaced by tablet a6ffdb473d5745d181a7121d10991705) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:26.208299 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b664a0b055d84bc09046969e26863021"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:26.208600 27348 tablet_service.cc:1515] Processing DeleteTablet for tablet a4c582b441f6467c99ba0b090abcf4cc with delete_type TABLET_DATA_DELETED (replaced by tablet a6ffdb473d5745d181a7121d10991705) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:26.208891 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b664a0b055d84bc09046969e26863021 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:26.209542 27348 tablet_service.cc:1468] Processing CreateTablet for tablet cd14f3d496ea429c98bcb73ca2d8a47d (DEFAULT_TABLE table=default.Table_c4b81b1196c847c09dcf2d9281c75b26 [id=f2b2542fe75144bb9cb4821a55e35476]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.210412 27216 tablet_service.cc:1515] Processing DeleteTablet for tablet a0293a69f1c246d08d414e95729b4ab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:26.211529 27216 tablet_service.cc:1468] Processing CreateTablet for tablet cd14f3d496ea429c98bcb73ca2d8a47d (DEFAULT_TABLE table=default.Table_c4b81b1196c847c09dcf2d9281c75b26 [id=f2b2542fe75144bb9cb4821a55e35476]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.212723 27351 tablet_service.cc:1468] Processing CreateTablet for tablet 4aea507c5e214a57a80bd4de1a6ccd67 (DEFAULT_TABLE table=default.Table_a3ccbeb42cef4247a4f9b6b349af9371 [id=cc99c3326ca04692af63ce855bbb1caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.213621 27354 tablet_service.cc:1515] Processing DeleteTablet for tablet 121a8ba102454cb4807483fac16e3781 with delete_type TABLET_DATA_DELETED (replaced by tablet e64c4a9179b6491ba92df75e0cc1bc50) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:26.214390 27229 tablet_service.cc:1515] Processing DeleteTablet for tablet 121a8ba102454cb4807483fac16e3781 with delete_type TABLET_DATA_DELETED (replaced by tablet e64c4a9179b6491ba92df75e0cc1bc50) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:26.216142 27229 tablet_service.cc:1468] Processing CreateTablet for tablet 4aea507c5e214a57a80bd4de1a6ccd67 (DEFAULT_TABLE table=default.Table_a3ccbeb42cef4247a4f9b6b349af9371 [id=cc99c3326ca04692af63ce855bbb1caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.218051 27354 tablet_service.cc:1468] Processing CreateTablet for tablet 6b87e04ec9dd47bb9bba5dcb0ed39a5f (DEFAULT_TABLE table=default.Table_cb71c02ad33b47768df2ee87bc601760 [id=8e4226a54ada4ee8a83a6f9e1168ab1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.218845 27693 tablet_bootstrap.cc:492] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:26.219199 27222 tablet_service.cc:1468] Processing CreateTablet for tablet 6b87e04ec9dd47bb9bba5dcb0ed39a5f (DEFAULT_TABLE table=default.Table_cb71c02ad33b47768df2ee87bc601760 [id=8e4226a54ada4ee8a83a6f9e1168ab1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.219637 27693 ts_tablet_manager.cc:1397] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.179s user 0.010s sys 0.020s
I20250903 20:05:26.219697 27353 tablet_service.cc:1468] Processing CreateTablet for tablet 68201d1d224646d3954cb2aa58a3c37e (DEFAULT_TABLE table=default.Table_7356ad4dcf4d4c28a78fdb0988bd6059 [id=794e6b016eda4eb08189d9bd4832ee77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.220638 27693 raft_consensus.cc:357] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.220981 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 68201d1d224646d3954cb2aa58a3c37e (DEFAULT_TABLE table=default.Table_7356ad4dcf4d4c28a78fdb0988bd6059 [id=794e6b016eda4eb08189d9bd4832ee77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.222546 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_6b51dff3c41c4a67b5bad68460ae919f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:26.222951 27213 tablet_service.cc:1468] Processing CreateTablet for tablet c6583a9e2c2748e5bf450ae5d09bf272 (DEFAULT_TABLE table=default.Table_ced04d6dca6d46a792f30d4f704d1468 [id=174224f17e2e4749b6a901d5df850c10]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:26.223174 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b51dff3c41c4a67b5bad68460ae919f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:26.221931 27347 tablet_service.cc:1468] Processing CreateTablet for tablet c6583a9e2c2748e5bf450ae5d09bf272 (DEFAULT_TABLE table=default.Table_ced04d6dca6d46a792f30d4f704d1468 [id=174224f17e2e4749b6a901d5df850c10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.223441 27693 raft_consensus.cc:383] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.223542 27693 raft_consensus.cc:738] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:26.224256 27359 tablet_service.cc:1468] Processing CreateTablet for tablet 28afd941430b45ed95d5f9bb49a2220a (DEFAULT_TABLE table=default.Table_042efc3626d2434fa91ad982d346f3ed [id=2b09f16232be41c5be9226a06f94ec3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.224669 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet cd6d6bb514134bdf8e3bc0aaad180732 (table default.Table_ef30e0f336384b50bf8293fe525252cf [id=b2f5b8646204428993ce29cf7b2e4caa]) successfully deleted
I20250903 20:05:26.224750 27223 tablet_service.cc:1468] Processing CreateTablet for tablet 28afd941430b45ed95d5f9bb49a2220a (DEFAULT_TABLE table=default.Table_042efc3626d2434fa91ad982d346f3ed [id=2b09f16232be41c5be9226a06f94ec3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.225076 27693 consensus_queue.cc:260] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.225724 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b69c590ef46648439e74d958bf9ffdf3" } new_table_name: "default.Table_412e2b595d4e4e83b26634356476d069" modify_external_catalogs: true
W20250903 20:05:26.225780 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 005d7bc2c7534f288caec8326fdf2416 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 005d7bc2c7534f288caec8326fdf2416 already in progress: creating tablet
I20250903 20:05:26.225860 27693 ts_tablet_manager.cc:1428] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250903 20:05:26.227037 27692 tablet_bootstrap.cc:492] T a0293a69f1c246d08d414e95729b4ab4 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:26.227250 27692 ts_tablet_manager.cc:1397] T a0293a69f1c246d08d414e95729b4ab4 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.183s user 0.012s sys 0.017s
I20250903 20:05:26.228148 27692 raft_consensus.cc:357] T a0293a69f1c246d08d414e95729b4ab4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.228404 27692 raft_consensus.cc:383] T a0293a69f1c246d08d414e95729b4ab4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.228503 27692 raft_consensus.cc:738] T a0293a69f1c246d08d414e95729b4ab4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:26.228787 27692 consensus_queue.cc:260] T a0293a69f1c246d08d414e95729b4ab4 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.229399 27692 ts_tablet_manager.cc:1428] T a0293a69f1c246d08d414e95729b4ab4 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:26.229727 27692 tablet_bootstrap.cc:492] T 9685cba2f73b412382a69dab2c7f50c1 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:26.230756 27065 catalog_manager.cc:5582] T 516830fc42e84af693b3af52caee30ee P e72ef1dbcc5143cf9f04a808f101c80f reported cstate change: term changed from 0 to 1, leader changed from <none> to e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67). New cstate: current_term: 1 leader_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: HEALTHY } } }
I20250903 20:05:26.232532 27693 tablet_bootstrap.cc:492] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:26.234045 27693 tablet_bootstrap.cc:654] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:26.231640 27692 tablet_bootstrap.cc:654] T 9685cba2f73b412382a69dab2c7f50c1 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:26.249594 27695 tablet_bootstrap.cc:654] T 961ade36f7d14eba91bb16d5eae2e254 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:26.252030 27065 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 516830fc42e84af693b3af52caee30ee (table default.Table_4d303308b63c4ce09056e226d1a30498 [id=098bc9a9b0a24a4a9edb7859310ac9b3]). Expected version 3 got 0
I20250903 20:05:26.253278 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8412226d9d574c938172429cf58ee210. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.253300 27684 ts_tablet_manager.cc:1918] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:26.253409 27684 log.cc:1199] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/cb4d99139516405bb65c7480ddf259f1
I20250903 20:05:26.253682 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a87f5eb2868f4b28bd18636d6c2a458d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.253738 27684 ts_tablet_manager.cc:1939] T cb4d99139516405bb65c7480ddf259f1 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:26.253916 27353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68201d1d224646d3954cb2aa58a3c37e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.254092 27354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b87e04ec9dd47bb9bba5dcb0ed39a5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.254261 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd14f3d496ea429c98bcb73ca2d8a47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.254432 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6583a9e2c2748e5bf450ae5d09bf272. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.254815 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28afd941430b45ed95d5f9bb49a2220a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.255151 27351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aea507c5e214a57a80bd4de1a6ccd67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.255492 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 167b359047ac457c9d165ec199cd9163. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.256011 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31186292b25f4c3cb23597f66dcbd282. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.257128 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec8cea2d64b642689d0bc11295745ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.257658 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3618bfd17de4408d870cd206b185d951. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.258188 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57943651b6374843ade249d11d6183cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.258754 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e15da14bcbdc4c80820795f46ef89b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.259259 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e35574d360334150bb494e13f55d3656. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.266284 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 771) took 1270 ms. Trace:
I20250903 20:05:26.266399 27684 rpcz_store.cc:276] 0903 20:05:24.995522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:24.997187 (+ 1665us) service_pool.cc:225] Handling call
0903 20:05:26.050908 (+1053721us) ts_tablet_manager.cc:1145] Deleting tablet cb4d99139516405bb65c7480ddf259f1
0903 20:05:26.050921 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:26.150410 (+ 99489us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:26.253179 (+102769us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:26.266261 (+ 13082us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1052850}
I20250903 20:05:26.266737 27684 tablet_replica.cc:331] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:26.266925 27684 raft_consensus.cc:2241] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:26.267165 27684 pending_rounds.cc:70] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f: Trying to abort 1 pending ops.
I20250903 20:05:26.267154 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 167b359047ac457c9d165ec199cd9163. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.267204 27684 pending_rounds.cc:77] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384975117307904 op_type: NO_OP noop_request { }
I20250903 20:05:26.267287 27684 raft_consensus.cc:2887] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:26.267361 27684 raft_consensus.cc:2270] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:26.267350 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6583a9e2c2748e5bf450ae5d09bf272. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.267760 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e35574d360334150bb494e13f55d3656. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.267913 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3618bfd17de4408d870cd206b185d951. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.268268 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b87e04ec9dd47bb9bba5dcb0ed39a5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.268589 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68201d1d224646d3954cb2aa58a3c37e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.268878 27216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd14f3d496ea429c98bcb73ca2d8a47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.269410 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a87f5eb2868f4b28bd18636d6c2a458d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.269776 27223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28afd941430b45ed95d5f9bb49a2220a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.269855 27684 ts_tablet_manager.cc:1905] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:26.270128 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8412226d9d574c938172429cf58ee210. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.270541 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57943651b6374843ade249d11d6183cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.270982 27229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aea507c5e214a57a80bd4de1a6ccd67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.272740 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e15da14bcbdc4c80820795f46ef89b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.273228 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31186292b25f4c3cb23597f66dcbd282. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.273343 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet cb4d99139516405bb65c7480ddf259f1 (table default.Table_a2fef2e8171c466e86ce74826ba6e281 [id=cbcfc7067aca4796b39e3eb195f38903]) successfully deleted
I20250903 20:05:26.273682 28625 tablet.cc:1722] T 912eee5b48ef467bbd6bbe34f8aa390c P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250903 20:05:26.273818 27228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec8cea2d64b642689d0bc11295745ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.274297 27211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b437eca9eb0b4c60bf797121a5a8bad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.276412 27693 tablet_bootstrap.cc:492] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:26.276535 27693 ts_tablet_manager.cc:1397] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.044s user 0.001s sys 0.005s
I20250903 20:05:26.277207 27693 raft_consensus.cc:357] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.277388 27693 raft_consensus.cc:383] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.277446 27693 raft_consensus.cc:738] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:26.277662 27693 consensus_queue.cc:260] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.280380 27685 ts_tablet_manager.cc:1918] T ac9b5d4f39db4f05a15316585040d142 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:26.280562 27685 log.cc:1199] T ac9b5d4f39db4f05a15316585040d142 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/ac9b5d4f39db4f05a15316585040d142
I20250903 20:05:26.280961 27685 ts_tablet_manager.cc:1939] T ac9b5d4f39db4f05a15316585040d142 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:26.281330 27693 ts_tablet_manager.cc:1428] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250903 20:05:26.281708 27693 tablet_bootstrap.cc:492] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:26.283258 27692 tablet_bootstrap.cc:492] T 9685cba2f73b412382a69dab2c7f50c1 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:26.283463 27692 ts_tablet_manager.cc:1397] T 9685cba2f73b412382a69dab2c7f50c1 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.054s user 0.004s sys 0.003s
I20250903 20:05:26.284340 27692 raft_consensus.cc:357] T 9685cba2f73b412382a69dab2c7f50c1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.284658 27692 raft_consensus.cc:383] T 9685cba2f73b412382a69dab2c7f50c1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.284766 27692 raft_consensus.cc:738] T 9685cba2f73b412382a69dab2c7f50c1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:26.285063 27692 consensus_queue.cc:260] T 9685cba2f73b412382a69dab2c7f50c1 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.285190 27248 raft_consensus.cc:2466] T c6d67cd0c0304c77ba571b414878dad2 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:26.285800 27462 leader_election.cc:304] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:26.286433 28596 raft_consensus.cc:2802] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:26.286581 28596 raft_consensus.cc:695] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:26.286814 28596 consensus_queue.cc:237] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.292829 27692 ts_tablet_manager.cc:1428] T 9685cba2f73b412382a69dab2c7f50c1 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250903 20:05:26.293678 27692 tablet_bootstrap.cc:492] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:26.296813 27695 tablet_bootstrap.cc:492] T 961ade36f7d14eba91bb16d5eae2e254 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:26.297021 27695 ts_tablet_manager.cc:1397] T 961ade36f7d14eba91bb16d5eae2e254 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.115s user 0.000s sys 0.003s
I20250903 20:05:26.298125 28627 tablet.cc:1722] T 912eee5b48ef467bbd6bbe34f8aa390c P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250903 20:05:26.299887 27693 tablet_bootstrap.cc:654] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:26.298578 27695 raft_consensus.cc:357] T 961ade36f7d14eba91bb16d5eae2e254 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.301355 27695 raft_consensus.cc:383] T 961ade36f7d14eba91bb16d5eae2e254 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.301569 27695 raft_consensus.cc:738] T 961ade36f7d14eba91bb16d5eae2e254 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:26.302000 27695 consensus_queue.cc:260] T 961ade36f7d14eba91bb16d5eae2e254 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.304111 28626 tablet.cc:1722] T 912eee5b48ef467bbd6bbe34f8aa390c P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250903 20:05:26.304381 27692 tablet_bootstrap.cc:654] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:26.313148 27693 tablet_bootstrap.cc:492] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:26.314534 27695 ts_tablet_manager.cc:1428] T 961ade36f7d14eba91bb16d5eae2e254 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.017s user 0.001s sys 0.002s
I20250903 20:05:26.314875 27693 ts_tablet_manager.cc:1397] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.001s
I20250903 20:05:26.315884 27693 raft_consensus.cc:357] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.318128 27695 tablet_bootstrap.cc:492] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:26.319911 27695 tablet_bootstrap.cc:654] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:26.320194 27693 raft_consensus.cc:383] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.320760 27693 raft_consensus.cc:738] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:26.321113 27693 consensus_queue.cc:260] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.321714 27693 ts_tablet_manager.cc:1428] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20250903 20:05:26.322516 27693 tablet_bootstrap.cc:492] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:26.324254 27693 tablet_bootstrap.cc:654] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:26.325618 27692 tablet_bootstrap.cc:492] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:26.326565 27692 ts_tablet_manager.cc:1397] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.000s
I20250903 20:05:26.327766 27692 raft_consensus.cc:357] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.329862 27692 raft_consensus.cc:383] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.330186 27692 raft_consensus.cc:738] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:26.330725 27692 consensus_queue.cc:260] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.332263 27692 ts_tablet_manager.cc:1428] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250903 20:05:26.333537 27692 tablet_bootstrap.cc:492] T b0d79c78e0024b7ba3bb3356be7c8ceb P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:26.337460 28630 raft_consensus.cc:491] T c9e90abef9814338b70a3550ab5686e0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:26.337815 28630 raft_consensus.cc:513] T c9e90abef9814338b70a3550ab5686e0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.339529 27382 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9e90abef9814338b70a3550ab5686e0" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:26.340174 27184 leader_election.cc:343] T c9e90abef9814338b70a3550ab5686e0 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: Tablet not RUNNING: STOPPED
I20250903 20:05:26.336037 27692 tablet_bootstrap.cc:654] T b0d79c78e0024b7ba3bb3356be7c8ceb P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:26.340798 28630 leader_election.cc:290] T c9e90abef9814338b70a3550ab5686e0 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:26.341118 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9e90abef9814338b70a3550ab5686e0" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:26.341627 27183 leader_election.cc:343] T c9e90abef9814338b70a3550ab5686e0 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:26.341763 27183 leader_election.cc:304] T c9e90abef9814338b70a3550ab5686e0 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:26.342370 28630 raft_consensus.cc:2747] T c9e90abef9814338b70a3550ab5686e0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:26.354015 27687 ts_tablet_manager.cc:1918] T c9e90abef9814338b70a3550ab5686e0 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:26.354166 27687 log.cc:1199] T c9e90abef9814338b70a3550ab5686e0 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/c9e90abef9814338b70a3550ab5686e0
I20250903 20:05:26.354516 27687 ts_tablet_manager.cc:1939] T c9e90abef9814338b70a3550ab5686e0 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:26.355051 27687 tablet_replica.cc:331] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:26.355247 27687 raft_consensus.cc:2241] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:26.355509 27687 pending_rounds.cc:70] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528: Trying to abort 1 pending ops.
I20250903 20:05:26.355572 27687 pending_rounds.cc:77] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384977198202880 op_type: NO_OP noop_request { }
I20250903 20:05:26.355666 27687 raft_consensus.cc:2887] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:26.355746 27687 raft_consensus.cc:2270] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:26.356333 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet c9e90abef9814338b70a3550ab5686e0 (table default.Table_5b44669e91a8427c8e7d96b866fdb3b5 [id=96167b0d9f9b47168a9fc59635b7136d]) successfully deleted
I20250903 20:05:26.365239 27687 ts_tablet_manager.cc:1905] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:26.381103 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 55b2cdee20264ae6b64ebe3607fe1ca6
I20250903 20:05:26.383792 28596 raft_consensus.cc:491] T 283a3afed7414a958b2dc4176f00bb01 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:26.383960 28596 raft_consensus.cc:513] T 283a3afed7414a958b2dc4176f00bb01 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.384590 28596 leader_election.cc:290] T 283a3afed7414a958b2dc4176f00bb01 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:26.385773 27462 leader_election.cc:343] T 283a3afed7414a958b2dc4176f00bb01 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 283a3afed7414a958b2dc4176f00bb01
I20250903 20:05:26.387195 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "283a3afed7414a958b2dc4176f00bb01" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:26.387463 27249 raft_consensus.cc:2391] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
I20250903 20:05:26.391672 27462 leader_election.cc:304] T 283a3afed7414a958b2dc4176f00bb01 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:26.392270 28596 raft_consensus.cc:3058] T 283a3afed7414a958b2dc4176f00bb01 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:26.398415 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 658) took 4199 ms. Trace:
I20250903 20:05:26.398550 27685 rpcz_store.cc:276] 0903 20:05:22.198720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.198845 (+ 125us) service_pool.cc:225] Handling call
0903 20:05:26.080741 (+3881896us) ts_tablet_manager.cc:1145] Deleting tablet ac9b5d4f39db4f05a15316585040d142
0903 20:05:26.080752 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:26.149646 (+ 68894us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:26.280299 (+130653us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:26.398388 (+118089us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3878273}
I20250903 20:05:26.399202 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 661) took 4196 ms. Trace:
I20250903 20:05:26.399286 27685 rpcz_store.cc:276] 0903 20:05:22.202860 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.204426 (+ 1566us) service_pool.cc:225] Handling call
0903 20:05:26.398818 (+4194392us) ts_tablet_manager.cc:1145] Deleting tablet 16601b34e4764a8aa14ab95ef8778202
0903 20:05:26.399041 (+ 223us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:26.399176 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4193036}
I20250903 20:05:26.400408 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 55b2cdee20264ae6b64ebe3607fe1ca6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:26.400637 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 659) took 4198 ms. Trace:
I20250903 20:05:26.400736 27685 rpcz_store.cc:276] 0903 20:05:22.202622 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.205279 (+ 2657us) service_pool.cc:225] Handling call
0903 20:05:26.399487 (+4194208us) ts_tablet_manager.cc:1145] Deleting tablet 441bb228c5c54aebb6aba2bc5761bd00
0903 20:05:26.400514 (+ 1027us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:26.400619 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4191353}
I20250903 20:05:26.401070 27685 tablet_replica.cc:331] T 7c398634d6304a12b35c0579fba4c813 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:26.401314 27685 raft_consensus.cc:2241] T 7c398634d6304a12b35c0579fba4c813 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:26.401495 27685 raft_consensus.cc:2270] T 7c398634d6304a12b35c0579fba4c813 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:26.402868 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet ac9b5d4f39db4f05a15316585040d142 (table default.Table_f6b26b9f6ef340989d2250b8d83647c2 [id=74b402d4851f48c2bfd495e82917bb38]) successfully deleted
W20250903 20:05:26.403194 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 16601b34e4764a8aa14ab95ef8778202 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16601b34e4764a8aa14ab95ef8778202 already in progress: creating tablet
W20250903 20:05:26.403455 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 441bb228c5c54aebb6aba2bc5761bd00 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 441bb228c5c54aebb6aba2bc5761bd00 already in progress: creating tablet
I20250903 20:05:26.402189 27685 ts_tablet_manager.cc:1905] T 7c398634d6304a12b35c0579fba4c813 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:26.412828 27296 heartbeater.cc:344] Connected to a master server at 127.26.65.125:44059
I20250903 20:05:26.414016 27223 tablet_service.cc:1515] Processing DeleteTablet for tablet 55b2cdee20264ae6b64ebe3607fe1ca6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:26.439378 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68201d1d224646d3954cb2aa58a3c37e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.440127 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6583a9e2c2748e5bf450ae5d09bf272. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.440248 28568 raft_consensus.cc:491] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:26.440450 28568 raft_consensus.cc:513] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.440665 27489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28afd941430b45ed95d5f9bb49a2220a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.443428 28568 leader_election.cc:290] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:26.448426 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "514e382479014fa7bf1f404efff11c86" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:26.448688 27249 raft_consensus.cc:2466] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:26.449156 27321 leader_election.cc:304] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:26.449535 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "514e382479014fa7bf1f404efff11c86" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:26.450428 27320 leader_election.cc:343] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:26.451923 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_7fbd1210b3c34421a34097b4e769674c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:26.452625 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fbd1210b3c34421a34097b4e769674c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:26.454164 28596 raft_consensus.cc:2747] T 283a3afed7414a958b2dc4176f00bb01 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:26.459331 28579 raft_consensus.cc:2802] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:26.459470 28579 raft_consensus.cc:491] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:26.459545 28579 raft_consensus.cc:3058] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:26.460453 27228 tablet_service.cc:1515] Processing DeleteTablet for tablet c9e90abef9814338b70a3550ab5686e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 607c9a9cbc5540e5bd0fa436fcba564c) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:26.462496 27064 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet a6ffdb473d5745d181a7121d10991705
I20250903 20:05:26.463303 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 17e0864a32d5452d84ac470b55fbaaf4
I20250903 20:05:26.466445 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet ab64604c56914379943c03129ccc2e7c
I20250903 20:05:26.467357 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet ef02f91c0964446daf62ac1dde0330ff
I20250903 20:05:26.467540 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 17e0864a32d5452d84ac470b55fbaaf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:26.468411 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet ab64604c56914379943c03129ccc2e7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:26.468873 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet ef02f91c0964446daf62ac1dde0330ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:26.469190 27221 tablet_service.cc:1515] Processing DeleteTablet for tablet ef02f91c0964446daf62ac1dde0330ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:26.469419 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet ab64604c56914379943c03129ccc2e7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:26.471370 27228 tablet_service.cc:1515] Processing DeleteTablet for tablet 17e0864a32d5452d84ac470b55fbaaf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:26.484903 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 9685cba2f73b412382a69dab2c7f50c1 with delete_type TABLET_DATA_DELETED (replaced by tablet 47c1287bff1145a5b2ab6d14dbd608c7) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:26.485771 27687 ts_tablet_manager.cc:1918] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:26.485867 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet 17e0864a32d5452d84ac470b55fbaaf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:26.486938 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet ab64604c56914379943c03129ccc2e7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:26.487272 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 989545be137341ef8d742bb1119cc51f consensus state has no leader
I20250903 20:05:26.487394 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet ef02f91c0964446daf62ac1dde0330ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:26.487989 27687 log.cc:1199] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/f2cc35f4244a42eeb4a65de749824c11
I20250903 20:05:26.488293 28596 raft_consensus.cc:491] T 0ad64c699b294679b6fec37b5ad23528 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:26.488492 27687 ts_tablet_manager.cc:1939] T f2cc35f4244a42eeb4a65de749824c11 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:26.488444 28596 raft_consensus.cc:513] T 0ad64c699b294679b6fec37b5ad23528 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.489145 28596 leader_election.cc:290] T 0ad64c699b294679b6fec37b5ad23528 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:26.490612 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0ad64c699b294679b6fec37b5ad23528" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:26.490888 27249 raft_consensus.cc:2391] T 0ad64c699b294679b6fec37b5ad23528 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
I20250903 20:05:26.494238 27382 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0ad64c699b294679b6fec37b5ad23528" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:26.496443 27462 leader_election.cc:304] T 0ad64c699b294679b6fec37b5ad23528 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:26.497018 28596 raft_consensus.cc:3058] T 0ad64c699b294679b6fec37b5ad23528 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:26.497601 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet 4157633525034a019fd6a80185892cec with delete_type TABLET_DATA_DELETED (replaced by tablet ed7c564138cc43e5a3bb4e92fb5060b6) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:26.498070 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet 7cc6cf579e2642e99fefed723b0ee900 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:26.498520 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_dd4d357b91334362990bf7b6b3b9e874"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:26.499801 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd4d357b91334362990bf7b6b3b9e874 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:26.499715 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_bcd1ecf3ed1543db8c20807203abf4f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:26.500422 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bcd1ecf3ed1543db8c20807203abf4f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:26.502547 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a2fef2e8171c466e86ce74826ba6e281" } new_table_name: "default.Table_c424d5c674514a0e8471cd35f357ea56" modify_external_catalogs: true
I20250903 20:05:26.502869 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_3ef308651a284cc08daa6de000cc2522"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:26.503476 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ef308651a284cc08daa6de000cc2522 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:26.503839 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5951c39f7b88401e867c14601b0c1ea1" } new_table_name: "default.Table_c53f423b09ee4dc49d8059d14e489484" modify_external_catalogs: true
I20250903 20:05:26.504942 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a1bcf29f05be4350b110670fdbf011d1" } new_table_name: "default.Table_a229319a87544f9ab3d13bdb4245c667" modify_external_catalogs: true
I20250903 20:05:26.515499 28630 raft_consensus.cc:491] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51578fb7bceb4e119b3ab1e58c4f5528)
I20250903 20:05:26.515682 28630 raft_consensus.cc:513] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:26.516674 28630 leader_election.cc:290] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
W20250903 20:05:26.518021 27184 leader_election.cc:343] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: b6b4713936134681aaef1732d46c6ab2
W20250903 20:05:26.519207 27183 leader_election.cc:343] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: b6b4713936134681aaef1732d46c6ab2
I20250903 20:05:26.519315 27183 leader_election.cc:304] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:26.519840 28630 raft_consensus.cc:2747] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:26.523541 27693 tablet_bootstrap.cc:492] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:26.523880 27693 ts_tablet_manager.cc:1397] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.201s user 0.003s sys 0.044s
I20250903 20:05:26.524662 27693 raft_consensus.cc:357] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.524979 27693 raft_consensus.cc:383] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.525075 27693 raft_consensus.cc:738] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:26.525369 27693 consensus_queue.cc:260] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.525967 27693 ts_tablet_manager.cc:1428] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:26.527053 27695 tablet_bootstrap.cc:492] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:26.527060 27692 tablet_bootstrap.cc:492] T b0d79c78e0024b7ba3bb3356be7c8ceb P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:26.527205 27692 ts_tablet_manager.cc:1397] T b0d79c78e0024b7ba3bb3356be7c8ceb P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.194s user 0.015s sys 0.027s
I20250903 20:05:26.529177 27695 ts_tablet_manager.cc:1397] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.211s user 0.026s sys 0.023s
I20250903 20:05:26.529975 27695 raft_consensus.cc:357] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.530252 27695 raft_consensus.cc:383] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.530369 27695 raft_consensus.cc:738] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:26.530714 27695 consensus_queue.cc:260] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.531692 27500 tablet_service.cc:1515] Processing DeleteTablet for tablet 55b2cdee20264ae6b64ebe3607fe1ca6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:26.528473 27692 raft_consensus.cc:357] T b0d79c78e0024b7ba3bb3356be7c8ceb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.532729 27692 raft_consensus.cc:383] T b0d79c78e0024b7ba3bb3356be7c8ceb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.532824 27692 raft_consensus.cc:738] T b0d79c78e0024b7ba3bb3356be7c8ceb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:26.533057 27692 consensus_queue.cc:260] T b0d79c78e0024b7ba3bb3356be7c8ceb P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.535755 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_70cfd4abdbdf43e2b7665f787126a656"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:26.536228 27695 ts_tablet_manager.cc:1428] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250903 20:05:26.536577 27695 tablet_bootstrap.cc:492] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
W20250903 20:05:26.537133 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70cfd4abdbdf43e2b7665f787126a656 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:26.538184 27695 tablet_bootstrap.cc:654] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:26.538663 27693 tablet_bootstrap.cc:492] T b0d79c78e0024b7ba3bb3356be7c8ceb P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:26.543187 27693 tablet_bootstrap.cc:654] T b0d79c78e0024b7ba3bb3356be7c8ceb P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:26.542150 27692 ts_tablet_manager.cc:1428] T b0d79c78e0024b7ba3bb3356be7c8ceb P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250903 20:05:26.546674 27692 tablet_bootstrap.cc:492] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:26.549172 27692 tablet_bootstrap.cc:654] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:26.556347 28596 raft_consensus.cc:2747] T 0ad64c699b294679b6fec37b5ad23528 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:26.556879 27684 ts_tablet_manager.cc:1918] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:26.556988 27684 log.cc:1199] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/8d2d64f103fc44d7b390559b92cbf4b4
I20250903 20:05:26.557381 27684 ts_tablet_manager.cc:1939] T 8d2d64f103fc44d7b390559b92cbf4b4 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:26.561321 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet f2cc35f4244a42eeb4a65de749824c11 (table default.Table_78e7a5b76ebc469396d8a3d670ab63ee [id=fb43441ca8f2415d99a628e81ef1642f]) successfully deleted
I20250903 20:05:26.552827 28579 raft_consensus.cc:513] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.590725 28579 leader_election.cc:290] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:26.559569 27687 tablet_replica.cc:331] T a0293a69f1c246d08d414e95729b4ab4 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:26.591732 27687 raft_consensus.cc:2241] T a0293a69f1c246d08d414e95729b4ab4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:26.591920 27687 raft_consensus.cc:2270] T a0293a69f1c246d08d414e95729b4ab4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:26.592485 27687 ts_tablet_manager.cc:1905] T a0293a69f1c246d08d414e95729b4ab4 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:26.571300 28568 raft_consensus.cc:491] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:26.593761 28568 raft_consensus.cc:513] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.594477 28568 leader_election.cc:290] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:26.595165 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b3bfc63aa0b4e56bec189e834dc6a8d" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:26.595898 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "514e382479014fa7bf1f404efff11c86" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
W20250903 20:05:26.596768 27320 leader_election.cc:343] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:26.596990 27320 leader_election.cc:343] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:26.539593 28630 consensus_queue.cc:1035] T d859c0222c3647dab3dddd105e4cf0fc P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:26.552241 27685 ts_tablet_manager.cc:1918] T 7c398634d6304a12b35c0579fba4c813 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:26.600257 27685 log.cc:1199] T 7c398634d6304a12b35c0579fba4c813 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/7c398634d6304a12b35c0579fba4c813
I20250903 20:05:26.600605 27685 ts_tablet_manager.cc:1939] T 7c398634d6304a12b35c0579fba4c813 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:26.600927 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 657) took 4402 ms. Trace:
I20250903 20:05:26.601007 27685 rpcz_store.cc:276] 0903 20:05:22.198607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.201351 (+ 2744us) service_pool.cc:225] Handling call
0903 20:05:26.401017 (+4199666us) ts_tablet_manager.cc:1145] Deleting tablet 7c398634d6304a12b35c0579fba4c813
0903 20:05:26.401026 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:26.457103 (+ 56077us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:26.552028 (+ 94925us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:26.600911 (+ 48883us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4175436}
I20250903 20:05:26.601370 27685 tablet_replica.cc:331] T 8d2d64f103fc44d7b390559b92cbf4b4 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:26.601630 27685 raft_consensus.cc:2241] T 8d2d64f103fc44d7b390559b92cbf4b4 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:26.601807 27685 raft_consensus.cc:2270] T 8d2d64f103fc44d7b390559b92cbf4b4 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:26.602131 27057 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_f1f4f57d96f04f0caff06d9a642f5c46 [id=fdc7899c8b334d309b834ae1daa14fa0] alter complete (version 4)
I20250903 20:05:26.602849 27057 master_service.cc:948] ReplaceTablet: received request to replace tablet 567630e2a93645e58f9bf05ffff87b32 from {username='slave'} at 127.0.0.1:45936
W20250903 20:05:26.603385 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6d4b8d4bf29e408cbdf9557c8b0d8a4b consensus state has no leader
I20250903 20:05:26.603950 27685 ts_tablet_manager.cc:1905] T 8d2d64f103fc44d7b390559b92cbf4b4 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:26.606323 28579 raft_consensus.cc:491] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:26.606508 28579 raft_consensus.cc:513] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.607236 28579 leader_election.cc:290] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:26.592898 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "514e382479014fa7bf1f404efff11c86" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:26.607492 27249 raft_consensus.cc:3058] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:26.595901 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b3bfc63aa0b4e56bec189e834dc6a8d" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:26.608006 27246 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685cba2f73b412382a69dab2c7f50c1" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:26.608176 27246 raft_consensus.cc:2466] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
W20250903 20:05:26.608556 27321 leader_election.cc:343] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:26.608742 27321 leader_election.cc:304] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:26.609282 27321 leader_election.cc:304] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:26.609565 28567 raft_consensus.cc:2747] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:26.610397 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685cba2f73b412382a69dab2c7f50c1" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:26.610589 27524 raft_consensus.cc:2466] T 9685cba2f73b412382a69dab2c7f50c1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:26.609611 28579 raft_consensus.cc:2802] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:26.611953 28579 raft_consensus.cc:491] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:26.612075 28579 raft_consensus.cc:3058] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:26.616856 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 7c398634d6304a12b35c0579fba4c813 (table default.Table_14af7742726b4562a02fd2c9b2895277 [id=6d72901e5102476890f466fd406a7127]) successfully deleted
I20250903 20:05:26.617288 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet 961ade36f7d14eba91bb16d5eae2e254 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:26.622503 26890 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 77688.6us
I20250903 20:05:26.625553 27695 tablet_bootstrap.cc:492] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:26.627192 28568 raft_consensus.cc:491] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:26.627363 28568 raft_consensus.cc:513] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.628101 28568 leader_election.cc:290] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:26.628572 27695 ts_tablet_manager.cc:1397] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.092s user 0.001s sys 0.003s
I20250903 20:05:26.628875 27246 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6788b1683f984949816edfc5b68de8db" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:26.626929 28630 raft_consensus.cc:491] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:26.629297 28630 raft_consensus.cc:513] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.629384 27695 raft_consensus.cc:357] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.629500 27695 raft_consensus.cc:383] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.629545 27695 raft_consensus.cc:738] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:26.629727 27695 consensus_queue.cc:260] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.629902 28630 leader_election.cc:290] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:26.630326 27695 ts_tablet_manager.cc:1428] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:26.630671 27695 tablet_bootstrap.cc:492] T 6788b1683f984949816edfc5b68de8db P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:26.630617 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bcb9d5f010f4297bab78c0183889238" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:26.631052 28596 raft_consensus.cc:491] T b0d79c78e0024b7ba3bb3356be7c8ceb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:26.631213 28596 raft_consensus.cc:513] T b0d79c78e0024b7ba3bb3356be7c8ceb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.631860 28596 leader_election.cc:290] T b0d79c78e0024b7ba3bb3356be7c8ceb P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:26.632020 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bcb9d5f010f4297bab78c0183889238" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:26.632232 27381 raft_consensus.cc:2466] T 1bcb9d5f010f4297bab78c0183889238 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
W20250903 20:05:26.631357 27183 leader_election.cc:343] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:26.632967 27184 leader_election.cc:304] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:26.633385 28522 raft_consensus.cc:2802] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:26.633462 28522 raft_consensus.cc:491] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:26.633525 28522 raft_consensus.cc:3058] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:26.634130 27321 leader_election.cc:343] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:26.634536 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet a0293a69f1c246d08d414e95729b4ab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:26.635473 27695 tablet_bootstrap.cc:654] T 6788b1683f984949816edfc5b68de8db P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:26.635277 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6788b1683f984949816edfc5b68de8db" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:26.636668 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 780) took 1396 ms. Trace:
I20250903 20:05:26.636713 27524 raft_consensus.cc:2466] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:26.636757 27684 rpcz_store.cc:276] 0903 20:05:25.239999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:25.240063 (+ 64us) service_pool.cc:225] Handling call
0903 20:05:26.266683 (+1026620us) ts_tablet_manager.cc:1145] Deleting tablet 8d2d64f103fc44d7b390559b92cbf4b4
0903 20:05:26.266693 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:26.359280 (+ 92587us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:26.556731 (+197451us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:26.636650 (+ 79919us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1026329}
I20250903 20:05:26.637286 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b0d79c78e0024b7ba3bb3356be7c8ceb" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:26.637403 27684 tablet_replica.cc:331] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:26.637609 27684 raft_consensus.cc:2241] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:26.637944 27684 raft_consensus.cc:2270] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:26.638011 27462 leader_election.cc:343] T b0d79c78e0024b7ba3bb3356be7c8ceb P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:26.638396 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 8d2d64f103fc44d7b390559b92cbf4b4 (table default.Table_bb8c68be79034a5c8469862c43953131 [id=f0171ee670294ce58ab84483fd71ab7e]) successfully deleted
I20250903 20:05:26.638587 27320 leader_election.cc:304] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:26.639184 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet 9685cba2f73b412382a69dab2c7f50c1 with delete_type TABLET_DATA_DELETED (replaced by tablet 47c1287bff1145a5b2ab6d14dbd608c7) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:26.636714 27489 tablet_service.cc:1515] Processing DeleteTablet for tablet 6788b1683f984949816edfc5b68de8db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:26.639602 28568 raft_consensus.cc:2802] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:26.639715 28568 raft_consensus.cc:491] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:26.639808 28568 raft_consensus.cc:3058] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:26.636488 27246 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b0d79c78e0024b7ba3bb3356be7c8ceb" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:26.640969 27462 leader_election.cc:343] T b0d79c78e0024b7ba3bb3356be7c8ceb P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:26.641191 27462 leader_election.cc:304] T b0d79c78e0024b7ba3bb3356be7c8ceb P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:26.641677 27684 ts_tablet_manager.cc:1905] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:26.645128 27695 tablet_bootstrap.cc:492] T 6788b1683f984949816edfc5b68de8db P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:26.645715 28598 raft_consensus.cc:2747] T b0d79c78e0024b7ba3bb3356be7c8ceb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:26.646337 27692 tablet_bootstrap.cc:492] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:26.646478 27692 ts_tablet_manager.cc:1397] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.100s user 0.008s sys 0.008s
I20250903 20:05:26.647226 27692 raft_consensus.cc:357] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.647399 27692 raft_consensus.cc:383] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.647445 27692 raft_consensus.cc:738] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:26.647797 27692 consensus_queue.cc:260] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:26.648247 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 794865d716134382a939186d7144c4bb consensus state has no leader
W20250903 20:05:26.650880 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b2de48441b5c43c091ac43ba02bf3fd6 consensus state has no leader
I20250903 20:05:26.652266 27687 ts_tablet_manager.cc:1918] T a0293a69f1c246d08d414e95729b4ab4 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:26.652386 27687 log.cc:1199] T a0293a69f1c246d08d414e95729b4ab4 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/a0293a69f1c246d08d414e95729b4ab4
I20250903 20:05:26.652673 27687 ts_tablet_manager.cc:1939] T a0293a69f1c246d08d414e95729b4ab4 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:26.653265 27692 ts_tablet_manager.cc:1428] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250903 20:05:26.654000 27692 tablet_bootstrap.cc:492] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:26.658645 27695 ts_tablet_manager.cc:1397] T 6788b1683f984949816edfc5b68de8db P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20250903 20:05:26.661649 28579 raft_consensus.cc:513] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.663239 27246 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685cba2f73b412382a69dab2c7f50c1" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:26.663446 27246 raft_consensus.cc:3058] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:26.663586 28579 leader_election.cc:290] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:26.663668 27695 raft_consensus.cc:357] T 6788b1683f984949816edfc5b68de8db P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.663928 27695 raft_consensus.cc:383] T 6788b1683f984949816edfc5b68de8db P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.664050 27695 raft_consensus.cc:738] T 6788b1683f984949816edfc5b68de8db P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:26.664382 27695 consensus_queue.cc:260] T 6788b1683f984949816edfc5b68de8db P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.665645 27695 ts_tablet_manager.cc:1428] T 6788b1683f984949816edfc5b68de8db P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:26.666038 27695 tablet_bootstrap.cc:492] T a4c582b441f6467c99ba0b090abcf4cc P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:26.667201 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685cba2f73b412382a69dab2c7f50c1" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:26.667428 27524 raft_consensus.cc:3058] T 9685cba2f73b412382a69dab2c7f50c1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:26.667645 27695 tablet_bootstrap.cc:654] T a4c582b441f6467c99ba0b090abcf4cc P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:26.669498 27692 tablet_bootstrap.cc:654] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:26.661279 27249 raft_consensus.cc:2466] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:26.671419 27321 leader_election.cc:304] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:26.672008 28596 raft_consensus.cc:491] T 7cc6cf579e2642e99fefed723b0ee900 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:26.672237 27693 tablet_bootstrap.cc:492] T b0d79c78e0024b7ba3bb3356be7c8ceb P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:26.672168 28596 raft_consensus.cc:513] T 7cc6cf579e2642e99fefed723b0ee900 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.672366 27693 ts_tablet_manager.cc:1397] T b0d79c78e0024b7ba3bb3356be7c8ceb P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.134s user 0.001s sys 0.002s
I20250903 20:05:26.672883 28596 leader_election.cc:290] T 7cc6cf579e2642e99fefed723b0ee900 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:26.673074 27693 raft_consensus.cc:357] T b0d79c78e0024b7ba3bb3356be7c8ceb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.673337 27693 raft_consensus.cc:383] T b0d79c78e0024b7ba3bb3356be7c8ceb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.673431 27693 raft_consensus.cc:738] T b0d79c78e0024b7ba3bb3356be7c8ceb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:26.674136 27693 consensus_queue.cc:260] T b0d79c78e0024b7ba3bb3356be7c8ceb P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.674841 28567 raft_consensus.cc:2802] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:26.674976 28567 raft_consensus.cc:695] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:26.675230 27693 ts_tablet_manager.cc:1428] T b0d79c78e0024b7ba3bb3356be7c8ceb P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:26.675230 28567 consensus_queue.cc:237] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.675712 27693 tablet_bootstrap.cc:492] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
W20250903 20:05:26.677343 27462 leader_election.cc:343] T 7cc6cf579e2642e99fefed723b0ee900 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 7cc6cf579e2642e99fefed723b0ee900
I20250903 20:05:26.678342 27693 tablet_bootstrap.cc:654] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:26.678617 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cc6cf579e2642e99fefed723b0ee900" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:26.679237 27249 raft_consensus.cc:2391] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
I20250903 20:05:26.680429 27462 leader_election.cc:304] T 7cc6cf579e2642e99fefed723b0ee900 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:26.681032 28596 raft_consensus.cc:3058] T 7cc6cf579e2642e99fefed723b0ee900 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:26.682574 27695 tablet_bootstrap.cc:492] T a4c582b441f6467c99ba0b090abcf4cc P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:26.682723 27695 ts_tablet_manager.cc:1397] T a4c582b441f6467c99ba0b090abcf4cc P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20250903 20:05:26.685809 27695 raft_consensus.cc:357] T a4c582b441f6467c99ba0b090abcf4cc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.686046 27695 raft_consensus.cc:383] T a4c582b441f6467c99ba0b090abcf4cc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.686118 27695 raft_consensus.cc:738] T a4c582b441f6467c99ba0b090abcf4cc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:26.686815 27695 consensus_queue.cc:260] T a4c582b441f6467c99ba0b090abcf4cc P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.687522 27693 tablet_bootstrap.cc:492] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:26.687593 27695 ts_tablet_manager.cc:1428] T a4c582b441f6467c99ba0b090abcf4cc P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250903 20:05:26.687953 27695 tablet_bootstrap.cc:492] T 33fb6b357bda43e7abeaed5634183ef6 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:26.688167 27693 ts_tablet_manager.cc:1397] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.005s
I20250903 20:05:26.690776 27692 tablet_bootstrap.cc:492] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:26.691020 27692 ts_tablet_manager.cc:1397] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.001s
I20250903 20:05:26.692153 27692 raft_consensus.cc:357] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.692476 27692 raft_consensus.cc:383] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.692601 27692 raft_consensus.cc:738] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:26.692914 27692 consensus_queue.cc:260] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.693535 27692 ts_tablet_manager.cc:1428] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250903 20:05:26.694056 27692 tablet_bootstrap.cc:492] T 1bcb9d5f010f4297bab78c0183889238 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:26.694306 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_fa60556c9a9b433a9eff6f68ee824909"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:26.689064 27693 raft_consensus.cc:357] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.694912 27693 raft_consensus.cc:383] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.696836 27692 tablet_bootstrap.cc:654] T 1bcb9d5f010f4297bab78c0183889238 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:26.697121 27693 raft_consensus.cc:738] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:26.699180 27249 raft_consensus.cc:1273] T 516830fc42e84af693b3af52caee30ee P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Refusing update from remote peer e72ef1dbcc5143cf9f04a808f101c80f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 20:05:26.699226 27693 consensus_queue.cc:260] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.699940 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet a0293a69f1c246d08d414e95729b4ab4 (table default.Table_61bb115ae97549f7ab9c44e52b75acf0 [id=4ce7195efb6e432e9b232b8a4749179f]) successfully deleted
I20250903 20:05:26.700172 27381 raft_consensus.cc:1273] T 516830fc42e84af693b3af52caee30ee P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Refusing update from remote peer e72ef1dbcc5143cf9f04a808f101c80f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 20:05:26.699945 27693 ts_tablet_manager.cc:1428] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250903 20:05:26.700402 27059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_eb74cc68ea8b4ab0bb7eb3937ac1aa4d" } modify_external_catalogs: true
I20250903 20:05:26.701484 27687 tablet_replica.cc:331] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:26.701680 27687 raft_consensus.cc:2241] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:26.701910 27687 pending_rounds.cc:70] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528: Trying to abort 1 pending ops.
I20250903 20:05:26.701992 27687 pending_rounds.cc:77] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384979664347136 op_type: NO_OP noop_request { }
I20250903 20:05:26.702080 27687 raft_consensus.cc:2887] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:26.701982 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_4e5f622e7f16447aa8286aca4aa3c983"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:26.702162 27687 raft_consensus.cc:2270] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:26.704579 27695 tablet_bootstrap.cc:654] T 33fb6b357bda43e7abeaed5634183ef6 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:26.707603 28579 raft_consensus.cc:491] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:26.707775 28579 raft_consensus.cc:513] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.709144 28579 leader_election.cc:290] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:26.712517 27687 ts_tablet_manager.cc:1905] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:26.716506 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_8373ad89ca4a41c0ab4d117a017b543c" } new_table_name: "default.Table_01bdabab48594c29a7af573f7062658f" modify_external_catalogs: true
I20250903 20:05:26.719247 27061 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 3b3bfc63aa0b4e56bec189e834dc6a8d (table default.Table_9b20469406c94986ab496ea3e678b586 [id=6d6f5e8cadf9445d8695c075cfabb2ea]). Expected version 1 got 0
I20250903 20:05:26.719857 27693 tablet_bootstrap.cc:492] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:26.722265 27061 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 0f5b87c1c4a24f8abdbfd6194f3b2c4a (table default.Table_6e9cf5ead5a24831a6a14d131dd8397e [id=524a27cbe1ad4c10aa2fd4f4074105c6]). Expected version 3 got 0
I20250903 20:05:26.722440 27693 tablet_bootstrap.cc:654] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:26.700711 28547 consensus_queue.cc:1035] T 516830fc42e84af693b3af52caee30ee P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:26.709234 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a4c582b441f6467c99ba0b090abcf4cc" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:26.725239 27320 leader_election.cc:343] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:26.725878 28598 consensus_queue.cc:1035] T 516830fc42e84af693b3af52caee30ee P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:26.723418 27057 master_service.cc:948] ReplaceTablet: received request to replace tablet 5b26ed39f2144610b786f49e62e6f0a3 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:26.730190 28649 raft_consensus.cc:491] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:26.730449 28649 raft_consensus.cc:513] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.731592 27059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_eb74cc68ea8b4ab0bb7eb3937ac1aa4d" } modify_external_catalogs: true
I20250903 20:05:26.697953 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_895e601507754fd399673ccb24b57a17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:26.732762 27184 leader_election.cc:343] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: a0293a69f1c246d08d414e95729b4ab4
W20250903 20:05:26.732849 27051 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 548b50819f694fa3ad508da9f442a99c (table default.Table_b838fda30d4043a5ad578ec527c1d105 [id=1eb6b859ca0049699466cc72e5a2b6f7], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 548b50819f694fa3ad508da9f442a99c consensus state has no leader
W20250903 20:05:26.735108 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 3d875d29e66d48ed872e38d20f30d9c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d875d29e66d48ed872e38d20f30d9c0 already in progress: creating tablet
W20250903 20:05:26.735427 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 24ddae9179cd4240914cbbae31d8b851 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24ddae9179cd4240914cbbae31d8b851 already in progress: creating tablet
I20250903 20:05:26.735615 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0293a69f1c246d08d414e95729b4ab4" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:26.735809 28649 leader_election.cc:290] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:26.735823 27522 raft_consensus.cc:2466] T a0293a69f1c246d08d414e95729b4ab4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
W20250903 20:05:26.735853 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e5f622e7f16447aa8286aca4aa3c983 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:26.736101 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa60556c9a9b433a9eff6f68ee824909 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:26.736246 27183 leader_election.cc:304] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:26.736578 28630 raft_consensus.cc:2802] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250903 20:05:26.736619 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 45f7068bafaf48d290ded9d1272c0e46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 45f7068bafaf48d290ded9d1272c0e46 already in progress: creating tablet
I20250903 20:05:26.736675 28630 raft_consensus.cc:491] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:26.736738 28630 raft_consensus.cc:3058] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:26.736905 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet b6663ba96faa44b88fe8b6def4ff6951 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6663ba96faa44b88fe8b6def4ff6951 already in progress: creating tablet
I20250903 20:05:26.735946 28649 consensus_queue.cc:1035] T d859c0222c3647dab3dddd105e4cf0fc P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250903 20:05:26.737205 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_895e601507754fd399673ccb24b57a17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:26.737759 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 45d9b6ff520348628d19671127b9426c because the tablet was not found. No further retry: Not found: Tablet not found: 45d9b6ff520348628d19671127b9426c
W20250903 20:05:26.737978 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet d93258c032524b71997389b81846a5be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d93258c032524b71997389b81846a5be already in progress: creating tablet
I20250903 20:05:26.739570 27351 tablet_service.cc:1468] Processing CreateTablet for tablet 3590506b3fd44a6c84aa469c14ed3e9f (DEFAULT_TABLE table=default.Table_2623cd1b586e40818f657e9fc9f5f570 [id=7b22e0e04f1b4432996ecde7fd4e60c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.740183 27064 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet b0d79c78e0024b7ba3bb3356be7c8ceb (table default.Table_94e9e32ef1134f07ade1cdf8da69ead6 [id=e248f9c4b9c9406395aaea7ba3feb9ba]). Expected version 1 got 0
I20250903 20:05:26.740516 27693 tablet_bootstrap.cc:492] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:26.740700 27693 ts_tablet_manager.cc:1397] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.004s
I20250903 20:05:26.744264 27692 tablet_bootstrap.cc:492] T 1bcb9d5f010f4297bab78c0183889238 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:26.744448 27692 ts_tablet_manager.cc:1397] T 1bcb9d5f010f4297bab78c0183889238 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.050s user 0.001s sys 0.002s
I20250903 20:05:26.745649 28522 raft_consensus.cc:513] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.746384 28522 leader_election.cc:290] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:26.746822 27685 ts_tablet_manager.cc:1918] T 8d2d64f103fc44d7b390559b92cbf4b4 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:26.746917 27685 log.cc:1199] T 8d2d64f103fc44d7b390559b92cbf4b4 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/8d2d64f103fc44d7b390559b92cbf4b4
I20250903 20:05:26.747227 27685 ts_tablet_manager.cc:1939] T 8d2d64f103fc44d7b390559b92cbf4b4 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:26.747582 27524 raft_consensus.cc:2466] T 9685cba2f73b412382a69dab2c7f50c1 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:26.748116 27246 raft_consensus.cc:2466] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:26.749794 28630 raft_consensus.cc:513] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.750548 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bcb9d5f010f4297bab78c0183889238" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:26.750991 27320 leader_election.cc:304] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:26.751333 28579 raft_consensus.cc:2802] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:26.751458 28579 raft_consensus.cc:695] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:26.751729 28579 consensus_queue.cc:237] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.753276 28568 raft_consensus.cc:513] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.753919 28568 leader_election.cc:290] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:26.753948 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bcb9d5f010f4297bab78c0183889238" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:26.754743 27381 raft_consensus.cc:3058] T 1bcb9d5f010f4297bab78c0183889238 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:26.754714 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6788b1683f984949816edfc5b68de8db" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:26.754889 27526 raft_consensus.cc:3058] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:26.755846 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 665) took 4377 ms. Trace:
I20250903 20:05:26.755961 27685 rpcz_store.cc:276] 0903 20:05:22.378570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.378691 (+ 121us) service_pool.cc:225] Handling call
0903 20:05:26.601309 (+4222618us) ts_tablet_manager.cc:1145] Deleting tablet 8d2d64f103fc44d7b390559b92cbf4b4
0903 20:05:26.601322 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:26.652945 (+ 51623us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:26.746694 (+ 93749us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:26.755822 (+ 9128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4222397}
I20250903 20:05:26.756476 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 677) took 4318 ms. Trace:
I20250903 20:05:26.756556 27685 rpcz_store.cc:276] 0903 20:05:22.437673 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.437763 (+ 90us) service_pool.cc:225] Handling call
0903 20:05:26.756329 (+4318566us) ts_tablet_manager.cc:1145] Deleting tablet 53881ad30bc843839008ace4012a3f70
0903 20:05:26.756340 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:26.756458 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4316980}
I20250903 20:05:26.757911 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 679) took 4320 ms. Trace:
I20250903 20:05:26.757999 27685 rpcz_store.cc:276] 0903 20:05:22.437873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.437927 (+ 54us) service_pool.cc:225] Handling call
0903 20:05:26.757787 (+4319860us) ts_tablet_manager.cc:1145] Deleting tablet 4e99665f920f4de590bd6d9e918c7182
0903 20:05:26.757797 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:26.757893 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4318384}
I20250903 20:05:26.758724 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 675) took 4321 ms. Trace:
I20250903 20:05:26.758803 27685 rpcz_store.cc:276] 0903 20:05:22.437499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.459529 (+ 22030us) service_pool.cc:225] Handling call
0903 20:05:26.758183 (+4298654us) ts_tablet_manager.cc:1145] Deleting tablet 9a71290c27ac4a22bd9aa8e14e5834f5
0903 20:05:26.758191 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:26.758699 (+ 508us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4298481}
I20250903 20:05:26.759191 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 676) took 4321 ms. Trace:
I20250903 20:05:26.759259 27685 rpcz_store.cc:276] 0903 20:05:22.437590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.459748 (+ 22158us) service_pool.cc:225] Handling call
0903 20:05:26.759008 (+4299260us) ts_tablet_manager.cc:1145] Deleting tablet 8093e36d79894ed3aa8e92c0bca4f93d
0903 20:05:26.759015 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:26.759180 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4299129}
I20250903 20:05:26.759517 27685 tablet_replica.cc:331] T 885fff47fea14b84ac6b34a57b827b4b P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:26.759544 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 8d2d64f103fc44d7b390559b92cbf4b4 (table default.Table_bb8c68be79034a5c8469862c43953131 [id=f0171ee670294ce58ab84483fd71ab7e]) successfully deleted
I20250903 20:05:26.759707 27685 raft_consensus.cc:2241] T 885fff47fea14b84ac6b34a57b827b4b P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:26.759881 27685 raft_consensus.cc:2270] T 885fff47fea14b84ac6b34a57b827b4b P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:26.759939 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 53881ad30bc843839008ace4012a3f70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53881ad30bc843839008ace4012a3f70 already in progress: creating tablet
I20250903 20:05:26.760464 27685 ts_tablet_manager.cc:1905] T 885fff47fea14b84ac6b34a57b827b4b P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:26.761574 27351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3590506b3fd44a6c84aa469c14ed3e9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.762779 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet f2cc35f4244a42eeb4a65de749824c11 with delete_type TABLET_DATA_DELETED (replaced by tablet fc634116a78f45dd833034ddda90e9bf) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:26.763094 27184 leader_election.cc:343] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: a0293a69f1c246d08d414e95729b4ab4
I20250903 20:05:26.766711 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e8b42e26d9e6426e96d80135d5015108" } modify_external_catalogs: true
I20250903 20:05:26.767460 28630 leader_election.cc:290] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:26.767859 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0293a69f1c246d08d414e95729b4ab4" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:26.768069 27518 raft_consensus.cc:3058] T a0293a69f1c246d08d414e95729b4ab4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:26.769011 27684 ts_tablet_manager.cc:1918] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:26.769126 27684 log.cc:1199] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/fe902f1b58924be1ac3a5585413f7789
I20250903 20:05:26.769508 27684 ts_tablet_manager.cc:1939] T fe902f1b58924be1ac3a5585413f7789 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:26.770105 27489 tablet_service.cc:1468] Processing CreateTablet for tablet 3590506b3fd44a6c84aa469c14ed3e9f (DEFAULT_TABLE table=default.Table_2623cd1b586e40818f657e9fc9f5f570 [id=7b22e0e04f1b4432996ecde7fd4e60c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.770188 27493 tablet_service.cc:1468] Processing CreateTablet for tablet edac2355f67045a6acffaae723d9186f (DEFAULT_TABLE table=default.Table_7afeb3c17d814683916ce7fc2788310e [id=82f2d131a6424b678e49c46e530615fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.770586 27489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3590506b3fd44a6c84aa469c14ed3e9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.770848 27500 tablet_service.cc:1468] Processing CreateTablet for tablet f3185f9cf7c746418beeefb9e34758ff (DEFAULT_TABLE table=default.Table_0e14fb71c06946bdabb837fff47ec0e9 [id=bb5a0e9032ca4919bf925978f1329e74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.771245 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3185f9cf7c746418beeefb9e34758ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.774466 27499 tablet_service.cc:1468] Processing CreateTablet for tablet 475d05d087c34159b093b92cc308bcaf (DEFAULT_TABLE table=default.Table_def69a11d277479cbfc58718d39b63ac [id=1b66ec0c597848afbea739014dada98b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.774889 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 475d05d087c34159b093b92cc308bcaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.745443 27692 raft_consensus.cc:357] T 1bcb9d5f010f4297bab78c0183889238 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.776433 27692 raft_consensus.cc:383] T 1bcb9d5f010f4297bab78c0183889238 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.776541 27692 raft_consensus.cc:738] T 1bcb9d5f010f4297bab78c0183889238 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:26.776777 27692 consensus_queue.cc:260] T 1bcb9d5f010f4297bab78c0183889238 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.776959 27524 raft_consensus.cc:3058] T 1bcb9d5f010f4297bab78c0183889238 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:26.778570 27497 tablet_service.cc:1468] Processing CreateTablet for tablet 4120e1abd8434abb9cd92e3691d64cc7 (DEFAULT_TABLE table=default.Table_3fe8a44f17364b249736f1722bd1c9a5 [id=0f8ffc9a5d6e49c885561a232d47ec3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.778980 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4120e1abd8434abb9cd92e3691d64cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.742069 27693 raft_consensus.cc:357] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:26.779704 27693 raft_consensus.cc:383] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.779817 27246 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a4c582b441f6467c99ba0b090abcf4cc" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:26.779966 27693 raft_consensus.cc:738] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:26.780022 27246 raft_consensus.cc:2466] T a4c582b441f6467c99ba0b090abcf4cc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:26.780274 27693 consensus_queue.cc:260] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:26.780494 27246 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6788b1683f984949816edfc5b68de8db" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:26.780638 27246 raft_consensus.cc:3058] T 6788b1683f984949816edfc5b68de8db P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:26.780896 27693 ts_tablet_manager.cc:1428] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.040s user 0.002s sys 0.001s
I20250903 20:05:26.781255 27693 tablet_bootstrap.cc:492] T 2d9c1642fec841d497ff3496dd4564db P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:26.782570 27490 tablet_service.cc:1468] Processing CreateTablet for tablet 2c6a2e2e2ec34f8082a17048ca0d18b0 (DEFAULT_TABLE table=default.Table_a41978d6a03a4bfe8f2b4cbd5f615ffb [id=0bac991007ef440580f3df9d5df7b6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.783434 28579 raft_consensus.cc:491] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:26.783581 28579 raft_consensus.cc:513] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.783885 27321 leader_election.cc:304] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:26.784014 27490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c6a2e2e2ec34f8082a17048ca0d18b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.785030 28568 raft_consensus.cc:2802] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:26.785115 28568 raft_consensus.cc:491] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:26.785187 28568 raft_consensus.cc:3058] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:26.785466 27693 tablet_bootstrap.cc:654] T 2d9c1642fec841d497ff3496dd4564db P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:26.787719 27503 tablet_service.cc:1468] Processing CreateTablet for tablet 53b00774b745462091321ea47fcc2114 (DEFAULT_TABLE table=default.Table_0955d4bd84b645e3b796679b761bbba7 [id=fd02c556cd3c4c42941d24cb61939a6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.788141 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53b00774b745462091321ea47fcc2114. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.788304 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "961ade36f7d14eba91bb16d5eae2e254" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:26.788415 28579 leader_election.cc:290] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:26.788511 27248 raft_consensus.cc:2466] T 961ade36f7d14eba91bb16d5eae2e254 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:26.788878 27321 leader_election.cc:304] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:26.789213 28579 raft_consensus.cc:2802] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:26.789281 28579 raft_consensus.cc:491] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:26.789350 28579 raft_consensus.cc:3058] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:26.790418 27523 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "961ade36f7d14eba91bb16d5eae2e254" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:26.791224 27320 leader_election.cc:343] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:26.793017 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 27c21921025a464681bf157b2627d351 (DEFAULT_TABLE table=default.Table_5e14429fe4e340aaa9c2d5ad605c9a92 [id=6f17187b24224598855f0f8bbe2fe630]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.793474 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27c21921025a464681bf157b2627d351. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.742368 27064 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_71f1cf3f76a8469695fdc49c38ba8771" } modify_external_catalogs: true
I20250903 20:05:26.797573 27064 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_71f1cf3f76a8469695fdc49c38ba8771" } modify_external_catalogs: true
I20250903 20:05:26.803860 28626 tablet.cc:1722] T 516830fc42e84af693b3af52caee30ee P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250903 20:05:26.805760 27065 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_307490b0006e4ed493ee8b2e23b7877a" } modify_external_catalogs: true
I20250903 20:05:26.806183 27498 tablet_service.cc:1468] Processing CreateTablet for tablet 2ce6e8b068bf4c06916f0c23da1d5824 (DEFAULT_TABLE table=default.Table_257827f9b2804d37aa0556a8affcf93e [id=df3b81c7417c47a1b2601f04d557a122]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.806174 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e1977f95b5174bb2b40623d225aff528" } modify_external_catalogs: true
I20250903 20:05:26.806614 27498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ce6e8b068bf4c06916f0c23da1d5824. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.806778 27695 tablet_bootstrap.cc:492] T 33fb6b357bda43e7abeaed5634183ef6 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:26.806911 27695 ts_tablet_manager.cc:1397] T 33fb6b357bda43e7abeaed5634183ef6 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.119s user 0.009s sys 0.002s
I20250903 20:05:26.807739 27695 raft_consensus.cc:357] T 33fb6b357bda43e7abeaed5634183ef6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.807904 27695 raft_consensus.cc:383] T 33fb6b357bda43e7abeaed5634183ef6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.807978 27695 raft_consensus.cc:738] T 33fb6b357bda43e7abeaed5634183ef6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:26.808907 28625 tablet.cc:1722] T 516830fc42e84af693b3af52caee30ee P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250903 20:05:26.810187 27502 tablet_service.cc:1468] Processing CreateTablet for tablet d60662d1c2284f22a0e191fa70520ce2 (DEFAULT_TABLE table=default.Table_f073b600fed2421f9a25aca9c22751e3 [id=076c85d63cae4070850b4e0a9df8a7e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.811282 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 81852d8e06ae4c6bb715a88d8c398bfc (DEFAULT_TABLE table=default.Table_e8b42e26d9e6426e96d80135d5015108 [id=d1d672b187634a0e9d534469fc66f803]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:26.812331 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 4e99665f920f4de590bd6d9e918c7182 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e99665f920f4de590bd6d9e918c7182 already in progress: creating tablet
W20250903 20:05:26.812624 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 9a71290c27ac4a22bd9aa8e14e5834f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a71290c27ac4a22bd9aa8e14e5834f5 already in progress: creating tablet
W20250903 20:05:26.812837 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 8093e36d79894ed3aa8e92c0bca4f93d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8093e36d79894ed3aa8e92c0bca4f93d already in progress: creating tablet
I20250903 20:05:26.813611 27220 tablet_service.cc:1468] Processing CreateTablet for tablet 3590506b3fd44a6c84aa469c14ed3e9f (DEFAULT_TABLE table=default.Table_2623cd1b586e40818f657e9fc9f5f570 [id=7b22e0e04f1b4432996ecde7fd4e60c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.814626 27693 tablet_bootstrap.cc:492] T 2d9c1642fec841d497ff3496dd4564db P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:26.814762 27693 ts_tablet_manager.cc:1397] T 2d9c1642fec841d497ff3496dd4564db P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.003s
I20250903 20:05:26.814800 27349 tablet_service.cc:1468] Processing CreateTablet for tablet edac2355f67045a6acffaae723d9186f (DEFAULT_TABLE table=default.Table_7afeb3c17d814683916ce7fc2788310e [id=82f2d131a6424b678e49c46e530615fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.815572 27693 raft_consensus.cc:357] T 2d9c1642fec841d497ff3496dd4564db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:26.815620 27214 tablet_service.cc:1468] Processing CreateTablet for tablet edac2355f67045a6acffaae723d9186f (DEFAULT_TABLE table=default.Table_7afeb3c17d814683916ce7fc2788310e [id=82f2d131a6424b678e49c46e530615fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.815739 27693 raft_consensus.cc:383] T 2d9c1642fec841d497ff3496dd4564db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.815793 27693 raft_consensus.cc:738] T 2d9c1642fec841d497ff3496dd4564db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:26.816045 27693 consensus_queue.cc:260] T 2d9c1642fec841d497ff3496dd4564db P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:26.816471 27357 tablet_service.cc:1468] Processing CreateTablet for tablet 27c21921025a464681bf157b2627d351 (DEFAULT_TABLE table=default.Table_5e14429fe4e340aaa9c2d5ad605c9a92 [id=6f17187b24224598855f0f8bbe2fe630]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.816799 27693 ts_tablet_manager.cc:1428] T 2d9c1642fec841d497ff3496dd4564db P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250903 20:05:26.813561 27695 consensus_queue.cc:260] T 33fb6b357bda43e7abeaed5634183ef6 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.817400 27695 ts_tablet_manager.cc:1428] T 33fb6b357bda43e7abeaed5634183ef6 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250903 20:05:26.817674 27695 tablet_bootstrap.cc:492] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:26.818437 27229 tablet_service.cc:1468] Processing CreateTablet for tablet 27c21921025a464681bf157b2627d351 (DEFAULT_TABLE table=default.Table_5e14429fe4e340aaa9c2d5ad605c9a92 [id=6f17187b24224598855f0f8bbe2fe630]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.819288 27362 tablet_service.cc:1468] Processing CreateTablet for tablet f3185f9cf7c746418beeefb9e34758ff (DEFAULT_TABLE table=default.Table_0e14fb71c06946bdabb837fff47ec0e9 [id=bb5a0e9032ca4919bf925978f1329e74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.819383 27695 tablet_bootstrap.cc:654] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:26.819887 27228 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bcb9d5f010f4297bab78c0183889238 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a9e55f0811345f7afa71645f38ebdaf) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:26.820297 27693 tablet_bootstrap.cc:492] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:26.820683 27228 tablet_service.cc:1468] Processing CreateTablet for tablet f3185f9cf7c746418beeefb9e34758ff (DEFAULT_TABLE table=default.Table_0e14fb71c06946bdabb837fff47ec0e9 [id=bb5a0e9032ca4919bf925978f1329e74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.821507 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 53b00774b745462091321ea47fcc2114 (DEFAULT_TABLE table=default.Table_0955d4bd84b645e3b796679b761bbba7 [id=fd02c556cd3c4c42941d24cb61939a6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.822146 27356 tablet_service.cc:1468] Processing CreateTablet for tablet 475d05d087c34159b093b92cc308bcaf (DEFAULT_TABLE table=default.Table_def69a11d277479cbfc58718d39b63ac [id=1b66ec0c597848afbea739014dada98b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.822512 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 53b00774b745462091321ea47fcc2114 (DEFAULT_TABLE table=default.Table_0955d4bd84b645e3b796679b761bbba7 [id=fd02c556cd3c4c42941d24cb61939a6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.823158 28567 raft_consensus.cc:491] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:26.823225 27221 tablet_service.cc:1468] Processing CreateTablet for tablet 475d05d087c34159b093b92cc308bcaf (DEFAULT_TABLE table=default.Table_def69a11d277479cbfc58718d39b63ac [id=1b66ec0c597848afbea739014dada98b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.823284 28567 raft_consensus.cc:513] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.823334 27352 tablet_service.cc:1468] Processing CreateTablet for tablet 4120e1abd8434abb9cd92e3691d64cc7 (DEFAULT_TABLE table=default.Table_3fe8a44f17364b249736f1722bd1c9a5 [id=0f8ffc9a5d6e49c885561a232d47ec3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.823928 28567 leader_election.cc:290] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:26.824092 27223 tablet_service.cc:1468] Processing CreateTablet for tablet 4120e1abd8434abb9cd92e3691d64cc7 (DEFAULT_TABLE table=default.Table_3fe8a44f17364b249736f1722bd1c9a5 [id=0f8ffc9a5d6e49c885561a232d47ec3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.824712 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f5b87c1c4a24f8abdbfd6194f3b2c4a" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:26.824914 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 2c6a2e2e2ec34f8082a17048ca0d18b0 (DEFAULT_TABLE table=default.Table_a41978d6a03a4bfe8f2b4cbd5f615ffb [id=0bac991007ef440580f3df9d5df7b6bb]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:26.825230 27321 leader_election.cc:343] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:26.825354 27523 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f5b87c1c4a24f8abdbfd6194f3b2c4a" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:26.825631 27523 raft_consensus.cc:2466] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:26.826084 27320 leader_election.cc:304] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:26.826493 28567 raft_consensus.cc:2802] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:26.826468 27211 tablet_service.cc:1468] Processing CreateTablet for tablet 2c6a2e2e2ec34f8082a17048ca0d18b0 (DEFAULT_TABLE table=default.Table_a41978d6a03a4bfe8f2b4cbd5f615ffb [id=0bac991007ef440580f3df9d5df7b6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.826562 28567 raft_consensus.cc:491] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:26.826619 28567 raft_consensus.cc:3058] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:26.827454 27693 tablet_bootstrap.cc:654] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:26.828238 27216 tablet_service.cc:1468] Processing CreateTablet for tablet 2ce6e8b068bf4c06916f0c23da1d5824 (DEFAULT_TABLE table=default.Table_257827f9b2804d37aa0556a8affcf93e [id=df3b81c7417c47a1b2601f04d557a122]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.827540 27348 tablet_service.cc:1468] Processing CreateTablet for tablet 2ce6e8b068bf4c06916f0c23da1d5824 (DEFAULT_TABLE table=default.Table_257827f9b2804d37aa0556a8affcf93e [id=df3b81c7417c47a1b2601f04d557a122]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.832484 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edac2355f67045a6acffaae723d9186f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.833732 28627 tablet.cc:1722] T 516830fc42e84af693b3af52caee30ee P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250903 20:05:26.839749 27695 tablet_bootstrap.cc:492] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:26.839963 27695 ts_tablet_manager.cc:1397] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20250903 20:05:26.840842 27246 raft_consensus.cc:2466] T 6788b1683f984949816edfc5b68de8db P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:26.840987 27381 raft_consensus.cc:2466] T 1bcb9d5f010f4297bab78c0183889238 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:26.841496 27687 ts_tablet_manager.cc:1918] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:26.841596 27687 log.cc:1199] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/514e382479014fa7bf1f404efff11c86
I20250903 20:05:26.842005 27687 ts_tablet_manager.cc:1939] T 514e382479014fa7bf1f404efff11c86 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:26.842901 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53b00774b745462091321ea47fcc2114. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.843477 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 475d05d087c34159b093b92cc308bcaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.844857 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53b00774b745462091321ea47fcc2114. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.845350 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ce6e8b068bf4c06916f0c23da1d5824. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.846560 27321 leader_election.cc:304] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:26.846918 28571 raft_consensus.cc:2802] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:26.847348 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4120e1abd8434abb9cd92e3691d64cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.847888 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c6a2e2e2ec34f8082a17048ca0d18b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.848474 27211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c6a2e2e2ec34f8082a17048ca0d18b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.849018 27216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ce6e8b068bf4c06916f0c23da1d5824. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.849529 27223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4120e1abd8434abb9cd92e3691d64cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.850013 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3185f9cf7c746418beeefb9e34758ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.850853 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27c21921025a464681bf157b2627d351. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.851322 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edac2355f67045a6acffaae723d9186f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.851786 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3590506b3fd44a6c84aa469c14ed3e9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.853015 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 475d05d087c34159b093b92cc308bcaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.853263 27695 raft_consensus.cc:357] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.853444 27695 raft_consensus.cc:383] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.853493 27695 raft_consensus.cc:738] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:26.853750 27695 consensus_queue.cc:260] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.854467 27695 ts_tablet_manager.cc:1428] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250903 20:05:26.854785 27695 tablet_bootstrap.cc:492] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:26.856684 27695 tablet_bootstrap.cc:654] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:26.856745 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edac2355f67045a6acffaae723d9186f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.858541 27228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3185f9cf7c746418beeefb9e34758ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.859088 27229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27c21921025a464681bf157b2627d351. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.860635 27505 tablet_service.cc:1468] Processing CreateTablet for tablet baac5b97f7eb4d18a0f294ca1595a383 (DEFAULT_TABLE table=default.Table_de11ae4b3b9b42c7b6e256023fb6a3a3 [id=7b36b49fa2b24fa495aec84c99fc47a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.861105 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet baac5b97f7eb4d18a0f294ca1595a383. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.871042 27184 leader_election.cc:304] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:26.871456 28655 raft_consensus.cc:2802] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:26.871642 28655 raft_consensus.cc:695] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:26.871925 27526 raft_consensus.cc:2466] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:26.871968 28655 consensus_queue.cc:237] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.873032 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d60662d1c2284f22a0e191fa70520ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.873760 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81852d8e06ae4c6bb715a88d8c398bfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.885080 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e8b42e26d9e6426e96d80135d5015108" } modify_external_catalogs: true
I20250903 20:05:26.885598 27065 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_307490b0006e4ed493ee8b2e23b7877a" } modify_external_catalogs: true
I20250903 20:05:26.888494 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e1977f95b5174bb2b40623d225aff528" } modify_external_catalogs: true
I20250903 20:05:26.895164 27695 tablet_bootstrap.cc:492] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:26.895385 27695 ts_tablet_manager.cc:1397] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.000s
I20250903 20:05:26.896667 27695 raft_consensus.cc:357] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.896904 27695 raft_consensus.cc:383] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.896952 27695 raft_consensus.cc:738] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:26.897198 27695 consensus_queue.cc:260] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.897918 27695 ts_tablet_manager.cc:1428] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250903 20:05:26.897929 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b4b15d77d07c4fa7856d9c198cf15450 consensus state has no leader
I20250903 20:05:26.899326 27695 tablet_bootstrap.cc:492] T f0efbb7eefb84ab6bee5b9625b91e298 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:26.901921 27695 tablet_bootstrap.cc:654] T f0efbb7eefb84ab6bee5b9625b91e298 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:26.905465 27494 tablet_service.cc:1468] Processing CreateTablet for tablet e64c4a9179b6491ba92df75e0cc1bc50 (DEFAULT_TABLE table=default.Table_10c65228c77a4a9d8ccc535d2792f59f [id=0c587492cd8141fa97c0d6ea3faef6b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.905910 27494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e64c4a9179b6491ba92df75e0cc1bc50. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.909482 27496 tablet_service.cc:1468] Processing CreateTablet for tablet dd27901224d6442389e8fc09c6b498ab (DEFAULT_TABLE table=default.Table_de5aee21a7e341a89dc0e5af838dfbcc [id=e3d14638feee40fda765816279694c1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.909945 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd27901224d6442389e8fc09c6b498ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.910106 27061 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 0f5b87c1c4a24f8abdbfd6194f3b2c4a (table default.Table_6e9cf5ead5a24831a6a14d131dd8397e [id=524a27cbe1ad4c10aa2fd4f4074105c6]). Expected version 3 got 0
I20250903 20:05:26.911720 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e1654a60dac84f5f8f1d80d5f7bd5b82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:26.912322 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1654a60dac84f5f8f1d80d5f7bd5b82 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:26.923987 27215 tablet_service.cc:1468] Processing CreateTablet for tablet baac5b97f7eb4d18a0f294ca1595a383 (DEFAULT_TABLE table=default.Table_de11ae4b3b9b42c7b6e256023fb6a3a3 [id=7b36b49fa2b24fa495aec84c99fc47a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.924443 27222 tablet_service.cc:1468] Processing CreateTablet for tablet 81852d8e06ae4c6bb715a88d8c398bfc (DEFAULT_TABLE table=default.Table_e8b42e26d9e6426e96d80135d5015108 [id=d1d672b187634a0e9d534469fc66f803]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.924952 27224 tablet_service.cc:1468] Processing CreateTablet for tablet e64c4a9179b6491ba92df75e0cc1bc50 (DEFAULT_TABLE table=default.Table_10c65228c77a4a9d8ccc535d2792f59f [id=0c587492cd8141fa97c0d6ea3faef6b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.925439 27226 tablet_service.cc:1468] Processing CreateTablet for tablet dd27901224d6442389e8fc09c6b498ab (DEFAULT_TABLE table=default.Table_de5aee21a7e341a89dc0e5af838dfbcc [id=e3d14638feee40fda765816279694c1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.925887 27213 tablet_service.cc:1468] Processing CreateTablet for tablet 67d34a6947f94f08bad2d935892dbfde (DEFAULT_TABLE table=default.Table_654e50df426048f8bc1852c3015cb4f5 [id=1b0ce78b0e3d40b0ab9a49662d56701f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.924106 27218 tablet_service.cc:1468] Processing CreateTablet for tablet d60662d1c2284f22a0e191fa70520ce2 (DEFAULT_TABLE table=default.Table_f073b600fed2421f9a25aca9c22751e3 [id=076c85d63cae4070850b4e0a9df8a7e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.926877 27488 tablet_service.cc:1515] Processing DeleteTablet for tablet c9e90abef9814338b70a3550ab5686e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 607c9a9cbc5540e5bd0fa436fcba564c) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:26.930873 28571 raft_consensus.cc:695] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:26.931214 28571 consensus_queue.cc:237] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.939018 28568 raft_consensus.cc:513] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.939747 28568 leader_election.cc:290] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:26.939891 28568 raft_consensus.cc:491] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:26.939988 28568 raft_consensus.cc:513] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.940493 28568 leader_election.cc:290] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:26.940367 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a4c582b441f6467c99ba0b090abcf4cc" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
W20250903 20:05:26.941148 27320 leader_election.cc:343] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:26.943077 28579 raft_consensus.cc:513] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:26.943796 28579 leader_election.cc:290] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:26.943858 28596 raft_consensus.cc:2747] T 7cc6cf579e2642e99fefed723b0ee900 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:26.946657 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 781) took 1702 ms. Trace:
I20250903 20:05:26.946755 27684 rpcz_store.cc:276] 0903 20:05:25.244427 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:25.244494 (+ 67us) service_pool.cc:225] Handling call
0903 20:05:26.637353 (+1392859us) ts_tablet_manager.cc:1145] Deleting tablet fe902f1b58924be1ac3a5585413f7789
0903 20:05:26.637363 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:26.674431 (+ 37068us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:26.768867 (+ 94436us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:26.946633 (+177766us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2609,"tablet-delete.queue_time_us":1392051}
I20250903 20:05:26.947227 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 782) took 1490 ms. Trace:
I20250903 20:05:26.947300 27684 rpcz_store.cc:276] 0903 20:05:25.457182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:25.457248 (+ 66us) service_pool.cc:225] Handling call
0903 20:05:26.947089 (+1489841us) ts_tablet_manager.cc:1145] Deleting tablet 005d7bc2c7534f288caec8326fdf2416
0903 20:05:26.947100 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:26.947207 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1489093}
I20250903 20:05:26.947374 27488 tablet_service.cc:1468] Processing CreateTablet for tablet 67d34a6947f94f08bad2d935892dbfde (DEFAULT_TABLE table=default.Table_654e50df426048f8bc1852c3015cb4f5 [id=1b0ce78b0e3d40b0ab9a49662d56701f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.947618 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 784) took 1291 ms. Trace:
I20250903 20:05:26.947685 27684 rpcz_store.cc:276] 0903 20:05:25.656124 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:25.656190 (+ 66us) service_pool.cc:225] Handling call
0903 20:05:26.947518 (+1291328us) ts_tablet_manager.cc:1145] Deleting tablet 24ddae9179cd4240914cbbae31d8b851
0903 20:05:26.947526 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:26.947602 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1291030}
I20250903 20:05:26.947809 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67d34a6947f94f08bad2d935892dbfde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.947960 27684 tablet_replica.cc:331] T cd92e3dafca6419f8eba7a4ef51a911e P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:26.948154 27684 raft_consensus.cc:2241] T cd92e3dafca6419f8eba7a4ef51a911e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:26.948189 27227 tablet_service.cc:1468] Processing CreateTablet for tablet 224fc8e6cc68476c8321c480da6ef0ed (DEFAULT_TABLE table=default.Table_108ee548249b426e934082d6bc507068 [id=e4eaa69876524e8492ea07824d26b877]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.948319 27684 raft_consensus.cc:2270] T cd92e3dafca6419f8eba7a4ef51a911e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:26.948897 27684 ts_tablet_manager.cc:1905] T cd92e3dafca6419f8eba7a4ef51a911e P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:26.951049 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "961ade36f7d14eba91bb16d5eae2e254" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:26.950770 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "33fb6b357bda43e7abeaed5634183ef6" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:26.951355 27248 raft_consensus.cc:3058] T 961ade36f7d14eba91bb16d5eae2e254 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:26.951714 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "33fb6b357bda43e7abeaed5634183ef6" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:26.951874 27249 raft_consensus.cc:2466] T 33fb6b357bda43e7abeaed5634183ef6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:26.952917 27321 leader_election.cc:304] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
W20250903 20:05:26.953373 27320 leader_election.cc:343] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:26.951050 27246 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a4c582b441f6467c99ba0b090abcf4cc" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:26.954000 27246 raft_consensus.cc:3058] T a4c582b441f6467c99ba0b090abcf4cc P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:26.954121 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "961ade36f7d14eba91bb16d5eae2e254" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
W20250903 20:05:26.954649 27320 leader_election.cc:343] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:26.955519 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 224fc8e6cc68476c8321c480da6ef0ed (DEFAULT_TABLE table=default.Table_108ee548249b426e934082d6bc507068 [id=e4eaa69876524e8492ea07824d26b877]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.955928 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 224fc8e6cc68476c8321c480da6ef0ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.959367 28579 raft_consensus.cc:2802] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:26.959481 28579 raft_consensus.cc:491] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:26.959582 28579 raft_consensus.cc:3058] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:26.961323 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet fe902f1b58924be1ac3a5585413f7789 (table default.Table_1840897b7e7f4f1e8052d16ca5e77a4f [id=78785eeb2c464e6d9f0cc5c8792376ec]) successfully deleted
W20250903 20:05:26.961781 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 005d7bc2c7534f288caec8326fdf2416 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 005d7bc2c7534f288caec8326fdf2416 already in progress: creating tablet
W20250903 20:05:26.962106 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 24ddae9179cd4240914cbbae31d8b851 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24ddae9179cd4240914cbbae31d8b851 already in progress: creating tablet
I20250903 20:05:26.966187 27693 tablet_bootstrap.cc:492] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:26.966388 27693 ts_tablet_manager.cc:1397] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.146s user 0.019s sys 0.013s
I20250903 20:05:26.967177 27693 raft_consensus.cc:357] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:26.967363 27693 raft_consensus.cc:383] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.967417 27693 raft_consensus.cc:738] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:26.967653 27693 consensus_queue.cc:260] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:26.968140 27693 ts_tablet_manager.cc:1428] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250903 20:05:26.968499 27693 tablet_bootstrap.cc:492] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:26.970188 27693 tablet_bootstrap.cc:654] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:26.977619 27693 tablet_bootstrap.cc:492] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:26.977883 27693 ts_tablet_manager.cc:1397] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20250903 20:05:26.978864 27693 raft_consensus.cc:357] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:26.979105 27693 raft_consensus.cc:383] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.979357 27693 raft_consensus.cc:738] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:26.979628 27693 consensus_queue.cc:260] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:26.980240 27693 ts_tablet_manager.cc:1428] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250903 20:05:26.980186 27358 tablet_service.cc:1468] Processing CreateTablet for tablet d60662d1c2284f22a0e191fa70520ce2 (DEFAULT_TABLE table=default.Table_f073b600fed2421f9a25aca9c22751e3 [id=076c85d63cae4070850b4e0a9df8a7e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.980655 27693 tablet_bootstrap.cc:492] T 59a1dd48d76241d7ba817070fbffd853 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:26.980744 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d60662d1c2284f22a0e191fa70520ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.982800 27693 tablet_bootstrap.cc:654] T 59a1dd48d76241d7ba817070fbffd853 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:26.985230 27350 tablet_service.cc:1468] Processing CreateTablet for tablet 81852d8e06ae4c6bb715a88d8c398bfc (DEFAULT_TABLE table=default.Table_e8b42e26d9e6426e96d80135d5015108 [id=d1d672b187634a0e9d534469fc66f803]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.985702 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81852d8e06ae4c6bb715a88d8c398bfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.988885 27695 tablet_bootstrap.cc:492] T f0efbb7eefb84ab6bee5b9625b91e298 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:26.989056 27695 ts_tablet_manager.cc:1397] T f0efbb7eefb84ab6bee5b9625b91e298 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.090s user 0.008s sys 0.016s
I20250903 20:05:26.989959 27695 raft_consensus.cc:357] T f0efbb7eefb84ab6bee5b9625b91e298 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:26.990176 27695 raft_consensus.cc:383] T f0efbb7eefb84ab6bee5b9625b91e298 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.990341 27695 raft_consensus.cc:738] T f0efbb7eefb84ab6bee5b9625b91e298 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:26.990540 27693 tablet_bootstrap.cc:492] T 59a1dd48d76241d7ba817070fbffd853 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:26.990566 27695 consensus_queue.cc:260] T f0efbb7eefb84ab6bee5b9625b91e298 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:26.990698 27693 ts_tablet_manager.cc:1397] T 59a1dd48d76241d7ba817070fbffd853 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.001s
I20250903 20:05:26.991102 27695 ts_tablet_manager.cc:1428] T f0efbb7eefb84ab6bee5b9625b91e298 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:26.991611 27693 raft_consensus.cc:357] T 59a1dd48d76241d7ba817070fbffd853 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:26.991721 27695 tablet_bootstrap.cc:492] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:26.991796 27693 raft_consensus.cc:383] T 59a1dd48d76241d7ba817070fbffd853 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:26.991894 27693 raft_consensus.cc:738] T 59a1dd48d76241d7ba817070fbffd853 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:26.992122 27693 consensus_queue.cc:260] T 59a1dd48d76241d7ba817070fbffd853 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:26.992731 27693 ts_tablet_manager.cc:1428] T 59a1dd48d76241d7ba817070fbffd853 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:26.993044 27693 tablet_bootstrap.cc:492] T 53881ad30bc843839008ace4012a3f70 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:26.994571 27354 tablet_service.cc:1468] Processing CreateTablet for tablet baac5b97f7eb4d18a0f294ca1595a383 (DEFAULT_TABLE table=default.Table_de11ae4b3b9b42c7b6e256023fb6a3a3 [id=7b36b49fa2b24fa495aec84c99fc47a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.994995 27354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet baac5b97f7eb4d18a0f294ca1595a383. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.995047 27693 tablet_bootstrap.cc:654] T 53881ad30bc843839008ace4012a3f70 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:26.995059 27695 tablet_bootstrap.cc:654] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:26.996341 27347 tablet_service.cc:1468] Processing CreateTablet for tablet dd27901224d6442389e8fc09c6b498ab (DEFAULT_TABLE table=default.Table_de5aee21a7e341a89dc0e5af838dfbcc [id=e3d14638feee40fda765816279694c1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.996466 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 224fc8e6cc68476c8321c480da6ef0ed (DEFAULT_TABLE table=default.Table_108ee548249b426e934082d6bc507068 [id=e4eaa69876524e8492ea07824d26b877]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.996716 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd27901224d6442389e8fc09c6b498ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.996928 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 224fc8e6cc68476c8321c480da6ef0ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:26.998292 27353 tablet_service.cc:1468] Processing CreateTablet for tablet e64c4a9179b6491ba92df75e0cc1bc50 (DEFAULT_TABLE table=default.Table_10c65228c77a4a9d8ccc535d2792f59f [id=0c587492cd8141fa97c0d6ea3faef6b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:26.998665 27353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e64c4a9179b6491ba92df75e0cc1bc50. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.000353 27359 tablet_service.cc:1468] Processing CreateTablet for tablet 67d34a6947f94f08bad2d935892dbfde (DEFAULT_TABLE table=default.Table_654e50df426048f8bc1852c3015cb4f5 [id=1b0ce78b0e3d40b0ab9a49662d56701f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.000773 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67d34a6947f94f08bad2d935892dbfde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.010839 27060 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 516830fc42e84af693b3af52caee30ee (table default.Table_4d303308b63c4ce09056e226d1a30498 [id=098bc9a9b0a24a4a9edb7859310ac9b3]). Expected version 3 got 0
I20250903 20:05:27.011157 27060 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet b0d79c78e0024b7ba3bb3356be7c8ceb (table default.Table_94e9e32ef1134f07ade1cdf8da69ead6 [id=e248f9c4b9c9406395aaea7ba3feb9ba]). Expected version 1 got 0
I20250903 20:05:27.011329 28655 raft_consensus.cc:491] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:27.011359 27060 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet d859c0222c3647dab3dddd105e4cf0fc (table default.Table_dc24dd98fc4146eda8660576fd53a29d [id=7add4b636f1b4bffae663fdf432de05f]). Expected version 4 got 0
I20250903 20:05:27.011489 28655 raft_consensus.cc:513] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.012324 28655 leader_election.cc:290] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:27.013733 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet 961ade36f7d14eba91bb16d5eae2e254 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:27.013713 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b3bfc63aa0b4e56bec189e834dc6a8d" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:27.014032 27381 raft_consensus.cc:2466] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:27.014190 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet 33fb6b357bda43e7abeaed5634183ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:27.014609 27184 leader_election.cc:304] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:27.014935 28630 raft_consensus.cc:2802] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:27.015029 28630 raft_consensus.cc:491] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:27.015107 28630 raft_consensus.cc:3058] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:27.015367 28655 raft_consensus.cc:491] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:27.015484 28655 raft_consensus.cc:513] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.016093 28655 leader_election.cc:290] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:27.016794 27693 tablet_bootstrap.cc:492] T 53881ad30bc843839008ace4012a3f70 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:27.016884 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b0d79c78e0024b7ba3bb3356be7c8ceb" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:27.017087 27381 raft_consensus.cc:2466] T b0d79c78e0024b7ba3bb3356be7c8ceb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:27.017675 27693 ts_tablet_manager.cc:1397] T 53881ad30bc843839008ace4012a3f70 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.025s user 0.015s sys 0.002s
I20250903 20:05:27.018524 27693 raft_consensus.cc:357] T 53881ad30bc843839008ace4012a3f70 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.018894 27693 raft_consensus.cc:383] T 53881ad30bc843839008ace4012a3f70 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:27.019018 27693 raft_consensus.cc:738] T 53881ad30bc843839008ace4012a3f70 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:27.019330 27693 consensus_queue.cc:260] T 53881ad30bc843839008ace4012a3f70 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.019672 27184 leader_election.cc:304] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:27.019991 28655 raft_consensus.cc:2802] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:27.020079 28655 raft_consensus.cc:491] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:27.020152 28655 raft_consensus.cc:3058] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:27.020040 27693 ts_tablet_manager.cc:1428] T 53881ad30bc843839008ace4012a3f70 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:27.020989 27693 tablet_bootstrap.cc:492] T 17e0864a32d5452d84ac470b55fbaaf4 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:27.022347 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 553443eda16c44fa84b1db22b1363808
I20250903 20:05:27.022399 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet e0b9172ebb484382adb86e597f5cbc3b
I20250903 20:05:27.023011 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 44fef873a0644dd988322847f333ec95
I20250903 20:05:27.023289 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet ea3f82f4e4244219ab0919f3e341aead
I20250903 20:05:27.023576 27064 master_service.cc:948] ReplaceTablet: received request to replace tablet 1fbe54dc40dd47daa109ceeb07546b0a from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:27.024694 27212 tablet_service.cc:1515] Processing DeleteTablet for tablet 553443eda16c44fa84b1db22b1363808 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:27.024729 27501 tablet_service.cc:1515] Processing DeleteTablet for tablet 44fef873a0644dd988322847f333ec95 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:27.024931 27219 tablet_service.cc:1515] Processing DeleteTablet for tablet e0b9172ebb484382adb86e597f5cbc3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:27.025007 27212 tablet_service.cc:1515] Processing DeleteTablet for tablet ea3f82f4e4244219ab0919f3e341aead with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:27.025134 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet 44fef873a0644dd988322847f333ec95 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:27.025153 27492 tablet_service.cc:1515] Processing DeleteTablet for tablet 553443eda16c44fa84b1db22b1363808 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:27.025290 27058 master_service.cc:948] ReplaceTablet: received request to replace tablet 90baf83357494a0aae325908c118b471 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:27.025342 27491 tablet_service.cc:1515] Processing DeleteTablet for tablet e0b9172ebb484382adb86e597f5cbc3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:27.025471 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet ea3f82f4e4244219ab0919f3e341aead with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:27.025653 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 81852d8e06ae4c6bb715a88d8c398bfc
I20250903 20:05:27.026443 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet 81852d8e06ae4c6bb715a88d8c398bfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:27.026487 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet 81852d8e06ae4c6bb715a88d8c398bfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:27.027959 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5a979a57ff9e4e9796aa94e6398ead05" } new_table_name: "default.Table_6c8932057db44105a1ca8eee2cafc358" modify_external_catalogs: true
I20250903 20:05:27.028278 27693 tablet_bootstrap.cc:654] T 17e0864a32d5452d84ac470b55fbaaf4 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:27.029511 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_656f015bd5f84e7c92ecf269159af5de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:27.030493 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet e0b9172ebb484382adb86e597f5cbc3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:27.030876 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet 553443eda16c44fa84b1db22b1363808 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:27.031250 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet 44fef873a0644dd988322847f333ec95 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:27.031217 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_6f835fbb9a1f4aff87000ff5e65133f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:27.031615 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet ea3f82f4e4244219ab0919f3e341aead with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:27.031759 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f835fbb9a1f4aff87000ff5e65133f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:27.032112 27364 tablet_service.cc:1515] Processing DeleteTablet for tablet 81852d8e06ae4c6bb715a88d8c398bfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:27.035195 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_656f015bd5f84e7c92ecf269159af5de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:27.035960 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b3bfc63aa0b4e56bec189e834dc6a8d" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:27.035969 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b0d79c78e0024b7ba3bb3356be7c8ceb" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:27.036203 27526 raft_consensus.cc:2466] T b0d79c78e0024b7ba3bb3356be7c8ceb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
W20250903 20:05:27.036592 27183 leader_election.cc:343] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:27.038395 27248 raft_consensus.cc:2466] T 961ade36f7d14eba91bb16d5eae2e254 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:27.038439 28567 raft_consensus.cc:513] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.039083 28567 leader_election.cc:290] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:27.041153 27246 raft_consensus.cc:2466] T a4c582b441f6467c99ba0b090abcf4cc P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:27.041775 28630 raft_consensus.cc:513] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.042704 27687 tablet_replica.cc:331] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:27.042923 27687 raft_consensus.cc:2241] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:27.043056 27685 ts_tablet_manager.cc:1918] T 885fff47fea14b84ac6b34a57b827b4b P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:27.043174 27685 log.cc:1199] T 885fff47fea14b84ac6b34a57b827b4b P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/885fff47fea14b84ac6b34a57b827b4b
I20250903 20:05:27.043186 27687 pending_rounds.cc:70] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528: Trying to abort 1 pending ops.
I20250903 20:05:27.043257 27687 pending_rounds.cc:77] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384980713029632 op_type: NO_OP noop_request { }
I20250903 20:05:27.043352 27687 raft_consensus.cc:2887] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:27.043466 27685 ts_tablet_manager.cc:1939] T 885fff47fea14b84ac6b34a57b827b4b P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:27.043478 27687 raft_consensus.cc:2270] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:27.044487 27321 leader_election.cc:304] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:27.045084 27321 leader_election.cc:304] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:27.046936 28568 raft_consensus.cc:2802] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:27.047099 28568 raft_consensus.cc:695] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:27.047344 28568 consensus_queue.cc:237] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.047722 28630 leader_election.cc:290] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:27.050678 28567 raft_consensus.cc:2802] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:27.050686 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b3bfc63aa0b4e56bec189e834dc6a8d" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:27.050845 28567 raft_consensus.cc:695] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:27.051160 28567 consensus_queue.cc:237] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:27.051328 27183 leader_election.cc:343] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:27.052896 27687 ts_tablet_manager.cc:1905] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:27.053260 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b3bfc63aa0b4e56bec189e834dc6a8d" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:27.053447 27381 raft_consensus.cc:3058] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:27.053442 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f5b87c1c4a24f8abdbfd6194f3b2c4a" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:27.053664 27526 raft_consensus.cc:3058] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:27.057282 27518 raft_consensus.cc:2466] T a0293a69f1c246d08d414e95729b4ab4 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:27.057817 27183 leader_election.cc:304] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:27.058310 28630 raft_consensus.cc:2802] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:27.058537 28630 raft_consensus.cc:695] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:27.058849 28630 consensus_queue.cc:237] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.066323 27246 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f5b87c1c4a24f8abdbfd6194f3b2c4a" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:27.066598 27246 raft_consensus.cc:3058] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:27.067538 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 514e382479014fa7bf1f404efff11c86 (table default.Table_b0a0a2dfe31e4961871f334372541d05 [id=30918e7a5f1d43aab5eaddef7c485cd3]) successfully deleted
I20250903 20:05:27.068228 28662 consensus_queue.cc:1035] T 0ad64c699b294679b6fec37b5ad23528 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:27.072429 28522 raft_consensus.cc:491] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:27.072605 28522 raft_consensus.cc:513] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.073969 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_31cccd71f85842908650e5d1a49c342f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:27.074777 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31cccd71f85842908650e5d1a49c342f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:27.075359 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_716897f657da43e5986ff43425cedd20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:27.076354 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_716897f657da43e5986ff43425cedd20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:27.077769 28662 raft_consensus.cc:491] T 53881ad30bc843839008ace4012a3f70 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:27.077956 28662 raft_consensus.cc:513] T 53881ad30bc843839008ace4012a3f70 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.078614 28662 leader_election.cc:290] T 53881ad30bc843839008ace4012a3f70 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:27.078742 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d7d1a268897341aeb586cae942a487b3" } new_table_name: "default.Table_033ca622582f413c9b0ddf3fe0bf78d1" modify_external_catalogs: true
I20250903 20:05:27.079313 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53881ad30bc843839008ace4012a3f70" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:27.080001 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53881ad30bc843839008ace4012a3f70" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:27.081059 27320 leader_election.cc:343] T 53881ad30bc843839008ace4012a3f70 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:27.027459 27062 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet d859c0222c3647dab3dddd105e4cf0fc (table default.Table_dc24dd98fc4146eda8660576fd53a29d [id=7add4b636f1b4bffae663fdf432de05f]). Expected version 4 got 0
I20250903 20:05:27.082293 27062 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 3b3bfc63aa0b4e56bec189e834dc6a8d (table default.Table_9b20469406c94986ab496ea3e678b586 [id=6d6f5e8cadf9445d8695c075cfabb2ea]). Expected version 1 got 0
I20250903 20:05:27.083302 27062 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 516830fc42e84af693b3af52caee30ee (table default.Table_4d303308b63c4ce09056e226d1a30498 [id=098bc9a9b0a24a4a9edb7859310ac9b3]). Expected version 3 got 0
I20250903 20:05:27.089941 27063 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet d859c0222c3647dab3dddd105e4cf0fc (table default.Table_dc24dd98fc4146eda8660576fd53a29d [id=7add4b636f1b4bffae663fdf432de05f]). Expected version 4 got 0
I20250903 20:05:27.090202 28567 raft_consensus.cc:491] T 2d9c1642fec841d497ff3496dd4564db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:27.090445 28567 raft_consensus.cc:513] T 2d9c1642fec841d497ff3496dd4564db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.091395 28567 leader_election.cc:290] T 2d9c1642fec841d497ff3496dd4564db P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:27.091629 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d9c1642fec841d497ff3496dd4564db" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:27.092255 27321 leader_election.cc:343] T 2d9c1642fec841d497ff3496dd4564db P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:27.092311 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d9c1642fec841d497ff3496dd4564db" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:27.092931 27320 leader_election.cc:343] T 2d9c1642fec841d497ff3496dd4564db P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:27.093101 27320 leader_election.cc:304] T 2d9c1642fec841d497ff3496dd4564db P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:27.102106 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet baac5b97f7eb4d18a0f294ca1595a383. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.102833 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 224fc8e6cc68476c8321c480da6ef0ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.103595 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d60662d1c2284f22a0e191fa70520ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.104008 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd27901224d6442389e8fc09c6b498ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.104177 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67d34a6947f94f08bad2d935892dbfde. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:27.108505 27321 leader_election.cc:343] T 53881ad30bc843839008ace4012a3f70 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:27.108683 27321 leader_election.cc:304] T 53881ad30bc843839008ace4012a3f70 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:27.109222 28571 raft_consensus.cc:2747] T 2d9c1642fec841d497ff3496dd4564db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:27.109505 28568 raft_consensus.cc:2747] T 53881ad30bc843839008ace4012a3f70 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:27.110263 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e64c4a9179b6491ba92df75e0cc1bc50. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.111285 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81852d8e06ae4c6bb715a88d8c398bfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.113483 27693 tablet_bootstrap.cc:492] T 17e0864a32d5452d84ac470b55fbaaf4 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:27.113729 27693 ts_tablet_manager.cc:1397] T 17e0864a32d5452d84ac470b55fbaaf4 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.093s user 0.006s sys 0.004s
I20250903 20:05:27.113768 28522 leader_election.cc:290] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:27.114670 27693 raft_consensus.cc:357] T 17e0864a32d5452d84ac470b55fbaaf4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.114862 27693 raft_consensus.cc:383] T 17e0864a32d5452d84ac470b55fbaaf4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:27.114987 27693 raft_consensus.cc:738] T 17e0864a32d5452d84ac470b55fbaaf4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
W20250903 20:05:27.114964 27183 leader_election.cc:343] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: de8ac9f984be46ec8f021928de12a090
I20250903 20:05:27.115348 27693 consensus_queue.cc:260] T 17e0864a32d5452d84ac470b55fbaaf4 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.115939 27693 ts_tablet_manager.cc:1428] T 17e0864a32d5452d84ac470b55fbaaf4 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:27.116286 27693 tablet_bootstrap.cc:492] T a4f3230f50ec4c95b528fead8c191224 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:27.118847 27693 tablet_bootstrap.cc:654] T a4f3230f50ec4c95b528fead8c191224 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:27.126940 27693 tablet_bootstrap.cc:492] T a4f3230f50ec4c95b528fead8c191224 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:27.127281 27693 ts_tablet_manager.cc:1397] T a4f3230f50ec4c95b528fead8c191224 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20250903 20:05:27.128327 27693 raft_consensus.cc:357] T a4f3230f50ec4c95b528fead8c191224 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.128702 27693 raft_consensus.cc:383] T a4f3230f50ec4c95b528fead8c191224 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:27.128825 27693 raft_consensus.cc:738] T a4f3230f50ec4c95b528fead8c191224 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:27.129140 27693 consensus_queue.cc:260] T a4f3230f50ec4c95b528fead8c191224 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.129694 27693 ts_tablet_manager.cc:1428] T a4f3230f50ec4c95b528fead8c191224 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:27.130067 27693 tablet_bootstrap.cc:492] T 7e2f98f760cf4df4bc2a10530e2157c6 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:27.133899 28655 raft_consensus.cc:513] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.134557 27684 ts_tablet_manager.cc:1918] T cd92e3dafca6419f8eba7a4ef51a911e P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:27.134605 28655 leader_election.cc:290] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:27.134672 27684 log.cc:1199] T cd92e3dafca6419f8eba7a4ef51a911e P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/cd92e3dafca6419f8eba7a4ef51a911e
I20250903 20:05:27.134801 28579 raft_consensus.cc:513] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.135003 27684 ts_tablet_manager.cc:1939] T cd92e3dafca6419f8eba7a4ef51a911e P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:27.135492 28579 leader_election.cc:290] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:27.136492 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b0d79c78e0024b7ba3bb3356be7c8ceb" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:27.136663 27518 raft_consensus.cc:3058] T b0d79c78e0024b7ba3bb3356be7c8ceb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:27.137801 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "33fb6b357bda43e7abeaed5634183ef6" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:27.137975 27248 raft_consensus.cc:3058] T 33fb6b357bda43e7abeaed5634183ef6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:27.145310 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_fed56630d54540548ffb7f00228a1ee9" } new_table_name: "default.Table_235bfc17555844a39ca65275eae2cf99" modify_external_catalogs: true
W20250903 20:05:27.146520 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1ec70a427065468394e7fd9db9143847 consensus state has no leader
I20250903 20:05:27.146806 27065 master_service.cc:948] ReplaceTablet: received request to replace tablet a686ea49a29a4dd7b6e5dce5e1f9935a from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:27.147512 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_2943f6ec93054a3e8cd8b08ccb0ee025"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:27.148130 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2943f6ec93054a3e8cd8b08ccb0ee025 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:27.149937 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_127e4ebeb1d54cb48599c668f7c2aa49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:27.151521 27057 master_service.cc:948] ReplaceTablet: received request to replace tablet ce3be9d44147420d953e716a347352c9 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:27.152045 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 678) took 4714 ms. Trace:
I20250903 20:05:27.152128 27685 rpcz_store.cc:276] 0903 20:05:22.437789 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.459907 (+ 22118us) service_pool.cc:225] Handling call
0903 20:05:26.759474 (+4299567us) ts_tablet_manager.cc:1145] Deleting tablet 885fff47fea14b84ac6b34a57b827b4b
0903 20:05:26.759483 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:26.840855 (+ 81372us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:27.042932 (+202077us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:27.152022 (+109090us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4299406}
I20250903 20:05:27.152469 27685 tablet_replica.cc:331] T fa05f54ebc324b40855fa3288760e9e9 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:27.152658 27685 raft_consensus.cc:2241] T fa05f54ebc324b40855fa3288760e9e9 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:27.152804 27685 raft_consensus.cc:2270] T fa05f54ebc324b40855fa3288760e9e9 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:27.152902 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 838041eaf62e4d40a8cfdc0dab5494ba consensus state has no leader
I20250903 20:05:27.153254 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b7fa325810b64d068d80bcb17c18ae5d" } new_table_name: "default.Table_61d7e38771d54f1fb1f75280d225c5cc" modify_external_catalogs: true
I20250903 20:05:27.153390 27685 ts_tablet_manager.cc:1905] T fa05f54ebc324b40855fa3288760e9e9 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:27.156368 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 885fff47fea14b84ac6b34a57b827b4b (table default.Table_1c679659094d4666aaaf68f76f667890 [id=61a25739b03042f7bd5bb693b68656d5]) successfully deleted
I20250903 20:05:27.163267 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b0d79c78e0024b7ba3bb3356be7c8ceb" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:27.163482 27386 raft_consensus.cc:3058] T b0d79c78e0024b7ba3bb3356be7c8ceb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:27.165997 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6b51dff3c41c4a67b5bad68460ae919f" } new_table_name: "default.Table_a80111ac09d44f43bdfd7b3b9fda8171" modify_external_catalogs: true
W20250903 20:05:27.167146 27184 leader_election.cc:343] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: de8ac9f984be46ec8f021928de12a090
I20250903 20:05:27.167280 27184 leader_election.cc:304] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:27.167498 28655 raft_consensus.cc:2747] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:27.168430 27693 tablet_bootstrap.cc:654] T 7e2f98f760cf4df4bc2a10530e2157c6 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:27.169296 28567 consensus_queue.cc:1035] T 0ad64c699b294679b6fec37b5ad23528 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:27.172334 27695 tablet_bootstrap.cc:492] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:27.172462 27695 ts_tablet_manager.cc:1397] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.181s user 0.009s sys 0.004s
W20250903 20:05:27.173207 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_127e4ebeb1d54cb48599c668f7c2aa49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:27.173709 27695 raft_consensus.cc:357] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.173918 27695 raft_consensus.cc:383] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:27.173974 27695 raft_consensus.cc:738] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:27.174260 27695 consensus_queue.cc:260] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.176741 28567 raft_consensus.cc:491] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:27.177105 28567 raft_consensus.cc:513] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.178337 28567 leader_election.cc:290] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:27.178974 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "33fb6b357bda43e7abeaed5634183ef6" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:27.179282 27517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0efbb7eefb84ab6bee5b9625b91e298" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:27.179803 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0efbb7eefb84ab6bee5b9625b91e298" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:27.180044 27249 raft_consensus.cc:2466] T f0efbb7eefb84ab6bee5b9625b91e298 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:27.180524 27321 leader_election.cc:304] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
W20250903 20:05:27.181156 27320 leader_election.cc:343] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:27.181665 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 785) took 1524 ms. Trace:
I20250903 20:05:27.182047 27684 rpcz_store.cc:276] 0903 20:05:25.657402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:25.657663 (+ 261us) service_pool.cc:225] Handling call
0903 20:05:26.947903 (+1290240us) ts_tablet_manager.cc:1145] Deleting tablet cd92e3dafca6419f8eba7a4ef51a911e
0903 20:05:26.947915 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:27.039388 (+ 91473us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:27.134445 (+ 95057us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:27.181648 (+ 47203us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1290078}
I20250903 20:05:27.182910 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet cd92e3dafca6419f8eba7a4ef51a911e (table default.Table_c8d55d18c6bb4511ab10f35d2db6d2f9 [id=b15b2b0164104454a23d766feea4e0de]) successfully deleted
I20250903 20:05:27.183365 27684 tablet_replica.cc:331] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
W20250903 20:05:27.183705 27320 leader_election.cc:343] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:27.185011 27684 raft_consensus.cc:2241] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:27.181538 28568 raft_consensus.cc:2802] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:27.185653 28568 raft_consensus.cc:491] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:27.185776 28568 raft_consensus.cc:3058] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:27.187188 27684 raft_consensus.cc:2270] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:27.188328 27684 ts_tablet_manager.cc:1905] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:27.195739 28567 raft_consensus.cc:491] T 17e0864a32d5452d84ac470b55fbaaf4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:27.195926 28567 raft_consensus.cc:513] T 17e0864a32d5452d84ac470b55fbaaf4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.196868 28567 leader_election.cc:290] T 17e0864a32d5452d84ac470b55fbaaf4 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:27.197170 27517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17e0864a32d5452d84ac470b55fbaaf4" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:27.197191 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17e0864a32d5452d84ac470b55fbaaf4" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:27.197816 27321 leader_election.cc:343] T 17e0864a32d5452d84ac470b55fbaaf4 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:27.197993 27320 leader_election.cc:343] T 17e0864a32d5452d84ac470b55fbaaf4 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:27.198196 27320 leader_election.cc:304] T 17e0864a32d5452d84ac470b55fbaaf4 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:27.198541 28567 raft_consensus.cc:2747] T 17e0864a32d5452d84ac470b55fbaaf4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:27.211966 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8f4fef41c9714d41a65fef1b5473ce16 has no consensus state
W20250903 20:05:27.212338 27066 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 47500175a642457687aa4226195fce47 consensus state has no leader
I20250903 20:05:27.212466 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_2145fd2e97154b2fbb583eb52f977e70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:27.212935 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet ce3be9d44147420d953e716a347352c9
W20250903 20:05:27.213094 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2145fd2e97154b2fbb583eb52f977e70 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:27.213407 27057 catalog_manager.cc:6652] ReplaceTablet: tablet ce3be9d44147420d953e716a347352c9 deleted and replaced by tablet a09771c026a9474c9f61ed259d843cca
I20250903 20:05:27.214293 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet ce3be9d44147420d953e716a347352c9 with delete_type TABLET_DATA_DELETED (replaced by tablet a09771c026a9474c9f61ed259d843cca) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:27.214524 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2dc0f714cdb44ece90615de5a11bc71f consensus state has no leader
I20250903 20:05:27.214586 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet ce3be9d44147420d953e716a347352c9 with delete_type TABLET_DATA_DELETED (replaced by tablet a09771c026a9474c9f61ed259d843cca) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:27.215814 27693 tablet_bootstrap.cc:492] T 7e2f98f760cf4df4bc2a10530e2157c6 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:27.217535 27693 ts_tablet_manager.cc:1397] T 7e2f98f760cf4df4bc2a10530e2157c6 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.088s user 0.010s sys 0.006s
W20250903 20:05:27.218935 27462 consensus_peers.cc:489] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e78e6d212daf4589a6b4b65711e356cb. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:27.219549 27693 raft_consensus.cc:357] T 7e2f98f760cf4df4bc2a10530e2157c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.219719 27693 raft_consensus.cc:383] T 7e2f98f760cf4df4bc2a10530e2157c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:27.219794 27693 raft_consensus.cc:738] T 7e2f98f760cf4df4bc2a10530e2157c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:27.220001 27693 consensus_queue.cc:260] T 7e2f98f760cf4df4bc2a10530e2157c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.220410 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet ce3be9d44147420d953e716a347352c9 with delete_type TABLET_DATA_DELETED (replaced by tablet a09771c026a9474c9f61ed259d843cca) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:27.220471 27693 ts_tablet_manager.cc:1428] T 7e2f98f760cf4df4bc2a10530e2157c6 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:27.220736 27693 tablet_bootstrap.cc:492] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:27.222487 27693 tablet_bootstrap.cc:654] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:27.224372 27693 tablet_bootstrap.cc:492] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:27.225643 27059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_423c3bfc0b8c46bc9f81d7a9013e1468" } modify_external_catalogs: true
I20250903 20:05:27.225816 27059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_423c3bfc0b8c46bc9f81d7a9013e1468" } modify_external_catalogs: true
I20250903 20:05:27.227736 27693 ts_tablet_manager.cc:1397] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20250903 20:05:27.228284 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7afeb3c17d814683916ce7fc2788310e" } modify_external_catalogs: true
I20250903 20:05:27.228389 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_cb368db83ae64dc784cc454f9224c3e2" } modify_external_catalogs: true
I20250903 20:05:27.228456 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7afeb3c17d814683916ce7fc2788310e" } modify_external_catalogs: true
I20250903 20:05:27.228567 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_cb368db83ae64dc784cc454f9224c3e2" } modify_external_catalogs: true
I20250903 20:05:27.228529 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_01cecfeda8a247628e40e6d815ddb410"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:27.228597 27693 raft_consensus.cc:357] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.228775 27693 raft_consensus.cc:383] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:27.228838 27693 raft_consensus.cc:738] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
W20250903 20:05:27.229071 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01cecfeda8a247628e40e6d815ddb410 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:27.229087 27693 consensus_queue.cc:260] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.230383 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_4f815c3438474e3ca7b8e4b9d0be87d5" } modify_external_catalogs: true
I20250903 20:05:27.230587 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_4f815c3438474e3ca7b8e4b9d0be87d5" } modify_external_catalogs: true
I20250903 20:05:27.234565 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_f6d8f4f4883c410c82f937ebeae0141c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:27.235105 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6d8f4f4883c410c82f937ebeae0141c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:27.236447 27063 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_4d303308b63c4ce09056e226d1a30498 [id=098bc9a9b0a24a4a9edb7859310ac9b3] alter complete (version 3)
I20250903 20:05:27.236716 27359 tablet_service.cc:1468] Processing CreateTablet for tablet 1fce05bd4e6c425e952001ffdc25c78a (DEFAULT_TABLE table=default.Table_895e601507754fd399673ccb24b57a17 [id=1f3c305b81f1455aa8e0a41f91245276]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.236886 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d3841357472f410abd60951e763adb30" } modify_external_catalogs: true
I20250903 20:05:27.237084 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d3841357472f410abd60951e763adb30" } modify_external_catalogs: true
I20250903 20:05:27.237157 27518 raft_consensus.cc:2466] T b0d79c78e0024b7ba3bb3356be7c8ceb P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:27.237743 27183 leader_election.cc:304] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:27.238019 28655 raft_consensus.cc:2802] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:27.238109 28655 raft_consensus.cc:695] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:27.238355 28655 consensus_queue.cc:237] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.238559 27687 ts_tablet_manager.cc:1918] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:27.238667 27687 log.cc:1199] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/6788b1683f984949816edfc5b68de8db
I20250903 20:05:27.239077 27687 ts_tablet_manager.cc:1939] T 6788b1683f984949816edfc5b68de8db P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:27.239279 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fce05bd4e6c425e952001ffdc25c78a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.240869 27246 raft_consensus.cc:2466] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:27.242655 27381 raft_consensus.cc:2466] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:27.248524 27184 leader_election.cc:304] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:27.249545 27354 tablet_service.cc:1515] Processing DeleteTablet for tablet 53881ad30bc843839008ace4012a3f70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:27.249794 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bad3c509460422396a17a3c4ae3d1e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 13496efe34944b7280e7a684f180e6e0) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:27.249969 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f8cfe53b91b41d7ba7f9074e1005af3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:27.250149 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet f0efbb7eefb84ab6bee5b9625b91e298 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:27.250365 27348 tablet_service.cc:1515] Processing DeleteTablet for tablet 59a1dd48d76241d7ba817070fbffd853 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:27.260332 27693 ts_tablet_manager.cc:1428] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.032s user 0.000s sys 0.002s
I20250903 20:05:27.261557 27493 tablet_service.cc:1468] Processing CreateTablet for tablet 1fce05bd4e6c425e952001ffdc25c78a (DEFAULT_TABLE table=default.Table_895e601507754fd399673ccb24b57a17 [id=1f3c305b81f1455aa8e0a41f91245276]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.264200 27693 tablet_bootstrap.cc:492] T e0347eab6c8642beabdc2d7e7bbfdc9b P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:27.265976 27321 leader_election.cc:304] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:27.266588 28579 raft_consensus.cc:2802] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:27.266701 28579 raft_consensus.cc:695] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:27.266907 28579 consensus_queue.cc:237] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.268534 28522 raft_consensus.cc:2802] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:27.270148 27693 tablet_bootstrap.cc:654] T e0347eab6c8642beabdc2d7e7bbfdc9b P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:27.270546 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 85a3a5f474824714bf0de51437a4016f (DEFAULT_TABLE table=default.Table_70cfd4abdbdf43e2b7665f787126a656 [id=8b208dd0f61342dbb6de8dce40574190]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.270954 27499 tablet_service.cc:1468] Processing CreateTablet for tablet 3dd25f75f0744e2ca56bd526a5af59b9 (DEFAULT_TABLE table=default.Table_bcd1ecf3ed1543db8c20807203abf4f6 [id=69c4eb2f8fea4a94807390505aba7d93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.271739 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 9925f905d5894880adce8b5d52b3cbf1 (DEFAULT_TABLE table=default.Table_dd4d357b91334362990bf7b6b3b9e874 [id=4d5b351dca0d4e48a579ab5ea547b187]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.272069 27494 tablet_service.cc:1468] Processing CreateTablet for tablet 88ee77b79c994d25a9c67e95b14c1a86 (DEFAULT_TABLE table=default.Table_7fbd1210b3c34421a34097b4e769674c [id=079e7b6922bf4f6b8ede232158dac557]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.272383 27489 tablet_service.cc:1468] Processing CreateTablet for tablet 8f4fef41c9714d41a65fef1b5473ce16 (DEFAULT_TABLE table=default.Table_a80111ac09d44f43bdfd7b3b9fda8171 [id=e193f9ac61df4dafab34f633cc6f2666]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.274775 28598 consensus_queue.cc:1035] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:27.280707 27507 tablet_service.cc:1468] Processing CreateTablet for tablet f28fd3e1e0e14cf48a244580b4f355d0 (DEFAULT_TABLE table=default.Table_b664a0b055d84bc09046969e26863021 [id=9d52e208b8184536871e012af39c2691]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.280807 27495 tablet_service.cc:1468] Processing CreateTablet for tablet ed73c335b042476fa03f154d23a611ad (DEFAULT_TABLE table=default.Table_3ef308651a284cc08daa6de000cc2522 [id=eedea747a15a458ebd47e9e5b9d16830]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.292691 28596 consensus_queue.cc:1035] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:27.298691 27695 ts_tablet_manager.cc:1428] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.126s user 0.014s sys 0.006s
I20250903 20:05:27.299170 27695 tablet_bootstrap.cc:492] T 1bad3c509460422396a17a3c4ae3d1e8 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:27.299757 27693 tablet_bootstrap.cc:492] T e0347eab6c8642beabdc2d7e7bbfdc9b P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:27.300040 27693 ts_tablet_manager.cc:1397] T e0347eab6c8642beabdc2d7e7bbfdc9b P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.001s
I20250903 20:05:27.300359 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 1fce05bd4e6c425e952001ffdc25c78a (DEFAULT_TABLE table=default.Table_895e601507754fd399673ccb24b57a17 [id=1f3c305b81f1455aa8e0a41f91245276]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.300652 27695 tablet_bootstrap.cc:654] T 1bad3c509460422396a17a3c4ae3d1e8 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:27.300823 27212 tablet_service.cc:1468] Processing CreateTablet for tablet 85a3a5f474824714bf0de51437a4016f (DEFAULT_TABLE table=default.Table_70cfd4abdbdf43e2b7665f787126a656 [id=8b208dd0f61342dbb6de8dce40574190]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.300841 27693 raft_consensus.cc:357] T e0347eab6c8642beabdc2d7e7bbfdc9b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.300995 27693 raft_consensus.cc:383] T e0347eab6c8642beabdc2d7e7bbfdc9b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:27.301045 27693 raft_consensus.cc:738] T e0347eab6c8642beabdc2d7e7bbfdc9b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:27.301256 27219 tablet_service.cc:1468] Processing CreateTablet for tablet 3dd25f75f0744e2ca56bd526a5af59b9 (DEFAULT_TABLE table=default.Table_bcd1ecf3ed1543db8c20807203abf4f6 [id=69c4eb2f8fea4a94807390505aba7d93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.301240 27693 consensus_queue.cc:260] T e0347eab6c8642beabdc2d7e7bbfdc9b P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.301644 27210 tablet_service.cc:1468] Processing CreateTablet for tablet 9925f905d5894880adce8b5d52b3cbf1 (DEFAULT_TABLE table=default.Table_dd4d357b91334362990bf7b6b3b9e874 [id=4d5b351dca0d4e48a579ab5ea547b187]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.301731 27693 ts_tablet_manager.cc:1428] T e0347eab6c8642beabdc2d7e7bbfdc9b P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:27.302030 27693 tablet_bootstrap.cc:492] T 466e501e424e46bc98e8bbc64f97005e P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:27.303556 27693 tablet_bootstrap.cc:654] T 466e501e424e46bc98e8bbc64f97005e P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:27.303709 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_90b62da1b4554249986db49d13942d17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:27.304353 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90b62da1b4554249986db49d13942d17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:27.310065 27062 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 0ad64c699b294679b6fec37b5ad23528 (table default.Table_24e55a7c820c413f8514bd3ad7bd3269 [id=0904dc5d0c594bc3bbf32de2508c6195]). Expected version 3 got 0
I20250903 20:05:27.313932 27695 tablet_bootstrap.cc:492] T 1bad3c509460422396a17a3c4ae3d1e8 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:27.313999 27350 tablet_service.cc:1468] Processing CreateTablet for tablet 85a3a5f474824714bf0de51437a4016f (DEFAULT_TABLE table=default.Table_70cfd4abdbdf43e2b7665f787126a656 [id=8b208dd0f61342dbb6de8dce40574190]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.314137 27695 ts_tablet_manager.cc:1397] T 1bad3c509460422396a17a3c4ae3d1e8 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.002s
I20250903 20:05:27.314188 27362 tablet_service.cc:1468] Processing CreateTablet for tablet 88ee77b79c994d25a9c67e95b14c1a86 (DEFAULT_TABLE table=default.Table_7fbd1210b3c34421a34097b4e769674c [id=079e7b6922bf4f6b8ede232158dac557]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.314450 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85a3a5f474824714bf0de51437a4016f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.314639 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88ee77b79c994d25a9c67e95b14c1a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.315039 27695 raft_consensus.cc:357] T 1bad3c509460422396a17a3c4ae3d1e8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.315291 27695 raft_consensus.cc:383] T 1bad3c509460422396a17a3c4ae3d1e8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:27.315517 27695 raft_consensus.cc:738] T 1bad3c509460422396a17a3c4ae3d1e8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:27.315810 27695 consensus_queue.cc:260] T 1bad3c509460422396a17a3c4ae3d1e8 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.316507 27695 ts_tablet_manager.cc:1428] T 1bad3c509460422396a17a3c4ae3d1e8 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:27.316817 27695 tablet_bootstrap.cc:492] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:27.317065 27352 tablet_service.cc:1468] Processing CreateTablet for tablet 9925f905d5894880adce8b5d52b3cbf1 (DEFAULT_TABLE table=default.Table_dd4d357b91334362990bf7b6b3b9e874 [id=4d5b351dca0d4e48a579ab5ea547b187]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.317432 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9925f905d5894880adce8b5d52b3cbf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.314005 27356 tablet_service.cc:1468] Processing CreateTablet for tablet 3dd25f75f0744e2ca56bd526a5af59b9 (DEFAULT_TABLE table=default.Table_bcd1ecf3ed1543db8c20807203abf4f6 [id=69c4eb2f8fea4a94807390505aba7d93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.318105 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dd25f75f0744e2ca56bd526a5af59b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.318040 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 8f4fef41c9714d41a65fef1b5473ce16 (DEFAULT_TABLE table=default.Table_a80111ac09d44f43bdfd7b3b9fda8171 [id=e193f9ac61df4dafab34f633cc6f2666]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.318393 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f4fef41c9714d41a65fef1b5473ce16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.319648 27695 tablet_bootstrap.cc:654] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:27.321918 27349 tablet_service.cc:1468] Processing CreateTablet for tablet f28fd3e1e0e14cf48a244580b4f355d0 (DEFAULT_TABLE table=default.Table_b664a0b055d84bc09046969e26863021 [id=9d52e208b8184536871e012af39c2691]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.322345 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f28fd3e1e0e14cf48a244580b4f355d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.326730 27693 tablet_bootstrap.cc:492] T 466e501e424e46bc98e8bbc64f97005e P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:27.327256 27693 ts_tablet_manager.cc:1397] T 466e501e424e46bc98e8bbc64f97005e P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.002s
I20250903 20:05:27.320008 27213 tablet_service.cc:1515] Processing DeleteTablet for tablet 6788b1683f984949816edfc5b68de8db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:27.328404 28655 raft_consensus.cc:491] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:27.328575 28655 raft_consensus.cc:513] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.328677 27693 raft_consensus.cc:357] T 466e501e424e46bc98e8bbc64f97005e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.328742 27213 tablet_service.cc:1468] Processing CreateTablet for tablet 8f4fef41c9714d41a65fef1b5473ce16 (DEFAULT_TABLE table=default.Table_a80111ac09d44f43bdfd7b3b9fda8171 [id=e193f9ac61df4dafab34f633cc6f2666]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.328866 27693 raft_consensus.cc:383] T 466e501e424e46bc98e8bbc64f97005e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:27.328914 27693 raft_consensus.cc:738] T 466e501e424e46bc98e8bbc64f97005e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:27.329241 28655 leader_election.cc:290] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:27.329952 27693 consensus_queue.cc:260] T 466e501e424e46bc98e8bbc64f97005e P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.330093 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cfe53b91b41d7ba7f9074e1005af3" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:27.330296 27381 raft_consensus.cc:2466] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:27.330495 27693 ts_tablet_manager.cc:1428] T 466e501e424e46bc98e8bbc64f97005e P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250903 20:05:27.330576 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cfe53b91b41d7ba7f9074e1005af3" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:27.330716 27184 leader_election.cc:304] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:27.330991 28655 raft_consensus.cc:2802] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:27.331077 28655 raft_consensus.cc:491] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:27.331148 28655 raft_consensus.cc:3058] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:27.331283 27183 leader_election.cc:343] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:27.331550 27693 tablet_bootstrap.cc:492] T 8a223f75045345ae833f52bcc8f239f9 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
W20250903 20:05:27.333784 27524 consensus_meta.cc:287] T 1bcb9d5f010f4297bab78c0183889238 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent flushing consensus metadata: real 0.557s user 0.002s sys 0.001s
I20250903 20:05:27.333811 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fce05bd4e6c425e952001ffdc25c78a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.333861 27685 ts_tablet_manager.cc:1918] T fa05f54ebc324b40855fa3288760e9e9 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:27.333896 27524 raft_consensus.cc:2466] T 1bcb9d5f010f4297bab78c0183889238 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:27.333974 27685 log.cc:1199] T fa05f54ebc324b40855fa3288760e9e9 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/fa05f54ebc324b40855fa3288760e9e9
I20250903 20:05:27.334332 27685 ts_tablet_manager.cc:1939] T fa05f54ebc324b40855fa3288760e9e9 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:27.334530 27219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dd25f75f0744e2ca56bd526a5af59b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.334956 27210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9925f905d5894880adce8b5d52b3cbf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.335373 27212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85a3a5f474824714bf0de51437a4016f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.336836 27526 raft_consensus.cc:2466] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
W20250903 20:05:27.337432 27490 consensus_meta.cc:287] T 2c6a2e2e2ec34f8082a17048ca0d18b0 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent flushing consensus metadata: real 0.550s user 0.004s sys 0.000s
I20250903 20:05:27.338078 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f4fef41c9714d41a65fef1b5473ce16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.338150 27248 raft_consensus.cc:2466] T 33fb6b357bda43e7abeaed5634183ef6 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:27.338570 27386 raft_consensus.cc:2466] T b0d79c78e0024b7ba3bb3356be7c8ceb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
W20250903 20:05:27.338747 27497 consensus_meta.cc:287] T 4120e1abd8434abb9cd92e3691d64cc7 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent flushing consensus metadata: real 0.557s user 0.003s sys 0.004s
I20250903 20:05:27.338794 27684 ts_tablet_manager.cc:1918] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:27.338878 27684 log.cc:1199] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/6f0cba70240f4f38a4fa5ccbcd99b3b1
I20250903 20:05:27.338934 27321 leader_election.cc:304] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:27.339138 27500 consensus_meta.cc:287] T f3185f9cf7c746418beeefb9e34758ff P e72ef1dbcc5143cf9f04a808f101c80f: Time spent flushing consensus metadata: real 0.548s user 0.003s sys 0.000s
I20250903 20:05:27.339227 27684 ts_tablet_manager.cc:1939] T 6f0cba70240f4f38a4fa5ccbcd99b3b1 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:27.339440 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fce05bd4e6c425e952001ffdc25c78a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.339958 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85a3a5f474824714bf0de51437a4016f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.340353 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dd25f75f0744e2ca56bd526a5af59b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.340711 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9925f905d5894880adce8b5d52b3cbf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.341089 27494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88ee77b79c994d25a9c67e95b14c1a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.341451 27489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f4fef41c9714d41a65fef1b5473ce16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.341832 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f28fd3e1e0e14cf48a244580b4f355d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.342772 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed73c335b042476fa03f154d23a611ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.365696 27692 ts_tablet_manager.cc:1428] T 1bcb9d5f010f4297bab78c0183889238 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.621s user 0.006s sys 0.000s
W20250903 20:05:27.365851 27692 ts_tablet_manager.cc:1460] T 1bcb9d5f010f4297bab78c0183889238 P e72ef1dbcc5143cf9f04a808f101c80f: Tablet startup took 621ms
W20250903 20:05:27.365928 27692 ts_tablet_manager.cc:1462] T 1bcb9d5f010f4297bab78c0183889238 P e72ef1dbcc5143cf9f04a808f101c80f: Trace:
0903 20:05:12.671079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:12.680276 (+ 9197us) service_pool.cc:225] Handling call
0903 20:05:12.680615 (+ 339us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0903 20:05:12.680633 (+ 18us) ts_tablet_manager.cc:676] Creating new metadata...
0903 20:05:12.696774 (+ 16141us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:12.759046 (+ 62272us) tablet_replica.cc:206] Creating consensus instance
0903 20:05:12.759185 (+ 139us) inbound_call.cc:173] Queueing success response
0903 20:05:26.693928 (+13934743us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0903 20:05:26.744530 (+ 50602us) ts_tablet_manager.cc:1429] Starting tablet replica
0903 20:05:26.744778 (+ 248us) tablet_replica.cc:260] Starting instrumentation
0903 20:05:26.744965 (+ 187us) tablet_replica.cc:279] Starting consensus
0903 20:05:27.365270 (+620305us) spinlock_profiling.cc:243] Waited 560 ms on lock 0x616000106c54. stack: 00007f71bc6f7dc6 00007f71bc6f7b80 00007f71bb6df3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f71c8146c86 00007f71c814278d 00007f71c949295d 00007f71cbf730c3 00007f71cbf8967a 00007f71cbf893bd 0000000000a774dc 00007f71bc73e14f 00007f71bc74141b 00007f71bc741298
Metrics: {"mutex_wait_us":11251,"spinlock_wait_cycles":1287875840,"tablet-open.queue_time_us":13934772}
I20250903 20:05:27.366696 27692 tablet_bootstrap.cc:492] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:27.374042 27692 tablet_bootstrap.cc:654] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:27.377045 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_2e1fdc83e24243fca99d90876fb34882"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:27.377993 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e1fdc83e24243fca99d90876fb34882 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:27.379793 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 786) took 1721 ms. Trace:
I20250903 20:05:27.379907 27684 rpcz_store.cc:276] 0903 20:05:25.658274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:25.658333 (+ 59us) service_pool.cc:225] Handling call
0903 20:05:27.183313 (+1524980us) ts_tablet_manager.cc:1145] Deleting tablet 6f0cba70240f4f38a4fa5ccbcd99b3b1
0903 20:05:27.183323 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:27.239315 (+ 55992us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:27.338692 (+ 99377us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:27.379766 (+ 41074us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1524846}
I20250903 20:05:27.380326 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 783) took 1725 ms. Trace:
I20250903 20:05:27.380396 27684 rpcz_store.cc:276] 0903 20:05:25.655291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:25.658496 (+ 3205us) service_pool.cc:225] Handling call
0903 20:05:27.380174 (+1721678us) ts_tablet_manager.cc:1145] Deleting tablet 3d875d29e66d48ed872e38d20f30d9c0
0903 20:05:27.380183 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:27.380309 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1721548}
I20250903 20:05:27.380663 27684 tablet_replica.cc:331] T 8605bc329627438998812ed028327f53 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:27.380877 27684 raft_consensus.cc:2241] T 8605bc329627438998812ed028327f53 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:27.381021 27684 raft_consensus.cc:2270] T 8605bc329627438998812ed028327f53 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:27.381568 27684 ts_tablet_manager.cc:1905] T 8605bc329627438998812ed028327f53 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:27.382843 28596 raft_consensus.cc:491] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:27.382972 28596 raft_consensus.cc:513] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.383544 28596 leader_election.cc:290] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:27.385378 27359 tablet_service.cc:1468] Processing CreateTablet for tablet ed73c335b042476fa03f154d23a611ad (DEFAULT_TABLE table=default.Table_3ef308651a284cc08daa6de000cc2522 [id=eedea747a15a458ebd47e9e5b9d16830]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.385911 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed73c335b042476fa03f154d23a611ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.386328 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bcb9d5f010f4297bab78c0183889238 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a9e55f0811345f7afa71645f38ebdaf) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:27.388470 28579 raft_consensus.cc:2802] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:27.388622 28579 raft_consensus.cc:695] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:27.388917 28579 consensus_queue.cc:237] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.389621 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 6f0cba70240f4f38a4fa5ccbcd99b3b1 (table default.Table_5ca5478ef07f4acdbbedb24af75f9e04 [id=7fb235b85bba4f7ab1b77bc5d2b9b58c]) successfully deleted
I20250903 20:05:27.389855 27057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_895e601507754fd399673ccb24b57a17" } new_table_name: "default.Table_f04fc2303d4c487f8d21543807e16df5" modify_external_catalogs: true
W20250903 20:05:27.390275 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 3d875d29e66d48ed872e38d20f30d9c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d875d29e66d48ed872e38d20f30d9c0 already in progress: creating tablet
I20250903 20:05:27.342197 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 88ee77b79c994d25a9c67e95b14c1a86 (DEFAULT_TABLE table=default.Table_7fbd1210b3c34421a34097b4e769674c [id=079e7b6922bf4f6b8ede232158dac557]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.392366 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88ee77b79c994d25a9c67e95b14c1a86. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:27.392758 27462 leader_election.cc:343] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: c9e90abef9814338b70a3550ab5686e0
I20250903 20:05:27.347909 27221 tablet_service.cc:1468] Processing CreateTablet for tablet ed73c335b042476fa03f154d23a611ad (DEFAULT_TABLE table=default.Table_3ef308651a284cc08daa6de000cc2522 [id=eedea747a15a458ebd47e9e5b9d16830]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.396987 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed73c335b042476fa03f154d23a611ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.384366 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9e90abef9814338b70a3550ab5686e0" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:27.400661 27248 raft_consensus.cc:2466] T c9e90abef9814338b70a3550ab5686e0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:27.401158 27462 leader_election.cc:304] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:27.389483 27225 tablet_service.cc:1468] Processing CreateTablet for tablet f28fd3e1e0e14cf48a244580b4f355d0 (DEFAULT_TABLE table=default.Table_b664a0b055d84bc09046969e26863021 [id=9d52e208b8184536871e012af39c2691]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.401679 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f28fd3e1e0e14cf48a244580b4f355d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.401759 28596 raft_consensus.cc:2802] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:27.401882 28596 raft_consensus.cc:491] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:27.401984 28596 raft_consensus.cc:3058] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:27.402480 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet edac2355f67045a6acffaae723d9186f
I20250903 20:05:27.403247 27347 tablet_service.cc:1515] Processing DeleteTablet for tablet edac2355f67045a6acffaae723d9186f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:27.403463 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 91169fff8a6649828648ac97ccf65fdc
I20250903 20:05:27.403378 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_331a93e34699477294aedde756d58a8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:27.404032 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_331a93e34699477294aedde756d58a8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:27.404242 27347 tablet_service.cc:1515] Processing DeleteTablet for tablet 91169fff8a6649828648ac97ccf65fdc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:27.404234 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_711e173c8a754e2b850b887bdb54e5ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:27.404791 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_711e173c8a754e2b850b887bdb54e5ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:27.406384 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_52c0d1912aa744ebb10e196e7c81dc2e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:27.406922 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_52c0d1912aa744ebb10e196e7c81dc2e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:27.407485 28567 raft_consensus.cc:491] T 7e2f98f760cf4df4bc2a10530e2157c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:27.407624 28567 raft_consensus.cc:513] T 7e2f98f760cf4df4bc2a10530e2157c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.407893 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 91169fff8a6649828648ac97ccf65fdc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:27.408125 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet edac2355f67045a6acffaae723d9186f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:27.408314 28567 leader_election.cc:290] T 7e2f98f760cf4df4bc2a10530e2157c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:27.408288 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet a686ea49a29a4dd7b6e5dce5e1f9935a
I20250903 20:05:27.408800 27065 catalog_manager.cc:6652] ReplaceTablet: tablet a686ea49a29a4dd7b6e5dce5e1f9935a deleted and replaced by tablet 9657ef024273400398d5e76f15359c40
I20250903 20:05:27.409004 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e2f98f760cf4df4bc2a10530e2157c6" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:27.409173 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_18a6310ff569456f91088bbafcfd50f4" } new_table_name: "default.Table_80049807e89942ef883f2be3b0513c63" modify_external_catalogs: true
W20250903 20:05:27.409518 27321 leader_election.cc:343] T 7e2f98f760cf4df4bc2a10530e2157c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:27.409788 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e2f98f760cf4df4bc2a10530e2157c6" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:27.410333 27320 leader_election.cc:343] T 7e2f98f760cf4df4bc2a10530e2157c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:27.410435 27320 leader_election.cc:304] T 7e2f98f760cf4df4bc2a10530e2157c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:27.410832 28567 raft_consensus.cc:2747] T 7e2f98f760cf4df4bc2a10530e2157c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:27.412611 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 112c0dccb60c4c7c9efc164d9c8443ea
I20250903 20:05:27.413389 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet cf99a74acfdb4a17bbbce3a07467e9c1
I20250903 20:05:27.417316 27693 tablet_bootstrap.cc:654] T 8a223f75045345ae833f52bcc8f239f9 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:27.419816 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 30cc0d4ba8d444f1b98232f906ea0c7b
I20250903 20:05:27.426146 27693 tablet_bootstrap.cc:492] T 8a223f75045345ae833f52bcc8f239f9 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:27.426364 27693 ts_tablet_manager.cc:1397] T 8a223f75045345ae833f52bcc8f239f9 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.095s user 0.003s sys 0.004s
I20250903 20:05:27.428359 27695 tablet_bootstrap.cc:492] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:27.428895 28568 raft_consensus.cc:513] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.429837 28568 leader_election.cc:290] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:27.431125 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 821) took 1230 ms. Trace:
I20250903 20:05:27.431268 27687 rpcz_store.cc:276] 0903 20:05:26.200856 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.200922 (+ 66us) service_pool.cc:225] Handling call
0903 20:05:27.042658 (+841736us) ts_tablet_manager.cc:1145] Deleting tablet 6788b1683f984949816edfc5b68de8db
0903 20:05:27.042666 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:27.133835 (+ 91169us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:27.238416 (+104581us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:27.431075 (+192659us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":841540}
I20250903 20:05:27.431810 27687 tablet_replica.cc:331] T 1bcb9d5f010f4297bab78c0183889238 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:27.432050 27687 raft_consensus.cc:2241] T 1bcb9d5f010f4297bab78c0183889238 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:27.432245 27687 raft_consensus.cc:2270] T 1bcb9d5f010f4297bab78c0183889238 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:27.432964 27687 ts_tablet_manager.cc:1905] T 1bcb9d5f010f4297bab78c0183889238 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:27.434871 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_42e129c0c0c747e281d1dc5eb2862dd9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:27.435616 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42e129c0c0c747e281d1dc5eb2862dd9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:27.436034 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0efbb7eefb84ab6bee5b9625b91e298" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:27.437402 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c424d5c674514a0e8471cd35f357ea56" } new_table_name: "default.Table_ccdb4130549b4be5888f17097ae35106" modify_external_catalogs: true
I20250903 20:05:27.439059 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6ab02d2c339e463fbed5c3429c7c4ec9" } new_table_name: "default.Table_73a4ac2afbad4654ac623cd581a45eb2" modify_external_catalogs: true
I20250903 20:05:27.441254 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2b5e97597b354141839a4ec0ca6bf046" } new_table_name: "default.Table_a7ccf32c61af4373a384d46406910149" modify_external_catalogs: true
W20250903 20:05:27.443939 27320 leader_election.cc:343] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:27.447976 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet cf99a74acfdb4a17bbbce3a07467e9c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:27.448309 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 112c0dccb60c4c7c9efc164d9c8443ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:27.448547 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet a686ea49a29a4dd7b6e5dce5e1f9935a with delete_type TABLET_DATA_DELETED (replaced by tablet 9657ef024273400398d5e76f15359c40) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:27.448783 27489 tablet_service.cc:1515] Processing DeleteTablet for tablet 30cc0d4ba8d444f1b98232f906ea0c7b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:27.449393 28596 raft_consensus.cc:513] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.450093 27348 tablet_service.cc:1515] Processing DeleteTablet for tablet 112c0dccb60c4c7c9efc164d9c8443ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:27.450345 27347 tablet_service.cc:1515] Processing DeleteTablet for tablet a686ea49a29a4dd7b6e5dce5e1f9935a with delete_type TABLET_DATA_DELETED (replaced by tablet 9657ef024273400398d5e76f15359c40) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:27.450767 27347 tablet_service.cc:1515] Processing DeleteTablet for tablet cf99a74acfdb4a17bbbce3a07467e9c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:27.451184 28596 leader_election.cc:290] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:27.451385 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 6788b1683f984949816edfc5b68de8db (table default.Table_48f65d1e5e0b4836bb4df770bd94b4e0 [id=45702f222451484c80915421a0cdeb68]) successfully deleted
I20250903 20:05:27.451457 27347 tablet_service.cc:1515] Processing DeleteTablet for tablet 30cc0d4ba8d444f1b98232f906ea0c7b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:27.451668 27462 leader_election.cc:343] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: c9e90abef9814338b70a3550ab5686e0
I20250903 20:05:27.430339 28655 raft_consensus.cc:513] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.466871 28655 leader_election.cc:290] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:27.434829 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 680) took 4873 ms. Trace:
I20250903 20:05:27.467090 27685 rpcz_store.cc:276] 0903 20:05:22.561571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.565115 (+ 3544us) service_pool.cc:225] Handling call
0903 20:05:27.152422 (+4587307us) ts_tablet_manager.cc:1145] Deleting tablet fa05f54ebc324b40855fa3288760e9e9
0903 20:05:27.152432 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:27.238624 (+ 86192us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:27.333750 (+ 95126us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:27.434798 (+101048us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4586978}
I20250903 20:05:27.467314 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cfe53b91b41d7ba7f9074e1005af3" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:27.467527 27386 raft_consensus.cc:3058] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:27.467669 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 683) took 4871 ms. Trace:
I20250903 20:05:27.467764 27685 rpcz_store.cc:276] 0903 20:05:22.596651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.622295 (+ 25644us) service_pool.cc:225] Handling call
0903 20:05:27.467486 (+4845191us) ts_tablet_manager.cc:1145] Deleting tablet f9375fff5f5945078c4412aa21c06f0d
0903 20:05:27.467505 (+ 19us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:27.467645 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4844895}
I20250903 20:05:27.468142 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 681) took 4878 ms. Trace:
I20250903 20:05:27.468235 27685 rpcz_store.cc:276] 0903 20:05:22.589774 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.643648 (+ 53874us) service_pool.cc:225] Handling call
0903 20:05:27.468020 (+4824372us) ts_tablet_manager.cc:1145] Deleting tablet 659a127405d341c7b074d3445854256f
0903 20:05:27.468033 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:27.468124 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4824107}
I20250903 20:05:27.468550 27685 tablet_replica.cc:331] T cb4d99139516405bb65c7480ddf259f1 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:27.468801 27685 raft_consensus.cc:2241] T cb4d99139516405bb65c7480ddf259f1 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:27.468961 27685 raft_consensus.cc:2270] T cb4d99139516405bb65c7480ddf259f1 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:27.469610 27685 ts_tablet_manager.cc:1905] T cb4d99139516405bb65c7480ddf259f1 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:27.469940 27219 tablet_service.cc:1515] Processing DeleteTablet for tablet cf99a74acfdb4a17bbbce3a07467e9c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:27.436550 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0efbb7eefb84ab6bee5b9625b91e298" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:27.470804 27248 raft_consensus.cc:3058] T f0efbb7eefb84ab6bee5b9625b91e298 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:27.450570 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9e90abef9814338b70a3550ab5686e0" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:27.473666 27249 raft_consensus.cc:3058] T c9e90abef9814338b70a3550ab5686e0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:27.474632 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cfe53b91b41d7ba7f9074e1005af3" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:27.475008 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet fa05f54ebc324b40855fa3288760e9e9 (table default.Table_dacd8238c8334c1c9f66e745f5036a06 [id=7ec413d945394dcfa36492f7b59aaaa2]) successfully deleted
W20250903 20:05:27.475339 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet f9375fff5f5945078c4412aa21c06f0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9375fff5f5945078c4412aa21c06f0d already in progress: creating tablet
W20250903 20:05:27.475936 27183 leader_election.cc:343] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:27.477741 27057 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1fce05bd4e6c425e952001ffdc25c78a consensus state has no leader
W20250903 20:05:27.480482 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3618bfd17de4408d870cd206b185d951 consensus state has no leader
I20250903 20:05:27.454762 27213 tablet_service.cc:1515] Processing DeleteTablet for tablet 30cc0d4ba8d444f1b98232f906ea0c7b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:27.461427 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet 91169fff8a6649828648ac97ccf65fdc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:27.461699 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet 112c0dccb60c4c7c9efc164d9c8443ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:27.461895 27224 tablet_service.cc:1515] Processing DeleteTablet for tablet a686ea49a29a4dd7b6e5dce5e1f9935a with delete_type TABLET_DATA_DELETED (replaced by tablet 9657ef024273400398d5e76f15359c40) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:27.462566 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet edac2355f67045a6acffaae723d9186f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:27.482143 27695 ts_tablet_manager.cc:1397] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.165s user 0.001s sys 0.003s
I20250903 20:05:27.483052 27695 raft_consensus.cc:357] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.483254 27695 raft_consensus.cc:383] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:27.483311 27695 raft_consensus.cc:738] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
W20250903 20:05:27.483500 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6d4b8d4bf29e408cbdf9557c8b0d8a4b consensus state has no leader
W20250903 20:05:27.485905 27066 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a9f43781299e47a8aecc37a0b73d7a64 consensus state has no leader
I20250903 20:05:27.486617 27693 raft_consensus.cc:357] T 8a223f75045345ae833f52bcc8f239f9 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.487035 27693 raft_consensus.cc:383] T 8a223f75045345ae833f52bcc8f239f9 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:27.487098 27693 raft_consensus.cc:738] T 8a223f75045345ae833f52bcc8f239f9 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:27.487380 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_5f424141e5ff4643903a98bfeb011982"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:27.488104 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f424141e5ff4643903a98bfeb011982 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:27.488839 27693 consensus_queue.cc:260] T 8a223f75045345ae833f52bcc8f239f9 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.489465 27693 ts_tablet_manager.cc:1428] T 8a223f75045345ae833f52bcc8f239f9 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.063s user 0.001s sys 0.000s
I20250903 20:05:27.489550 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_d9f8debc2d944aa295651daee0dcb326"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:27.489773 27693 tablet_bootstrap.cc:492] T a72b85dfb34047528b2b28ff0d40f828 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
W20250903 20:05:27.490163 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9f8debc2d944aa295651daee0dcb326 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:27.490422 27695 consensus_queue.cc:260] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.491030 27695 ts_tablet_manager.cc:1428] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20250903 20:05:27.491338 27695 tablet_bootstrap.cc:492] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:27.493531 27693 tablet_bootstrap.cc:654] T a72b85dfb34047528b2b28ff0d40f828 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:27.496460 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c6cebb39bdb148d1bcbb47c4160a7be4 consensus state has no leader
I20250903 20:05:27.497320 27692 tablet_bootstrap.cc:492] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:27.497479 27692 ts_tablet_manager.cc:1397] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.131s user 0.001s sys 0.006s
I20250903 20:05:27.498358 27692 raft_consensus.cc:357] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.498644 27692 raft_consensus.cc:383] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:27.498757 27692 raft_consensus.cc:738] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:27.499092 27692 consensus_queue.cc:260] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.499688 27692 ts_tablet_manager.cc:1428] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:27.504644 27692 tablet_bootstrap.cc:492] T a4c582b441f6467c99ba0b090abcf4cc P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:27.504958 27695 tablet_bootstrap.cc:654] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:27.507562 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 514e382479014fa7bf1f404efff11c86 with delete_type TABLET_DATA_DELETED (replaced by tablet 07439db2c3b8470eb2b797cc68376aa7) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:27.508270 27692 tablet_bootstrap.cc:654] T a4c582b441f6467c99ba0b090abcf4cc P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:27.515861 27692 tablet_bootstrap.cc:492] T a4c582b441f6467c99ba0b090abcf4cc P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:27.516042 27692 ts_tablet_manager.cc:1397] T a4c582b441f6467c99ba0b090abcf4cc P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.002s
I20250903 20:05:27.517287 27362 tablet_service.cc:1468] Processing CreateTablet for tablet e4ec4cc1ef914cffa2d304cdf701cf33 (DEFAULT_TABLE table=default.Table_127e4ebeb1d54cb48599c668f7c2aa49 [id=c77b5f1151664c58a03a03a9445c3afe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.516899 27692 raft_consensus.cc:357] T a4c582b441f6467c99ba0b090abcf4cc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.517663 27692 raft_consensus.cc:383] T a4c582b441f6467c99ba0b090abcf4cc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:27.517726 27692 raft_consensus.cc:738] T a4c582b441f6467c99ba0b090abcf4cc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:27.518273 27692 consensus_queue.cc:260] T a4c582b441f6467c99ba0b090abcf4cc P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.518793 27692 ts_tablet_manager.cc:1428] T a4c582b441f6467c99ba0b090abcf4cc P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:27.520534 27692 tablet_bootstrap.cc:492] T f2cc35f4244a42eeb4a65de749824c11 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:27.521139 27506 tablet_service.cc:1468] Processing CreateTablet for tablet e4ec4cc1ef914cffa2d304cdf701cf33 (DEFAULT_TABLE table=default.Table_127e4ebeb1d54cb48599c668f7c2aa49 [id=c77b5f1151664c58a03a03a9445c3afe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.521742 27693 tablet_bootstrap.cc:492] T a72b85dfb34047528b2b28ff0d40f828 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:27.521879 27693 ts_tablet_manager.cc:1397] T a72b85dfb34047528b2b28ff0d40f828 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.000s
I20250903 20:05:27.522598 27692 tablet_bootstrap.cc:654] T f2cc35f4244a42eeb4a65de749824c11 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:27.522791 27693 raft_consensus.cc:357] T a72b85dfb34047528b2b28ff0d40f828 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.523039 27693 raft_consensus.cc:383] T a72b85dfb34047528b2b28ff0d40f828 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:27.523191 27693 raft_consensus.cc:738] T a72b85dfb34047528b2b28ff0d40f828 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:27.523649 27693 consensus_queue.cc:260] T a72b85dfb34047528b2b28ff0d40f828 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.524088 27695 tablet_bootstrap.cc:492] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:27.524122 27218 tablet_service.cc:1468] Processing CreateTablet for tablet e4ec4cc1ef914cffa2d304cdf701cf33 (DEFAULT_TABLE table=default.Table_127e4ebeb1d54cb48599c668f7c2aa49 [id=c77b5f1151664c58a03a03a9445c3afe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.524233 27695 ts_tablet_manager.cc:1397] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.002s
I20250903 20:05:27.525144 27693 ts_tablet_manager.cc:1428] T a72b85dfb34047528b2b28ff0d40f828 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:27.525136 27695 raft_consensus.cc:357] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.525313 27695 raft_consensus.cc:383] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:27.525375 27695 raft_consensus.cc:738] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:27.525600 27695 consensus_queue.cc:260] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.526113 27695 ts_tablet_manager.cc:1428] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:27.526429 27695 tablet_bootstrap.cc:492] T 1fbe54dc40dd47daa109ceeb07546b0a P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:27.527390 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_1cbf3c0f4048429680ea8fbf4b60a761"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:27.527884 27356 tablet_service.cc:1468] Processing CreateTablet for tablet cc863e1c03ef4a5dae16bad0abc3b099 (DEFAULT_TABLE table=default.Table_2943f6ec93054a3e8cd8b08ccb0ee025 [id=229d65d6292f4ca78c2e88e58b4e1617]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:27.528038 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1cbf3c0f4048429680ea8fbf4b60a761 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:27.529102 27062 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 0ad64c699b294679b6fec37b5ad23528 (table default.Table_24e55a7c820c413f8514bd3ad7bd3269 [id=0904dc5d0c594bc3bbf32de2508c6195]). Expected version 3 got 0
I20250903 20:05:27.529430 27062 catalog_manager.cc:5582] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66). New cstate: current_term: 1 leader_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:27.530179 27062 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 0f5b87c1c4a24f8abdbfd6194f3b2c4a (table default.Table_6e9cf5ead5a24831a6a14d131dd8397e [id=524a27cbe1ad4c10aa2fd4f4074105c6]). Expected version 3 got 0
I20250903 20:05:27.531141 27499 tablet_service.cc:1468] Processing CreateTablet for tablet cc863e1c03ef4a5dae16bad0abc3b099 (DEFAULT_TABLE table=default.Table_2943f6ec93054a3e8cd8b08ccb0ee025 [id=229d65d6292f4ca78c2e88e58b4e1617]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.531486 27687 ts_tablet_manager.cc:1918] T 1bcb9d5f010f4297bab78c0183889238 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:27.531625 27687 log.cc:1199] T 1bcb9d5f010f4297bab78c0183889238 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/1bcb9d5f010f4297bab78c0183889238
I20250903 20:05:27.532915 27684 ts_tablet_manager.cc:1918] T 8605bc329627438998812ed028327f53 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:27.533061 27684 log.cc:1199] T 8605bc329627438998812ed028327f53 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/8605bc329627438998812ed028327f53
I20250903 20:05:27.533360 27684 ts_tablet_manager.cc:1939] T 8605bc329627438998812ed028327f53 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:27.533694 27248 raft_consensus.cc:2466] T f0efbb7eefb84ab6bee5b9625b91e298 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
W20250903 20:05:27.534598 27462 consensus_peers.cc:489] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6d67cd0c0304c77ba571b414878dad2. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:27.537079 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4ec4cc1ef914cffa2d304cdf701cf33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.538537 27495 tablet_service.cc:1468] Processing CreateTablet for tablet 5bff3d77088c4b69a90365950641e300 (DEFAULT_TABLE table=default.Table_716897f657da43e5986ff43425cedd20 [id=95151fe8350941e2a8c36d5c1ffbc138]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.533715 27224 tablet_service.cc:1468] Processing CreateTablet for tablet cc863e1c03ef4a5dae16bad0abc3b099 (DEFAULT_TABLE table=default.Table_2943f6ec93054a3e8cd8b08ccb0ee025 [id=229d65d6292f4ca78c2e88e58b4e1617]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.541244 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc863e1c03ef4a5dae16bad0abc3b099. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.541666 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 5bff3d77088c4b69a90365950641e300 (DEFAULT_TABLE table=default.Table_716897f657da43e5986ff43425cedd20 [id=95151fe8350941e2a8c36d5c1ffbc138]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.542008 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bff3d77088c4b69a90365950641e300. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.544229 28522 raft_consensus.cc:695] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:27.544852 28522 consensus_queue.cc:237] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.547677 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc863e1c03ef4a5dae16bad0abc3b099. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.548259 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4ec4cc1ef914cffa2d304cdf701cf33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.553707 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bff3d77088c4b69a90365950641e300. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.536301 27350 tablet_service.cc:1468] Processing CreateTablet for tablet 5bff3d77088c4b69a90365950641e300 (DEFAULT_TABLE table=default.Table_716897f657da43e5986ff43425cedd20 [id=95151fe8350941e2a8c36d5c1ffbc138]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.536986 27321 leader_election.cc:304] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:27.555197 28579 raft_consensus.cc:2802] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:27.555126 27506 tablet_service.cc:1468] Processing CreateTablet for tablet a7c8657489734199b346033bd4afb02e (DEFAULT_TABLE table=default.Table_6f835fbb9a1f4aff87000ff5e65133f8 [id=79f46d6238d747af9202c084e031b6d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.555303 28579 raft_consensus.cc:695] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:27.555488 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7c8657489734199b346033bd4afb02e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.555534 28579 consensus_queue.cc:237] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.556530 28567 raft_consensus.cc:491] T 8a223f75045345ae833f52bcc8f239f9 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:27.556643 28567 raft_consensus.cc:513] T 8a223f75045345ae833f52bcc8f239f9 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.557147 28567 leader_election.cc:290] T 8a223f75045345ae833f52bcc8f239f9 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:27.547448 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4ec4cc1ef914cffa2d304cdf701cf33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.557562 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc863e1c03ef4a5dae16bad0abc3b099. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.557736 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bff3d77088c4b69a90365950641e300. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.558286 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a223f75045345ae833f52bcc8f239f9" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:27.558796 27320 leader_election.cc:343] T 8a223f75045345ae833f52bcc8f239f9 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:27.559846 28596 raft_consensus.cc:491] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:27.559959 28596 raft_consensus.cc:513] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.560503 28596 leader_election.cc:290] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:27.553340 27347 tablet_service.cc:1468] Processing CreateTablet for tablet a7c8657489734199b346033bd4afb02e (DEFAULT_TABLE table=default.Table_6f835fbb9a1f4aff87000ff5e65133f8 [id=79f46d6238d747af9202c084e031b6d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.561051 27214 tablet_service.cc:1468] Processing CreateTablet for tablet a7c8657489734199b346033bd4afb02e (DEFAULT_TABLE table=default.Table_6f835fbb9a1f4aff87000ff5e65133f8 [id=79f46d6238d747af9202c084e031b6d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.561163 27349 tablet_service.cc:1468] Processing CreateTablet for tablet dcdddd231de24ec2944a2ffac7b8fb58 (DEFAULT_TABLE table=default.Table_31cccd71f85842908650e5d1a49c342f [id=2c77d98ed226438688032a7cf62dbc93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.561337 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7c8657489734199b346033bd4afb02e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.561543 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcdddd231de24ec2944a2ffac7b8fb58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.562753 27246 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "514e382479014fa7bf1f404efff11c86" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:27.562966 27246 raft_consensus.cc:2391] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
I20250903 20:05:27.563064 27489 tablet_service.cc:1468] Processing CreateTablet for tablet dcdddd231de24ec2944a2ffac7b8fb58 (DEFAULT_TABLE table=default.Table_31cccd71f85842908650e5d1a49c342f [id=2c77d98ed226438688032a7cf62dbc93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.563406 27489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcdddd231de24ec2944a2ffac7b8fb58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.563766 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a223f75045345ae833f52bcc8f239f9" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:27.564716 27321 leader_election.cc:343] T 8a223f75045345ae833f52bcc8f239f9 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:27.564821 27321 leader_election.cc:304] T 8a223f75045345ae833f52bcc8f239f9 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:27.565212 28568 raft_consensus.cc:2747] T 8a223f75045345ae833f52bcc8f239f9 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:27.566663 27462 leader_election.cc:343] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 514e382479014fa7bf1f404efff11c86
I20250903 20:05:27.566774 27462 leader_election.cc:304] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:27.567130 28596 raft_consensus.cc:3058] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:27.531930 27687 ts_tablet_manager.cc:1939] T 1bcb9d5f010f4297bab78c0183889238 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:27.571024 27222 tablet_service.cc:1468] Processing CreateTablet for tablet dcdddd231de24ec2944a2ffac7b8fb58 (DEFAULT_TABLE table=default.Table_31cccd71f85842908650e5d1a49c342f [id=2c77d98ed226438688032a7cf62dbc93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.571512 27693 tablet_bootstrap.cc:492] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:27.573283 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 1fbe54dc40dd47daa109ceeb07546b0a with delete_type TABLET_DATA_DELETED (replaced by tablet 5b46bb34ac214b1ba23fba1a33f746ac) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:27.573875 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 8808c2668707437f9db006c7f9a91e70 (DEFAULT_TABLE table=default.Table_656f015bd5f84e7c92ecf269159af5de [id=d032d5b4a8e442cfacf4ef2bccea8401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.574481 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8808c2668707437f9db006c7f9a91e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.574770 27693 tablet_bootstrap.cc:654] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:27.577292 27695 tablet_bootstrap.cc:654] T 1fbe54dc40dd47daa109ceeb07546b0a P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:27.580307 27495 tablet_service.cc:1468] Processing CreateTablet for tablet 8808c2668707437f9db006c7f9a91e70 (DEFAULT_TABLE table=default.Table_656f015bd5f84e7c92ecf269159af5de [id=d032d5b4a8e442cfacf4ef2bccea8401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.580710 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8808c2668707437f9db006c7f9a91e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.585076 27692 tablet_bootstrap.cc:492] T f2cc35f4244a42eeb4a65de749824c11 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:27.585160 27352 tablet_service.cc:1515] Processing DeleteTablet for tablet e0347eab6c8642beabdc2d7e7bbfdc9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:27.585595 27692 ts_tablet_manager.cc:1397] T f2cc35f4244a42eeb4a65de749824c11 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.065s user 0.003s sys 0.000s
I20250903 20:05:27.585619 27352 tablet_service.cc:1515] Processing DeleteTablet for tablet a4f3230f50ec4c95b528fead8c191224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:27.586548 27692 raft_consensus.cc:357] T f2cc35f4244a42eeb4a65de749824c11 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.587643 27213 tablet_service.cc:1468] Processing CreateTablet for tablet 8808c2668707437f9db006c7f9a91e70 (DEFAULT_TABLE table=default.Table_656f015bd5f84e7c92ecf269159af5de [id=d032d5b4a8e442cfacf4ef2bccea8401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.588024 27692 raft_consensus.cc:383] T f2cc35f4244a42eeb4a65de749824c11 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:27.588168 27348 tablet_service.cc:1468] Processing CreateTablet for tablet ea2d1d9e7e724aa08dfec812dad9589e (DEFAULT_TABLE table=default.Table_fa60556c9a9b433a9eff6f68ee824909 [id=d844bf3779574ad4a9868cffbba3d91a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.588359 27692 raft_consensus.cc:738] T f2cc35f4244a42eeb4a65de749824c11 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:27.588549 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea2d1d9e7e724aa08dfec812dad9589e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.588654 27692 consensus_queue.cc:260] T f2cc35f4244a42eeb4a65de749824c11 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.589846 27695 tablet_bootstrap.cc:492] T 1fbe54dc40dd47daa109ceeb07546b0a P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:27.589964 27695 ts_tablet_manager.cc:1397] T 1fbe54dc40dd47daa109ceeb07546b0a P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.000s
I20250903 20:05:27.588558 27352 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e2f98f760cf4df4bc2a10530e2157c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:27.590642 27695 raft_consensus.cc:357] T 1fbe54dc40dd47daa109ceeb07546b0a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.590853 27695 raft_consensus.cc:383] T 1fbe54dc40dd47daa109ceeb07546b0a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:27.590942 27695 raft_consensus.cc:738] T 1fbe54dc40dd47daa109ceeb07546b0a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:27.591234 27695 consensus_queue.cc:260] T 1fbe54dc40dd47daa109ceeb07546b0a P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.591928 27695 ts_tablet_manager.cc:1428] T 1fbe54dc40dd47daa109ceeb07546b0a P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:27.592242 27695 tablet_bootstrap.cc:492] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:27.592792 27352 tablet_service.cc:1515] Processing DeleteTablet for tablet 466e501e424e46bc98e8bbc64f97005e with delete_type TABLET_DATA_DELETED (replaced by tablet 06a79706f1184fa49d10182ba669b0ab) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:27.599704 27064 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet d859c0222c3647dab3dddd105e4cf0fc (table default.Table_dc24dd98fc4146eda8660576fd53a29d [id=7add4b636f1b4bffae663fdf432de05f]). Expected version 4 got 0
I20250903 20:05:27.600087 27064 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 0ad64c699b294679b6fec37b5ad23528 (table default.Table_24e55a7c820c413f8514bd3ad7bd3269 [id=0904dc5d0c594bc3bbf32de2508c6195]). Expected version 3 got 0
I20250903 20:05:27.600382 27489 tablet_service.cc:1468] Processing CreateTablet for tablet ea2d1d9e7e724aa08dfec812dad9589e (DEFAULT_TABLE table=default.Table_fa60556c9a9b433a9eff6f68ee824909 [id=d844bf3779574ad4a9868cffbba3d91a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.600450 27064 catalog_manager.cc:5582] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 89f8f0ee68b940db9ce250c0604f934a reported cstate change: term changed from 0 to 1, leader changed from <none> to 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65). New cstate: current_term: 1 leader_uuid: "89f8f0ee68b940db9ce250c0604f934a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:27.600760 27489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea2d1d9e7e724aa08dfec812dad9589e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.600843 27064 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 3b3bfc63aa0b4e56bec189e834dc6a8d (table default.Table_9b20469406c94986ab496ea3e678b586 [id=6d6f5e8cadf9445d8695c075cfabb2ea]). Expected version 1 got 0
I20250903 20:05:27.601159 27064 catalog_manager.cc:5582] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a reported cstate change: term changed from 0 to 1, leader changed from <none> to 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65). New cstate: current_term: 1 leader_uuid: "89f8f0ee68b940db9ce250c0604f934a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:27.601521 27064 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet b0d79c78e0024b7ba3bb3356be7c8ceb (table default.Table_94e9e32ef1134f07ade1cdf8da69ead6 [id=e248f9c4b9c9406395aaea7ba3feb9ba]). Expected version 1 got 0
I20250903 20:05:27.601831 27064 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 0f5b87c1c4a24f8abdbfd6194f3b2c4a (table default.Table_6e9cf5ead5a24831a6a14d131dd8397e [id=524a27cbe1ad4c10aa2fd4f4074105c6]). Expected version 3 got 0
I20250903 20:05:27.603485 27219 tablet_service.cc:1468] Processing CreateTablet for tablet ea2d1d9e7e724aa08dfec812dad9589e (DEFAULT_TABLE table=default.Table_fa60556c9a9b433a9eff6f68ee824909 [id=d844bf3779574ad4a9868cffbba3d91a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.603925 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 787) took 1938 ms. Trace:
I20250903 20:05:27.604003 27684 rpcz_store.cc:276] 0903 20:05:25.664966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:25.665013 (+ 47us) service_pool.cc:225] Handling call
0903 20:05:27.380622 (+1715609us) ts_tablet_manager.cc:1145] Deleting tablet 8605bc329627438998812ed028327f53
0903 20:05:27.380630 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:27.453422 (+ 72792us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:27.532805 (+ 79383us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:27.603907 (+ 71102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1715379}
I20250903 20:05:27.604099 27352 tablet_service.cc:1468] Processing CreateTablet for tablet 653eade3172344d7aa801edf906ae1d4 (DEFAULT_TABLE table=default.Table_4e5f622e7f16447aa8286aca4aa3c983 [id=aa034b76f6b542bba0191f0f97d8e0ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.604297 27684 tablet_replica.cc:331] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:27.604481 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 653eade3172344d7aa801edf906ae1d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.604465 27684 raft_consensus.cc:2241] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:27.604710 27684 pending_rounds.cc:70] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f: Trying to abort 1 pending ops.
I20250903 20:05:27.604760 27684 pending_rounds.cc:77] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384978073034752 op_type: NO_OP noop_request { }
I20250903 20:05:27.604842 27684 raft_consensus.cc:2887] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:27.604916 27684 raft_consensus.cc:2270] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:27.605472 27684 ts_tablet_manager.cc:1905] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:27.608424 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_4d90c70f5e544f839c056ae573d08f58" } modify_external_catalogs: true
I20250903 20:05:27.608633 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_4d90c70f5e544f839c056ae573d08f58" } modify_external_catalogs: true
I20250903 20:05:27.609083 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 8605bc329627438998812ed028327f53 (table default.Table_8b0346420a89417b9ae2005be49e9f3a [id=b0b2e2d5798346708995a6b8dc028918]) successfully deleted
I20250903 20:05:27.610190 28579 raft_consensus.cc:491] T 59a1dd48d76241d7ba817070fbffd853 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:27.610371 28579 raft_consensus.cc:513] T 59a1dd48d76241d7ba817070fbffd853 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.611773 27695 tablet_bootstrap.cc:654] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:27.612092 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59a1dd48d76241d7ba817070fbffd853" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:27.612517 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_da3c2539a98b461abea7427e2b316bc2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:27.612675 27320 leader_election.cc:343] T 59a1dd48d76241d7ba817070fbffd853 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:27.612938 27692 ts_tablet_manager.cc:1428] T f2cc35f4244a42eeb4a65de749824c11 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
W20250903 20:05:27.613116 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da3c2539a98b461abea7427e2b316bc2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:27.613169 28579 leader_election.cc:290] T 59a1dd48d76241d7ba817070fbffd853 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:27.613445 27692 tablet_bootstrap.cc:492] T 3b3bfc63aa0b4e56bec189e834dc6a8d P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:27.613716 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59a1dd48d76241d7ba817070fbffd853" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:27.614147 27321 leader_election.cc:343] T 59a1dd48d76241d7ba817070fbffd853 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:27.614297 27321 leader_election.cc:304] T 59a1dd48d76241d7ba817070fbffd853 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:27.615553 28579 raft_consensus.cc:2747] T 59a1dd48d76241d7ba817070fbffd853 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:27.616005 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2145fd2e97154b2fbb583eb52f977e70" } modify_external_catalogs: true
I20250903 20:05:27.616276 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2145fd2e97154b2fbb583eb52f977e70" } modify_external_catalogs: true
I20250903 20:05:27.616367 27692 tablet_bootstrap.cc:654] T 3b3bfc63aa0b4e56bec189e834dc6a8d P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:27.617449 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet f2cc35f4244a42eeb4a65de749824c11 with delete_type TABLET_DATA_DELETED (replaced by tablet fc634116a78f45dd833034ddda90e9bf) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:27.618755 27695 tablet_bootstrap.cc:492] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:27.619007 27695 ts_tablet_manager.cc:1397] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20250903 20:05:27.619519 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 653eade3172344d7aa801edf906ae1d4 (DEFAULT_TABLE table=default.Table_4e5f622e7f16447aa8286aca4aa3c983 [id=aa034b76f6b542bba0191f0f97d8e0ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.619630 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 653eade3172344d7aa801edf906ae1d4 (DEFAULT_TABLE table=default.Table_4e5f622e7f16447aa8286aca4aa3c983 [id=aa034b76f6b542bba0191f0f97d8e0ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.619727 28477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296: Bootstrap replayed 1/1 log segments. Stats: ops{read=488 overwritten=0 applied=488 ignored=0} inserts{seen=449 ignored=0} mutations{seen=561 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250903 20:05:27.620054 27695 raft_consensus.cc:357] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.620261 27695 raft_consensus.cc:383] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:27.620332 27695 raft_consensus.cc:738] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:27.620462 28477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296: Bootstrap complete.
I20250903 20:05:27.620584 27695 consensus_queue.cc:260] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.621104 27695 ts_tablet_manager.cc:1428] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:27.622833 27695 tablet_bootstrap.cc:492] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:27.623499 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 5fe1de01bf7a4d418e765ea3e8cbecc1 (DEFAULT_TABLE table=default.Table_2145fd2e97154b2fbb583eb52f977e70 [id=89c8cbb163024233ba4507f136dfe869]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.623914 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fe1de01bf7a4d418e765ea3e8cbecc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.625952 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 5fe1de01bf7a4d418e765ea3e8cbecc1 (DEFAULT_TABLE table=default.Table_2145fd2e97154b2fbb583eb52f977e70 [id=89c8cbb163024233ba4507f136dfe869]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.626899 27249 raft_consensus.cc:2466] T c9e90abef9814338b70a3550ab5686e0 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:27.627033 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 5fe1de01bf7a4d418e765ea3e8cbecc1 (DEFAULT_TABLE table=default.Table_2145fd2e97154b2fbb583eb52f977e70 [id=89c8cbb163024233ba4507f136dfe869]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.627432 27685 ts_tablet_manager.cc:1918] T cb4d99139516405bb65c7480ddf259f1 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:27.627561 27685 log.cc:1199] T cb4d99139516405bb65c7480ddf259f1 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/cb4d99139516405bb65c7480ddf259f1
I20250903 20:05:27.627534 27386 raft_consensus.cc:2466] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:27.627885 27685 ts_tablet_manager.cc:1939] T cb4d99139516405bb65c7480ddf259f1 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:27.628166 27365 tablet_service.cc:1468] Processing CreateTablet for tablet a09771c026a9474c9f61ed259d843cca (DEFAULT_TABLE table=default.Table_191beed070214f6eac98ee16e25e34f8 [id=98b7dc72c66e4cfb92bd24ae61c6605f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.628752 27184 leader_election.cc:304] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:27.629076 28522 raft_consensus.cc:2802] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:27.629369 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a09771c026a9474c9f61ed259d843cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.630748 28522 raft_consensus.cc:695] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:27.631585 28522 consensus_queue.cc:237] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.633111 28596 raft_consensus.cc:2747] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:27.634711 27221 tablet_service.cc:1468] Processing CreateTablet for tablet a09771c026a9474c9f61ed259d843cca (DEFAULT_TABLE table=default.Table_191beed070214f6eac98ee16e25e34f8 [id=98b7dc72c66e4cfb92bd24ae61c6605f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.635905 27059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_480743ddb101436ca551a49c01a368a7" } modify_external_catalogs: true
I20250903 20:05:27.636140 27059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_480743ddb101436ca551a49c01a368a7" } modify_external_catalogs: true
I20250903 20:05:27.637506 27462 leader_election.cc:304] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:27.638432 28596 raft_consensus.cc:2802] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:27.638414 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 697) took 4672 ms. Trace:
I20250903 20:05:27.638554 28596 raft_consensus.cc:695] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:27.638526 27685 rpcz_store.cc:276] 0903 20:05:22.965753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:22.965834 (+ 81us) service_pool.cc:225] Handling call
0903 20:05:27.468479 (+4502645us) ts_tablet_manager.cc:1145] Deleting tablet cb4d99139516405bb65c7480ddf259f1
0903 20:05:27.468503 (+ 24us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:27.534078 (+ 65575us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:27.627288 (+ 93210us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:27.638395 (+ 11107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4502430}
I20250903 20:05:27.638777 28596 consensus_queue.cc:237] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.638924 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 698) took 4550 ms. Trace:
I20250903 20:05:27.638993 27685 rpcz_store.cc:276] 0903 20:05:23.088017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:23.088074 (+ 57us) service_pool.cc:225] Handling call
0903 20:05:27.638805 (+4550731us) ts_tablet_manager.cc:1145] Deleting tablet 09fa32312c9648fbbd6fd327e9738066
0903 20:05:27.638814 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:27.638911 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4550556}
I20250903 20:05:27.639287 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 699) took 4550 ms. Trace:
I20250903 20:05:27.639353 27685 rpcz_store.cc:276] 0903 20:05:23.089129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:23.089177 (+ 48us) service_pool.cc:225] Handling call
0903 20:05:27.639178 (+4550001us) ts_tablet_manager.cc:1145] Deleting tablet 5253d82a41494ce59bb96928179dfdd9
0903 20:05:27.639187 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:27.639271 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4549852}
I20250903 20:05:27.639622 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 700) took 4548 ms. Trace:
I20250903 20:05:27.639679 27685 rpcz_store.cc:276] 0903 20:05:23.090975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:23.091619 (+ 644us) service_pool.cc:225] Handling call
0903 20:05:27.639535 (+4547916us) ts_tablet_manager.cc:1145] Deleting tablet 443d77f2593b44afa3e5aae4c5b0a651
0903 20:05:27.639543 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:27.639610 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4547752}
I20250903 20:05:27.639969 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 702) took 4542 ms. Trace:
I20250903 20:05:27.640033 27685 rpcz_store.cc:276] 0903 20:05:23.097451 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:23.097680 (+ 229us) service_pool.cc:225] Handling call
0903 20:05:27.639879 (+4542199us) ts_tablet_manager.cc:1145] Deleting tablet 9767293447214a7eb68ed03d7373db7f
0903 20:05:27.639886 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:27.639955 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4542024}
I20250903 20:05:27.640338 27685 tablet_replica.cc:331] T cd6d6bb514134bdf8e3bc0aaad180732 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:27.641029 27685 raft_consensus.cc:2241] T cd6d6bb514134bdf8e3bc0aaad180732 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:27.641172 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet cb4d99139516405bb65c7480ddf259f1 (table default.Table_ccdb4130549b4be5888f17097ae35106 [id=cbcfc7067aca4796b39e3eb195f38903]) successfully deleted
W20250903 20:05:27.641440 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 09fa32312c9648fbbd6fd327e9738066 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09fa32312c9648fbbd6fd327e9738066 already in progress: creating tablet
I20250903 20:05:27.641420 27685 raft_consensus.cc:2270] T cd6d6bb514134bdf8e3bc0aaad180732 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:27.641644 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 5253d82a41494ce59bb96928179dfdd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5253d82a41494ce59bb96928179dfdd9 already in progress: creating tablet
W20250903 20:05:27.641832 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 443d77f2593b44afa3e5aae4c5b0a651 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 443d77f2593b44afa3e5aae4c5b0a651 already in progress: creating tablet
W20250903 20:05:27.642046 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 9767293447214a7eb68ed03d7373db7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9767293447214a7eb68ed03d7373db7f already in progress: creating tablet
I20250903 20:05:27.642493 27224 tablet_service.cc:1515] Processing DeleteTablet for tablet 33fb6b357bda43e7abeaed5634183ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:27.642735 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet f0efbb7eefb84ab6bee5b9625b91e298 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:27.645681 27685 ts_tablet_manager.cc:1905] T cd6d6bb514134bdf8e3bc0aaad180732 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:27.648598 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet a72b85dfb34047528b2b28ff0d40f828 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:27.650509 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a223f75045345ae833f52bcc8f239f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:27.653002 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f8346aab1d1f4850b85a74c2d00decca" } modify_external_catalogs: true
I20250903 20:05:27.653249 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f8346aab1d1f4850b85a74c2d00decca" } modify_external_catalogs: true
I20250903 20:05:27.653797 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ecc03aa0a01940e284383daf011ddc7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:27.654716 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_def69a11d277479cbfc58718d39b63ac" } modify_external_catalogs: true
I20250903 20:05:27.654892 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_def69a11d277479cbfc58718d39b63ac" } modify_external_catalogs: true
W20250903 20:05:27.655207 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ecc03aa0a01940e284383daf011ddc7b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:27.657354 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 822) took 1455 ms. Trace:
I20250903 20:05:27.657449 27687 rpcz_store.cc:276] 0903 20:05:26.201542 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.201607 (+ 65us) service_pool.cc:225] Handling call
0903 20:05:27.431727 (+1230120us) ts_tablet_manager.cc:1145] Deleting tablet 1bcb9d5f010f4297bab78c0183889238
0903 20:05:27.431739 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:27.528284 (+ 96545us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:27.531375 (+ 3091us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:27.657333 (+125958us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1229938}
I20250903 20:05:27.657728 27687 tablet_replica.cc:331] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:27.657891 27687 raft_consensus.cc:2241] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:27.658113 27687 pending_rounds.cc:70] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528: Trying to abort 1 pending ops.
I20250903 20:05:27.658171 27687 pending_rounds.cc:77] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384981203611648 op_type: NO_OP noop_request { }
I20250903 20:05:27.658300 27687 raft_consensus.cc:2887] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:27.658390 27687 raft_consensus.cc:2270] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:27.658648 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 733f8024e1fa401ca9614a97db91a19b
I20250903 20:05:27.659252 27687 ts_tablet_manager.cc:1905] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:27.660490 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet 733f8024e1fa401ca9614a97db91a19b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:27.661933 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fe1de01bf7a4d418e765ea3e8cbecc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.662616 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 653eade3172344d7aa801edf906ae1d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.665619 27494 tablet_service.cc:1468] Processing CreateTablet for tablet a09771c026a9474c9f61ed259d843cca (DEFAULT_TABLE table=default.Table_191beed070214f6eac98ee16e25e34f8 [id=98b7dc72c66e4cfb92bd24ae61c6605f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.665974 27494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a09771c026a9474c9f61ed259d843cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.666590 27692 tablet_bootstrap.cc:492] T 3b3bfc63aa0b4e56bec189e834dc6a8d P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:27.666853 27692 ts_tablet_manager.cc:1397] T 3b3bfc63aa0b4e56bec189e834dc6a8d P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.001s
I20250903 20:05:27.667346 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 1bcb9d5f010f4297bab78c0183889238 (table default.Table_05d58d894486468a8a15aaa4b4c0cabb [id=e69321e60a3a4e71a463a0011c0c6d65]) successfully deleted
I20250903 20:05:27.668099 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 733f8024e1fa401ca9614a97db91a19b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:27.668778 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet 733f8024e1fa401ca9614a97db91a19b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:27.669479 27362 tablet_service.cc:1468] Processing CreateTablet for tablet b5bff419ed31453e80ceec23ea05caad (DEFAULT_TABLE table=default.Table_e1654a60dac84f5f8f1d80d5f7bd5b82 [id=b73d27bcf57d4cbd9949db18af6e2e9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.675595 27499 tablet_service.cc:1468] Processing CreateTablet for tablet b5bff419ed31453e80ceec23ea05caad (DEFAULT_TABLE table=default.Table_e1654a60dac84f5f8f1d80d5f7bd5b82 [id=b73d27bcf57d4cbd9949db18af6e2e9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.676090 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5bff419ed31453e80ceec23ea05caad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.679028 28568 consensus_queue.cc:1035] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:27.670027 27692 raft_consensus.cc:357] T 3b3bfc63aa0b4e56bec189e834dc6a8d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.681610 27692 raft_consensus.cc:383] T 3b3bfc63aa0b4e56bec189e834dc6a8d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:27.681736 27218 tablet_service.cc:1468] Processing CreateTablet for tablet b5bff419ed31453e80ceec23ea05caad (DEFAULT_TABLE table=default.Table_e1654a60dac84f5f8f1d80d5f7bd5b82 [id=b73d27bcf57d4cbd9949db18af6e2e9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:27.681828 27692 raft_consensus.cc:738] T 3b3bfc63aa0b4e56bec189e834dc6a8d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:27.682423 28568 consensus_queue.cc:1035] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:27.682461 27692 consensus_queue.cc:260] T 3b3bfc63aa0b4e56bec189e834dc6a8d P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.683996 27695 tablet_bootstrap.cc:654] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:27.687677 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0d67f3bd51344527af6248e556039647" } new_table_name: "default.Table_6120f4e2a1c340b989f03b20673de1a6" modify_external_catalogs: true
I20250903 20:05:27.688262 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_3ea0dde685ef4d89b831c292658d9c51"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:27.688846 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ea0dde685ef4d89b831c292658d9c51 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:27.692795 27692 ts_tablet_manager.cc:1428] T 3b3bfc63aa0b4e56bec189e834dc6a8d P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.026s user 0.000s sys 0.004s
I20250903 20:05:27.693467 27692 tablet_bootstrap.cc:492] T 33fb6b357bda43e7abeaed5634183ef6 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:27.700171 27695 tablet_bootstrap.cc:492] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:27.701782 27695 ts_tablet_manager.cc:1397] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.079s user 0.006s sys 0.000s
I20250903 20:05:27.707710 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3453a2165fe44a3d93deee7b39853d3e" } new_table_name: "default.Table_e104db4b06b3487599962f74a4ca9f98" modify_external_catalogs: true
I20250903 20:05:27.709115 28705 raft_consensus.cc:491] T 3b3bfc63aa0b4e56bec189e834dc6a8d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:27.709302 28705 raft_consensus.cc:513] T 3b3bfc63aa0b4e56bec189e834dc6a8d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.710496 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b3bfc63aa0b4e56bec189e834dc6a8d" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:27.711596 28705 leader_election.cc:290] T 3b3bfc63aa0b4e56bec189e834dc6a8d P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:27.711882 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b3bfc63aa0b4e56bec189e834dc6a8d" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:27.712147 27386 raft_consensus.cc:2391] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 89f8f0ee68b940db9ce250c0604f934a in this term.
I20250903 20:05:27.711424 27695 raft_consensus.cc:357] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.712574 27462 leader_election.cc:304] T 3b3bfc63aa0b4e56bec189e834dc6a8d P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:27.712761 27695 raft_consensus.cc:383] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:27.713304 27695 raft_consensus.cc:738] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:27.709326 28477 raft_consensus.cc:357] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } }
I20250903 20:05:27.714809 28477 raft_consensus.cc:738] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35dd69f3fa7a4fd8843b97f8a5d68296, State: Initialized, Role: FOLLOWER
I20250903 20:05:27.715639 28596 raft_consensus.cc:3058] T 3b3bfc63aa0b4e56bec189e834dc6a8d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:27.716476 27692 tablet_bootstrap.cc:654] T 33fb6b357bda43e7abeaed5634183ef6 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:27.715713 27695 consensus_queue.cc:260] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.719899 28477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 488, Last appended: 3.488, Last appended by leader: 488, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } }
I20250903 20:05:27.721139 27695 ts_tablet_manager.cc:1428] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.017s user 0.003s sys 0.000s
I20250903 20:05:27.724469 27695 tablet_bootstrap.cc:492] T 17e0864a32d5452d84ac470b55fbaaf4 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:27.729308 28596 raft_consensus.cc:2747] T 3b3bfc63aa0b4e56bec189e834dc6a8d P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:27.729923 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5bff419ed31453e80ceec23ea05caad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.732429 28715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:27.732770 28715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 20:05:27.735255 28477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 20:05:27.737938 28567 raft_consensus.cc:491] T 466e501e424e46bc98e8bbc64f97005e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:27.738121 28567 raft_consensus.cc:513] T 466e501e424e46bc98e8bbc64f97005e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.738843 28567 leader_election.cc:290] T 466e501e424e46bc98e8bbc64f97005e P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:27.740093 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "466e501e424e46bc98e8bbc64f97005e" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:27.740710 27320 leader_election.cc:343] T 466e501e424e46bc98e8bbc64f97005e P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:27.740948 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "466e501e424e46bc98e8bbc64f97005e" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:27.741462 27321 leader_election.cc:343] T 466e501e424e46bc98e8bbc64f97005e P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:27.741568 27321 leader_election.cc:304] T 466e501e424e46bc98e8bbc64f97005e P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:27.742044 28567 raft_consensus.cc:2747] T 466e501e424e46bc98e8bbc64f97005e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:27.761587 27052 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 989545be137341ef8d742bb1119cc51f (table default.Table_412e2b595d4e4e83b26634356476d069 [id=1caafc550c7d4f478cb019f7e01b2aee], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 989545be137341ef8d742bb1119cc51f consensus state has no leader
I20250903 20:05:27.766747 27692 tablet_bootstrap.cc:492] T 33fb6b357bda43e7abeaed5634183ef6 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:27.767102 27692 ts_tablet_manager.cc:1397] T 33fb6b357bda43e7abeaed5634183ef6 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.074s user 0.002s sys 0.004s
I20250903 20:05:27.772343 27695 tablet_bootstrap.cc:654] T 17e0864a32d5452d84ac470b55fbaaf4 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:27.772466 27692 raft_consensus.cc:357] T 33fb6b357bda43e7abeaed5634183ef6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.774490 27692 raft_consensus.cc:383] T 33fb6b357bda43e7abeaed5634183ef6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:27.775403 27692 raft_consensus.cc:738] T 33fb6b357bda43e7abeaed5634183ef6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:27.777959 27692 consensus_queue.cc:260] T 33fb6b357bda43e7abeaed5634183ef6 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.783326 27692 ts_tablet_manager.cc:1428] T 33fb6b357bda43e7abeaed5634183ef6 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.016s user 0.002s sys 0.002s
I20250903 20:05:27.786347 27692 tablet_bootstrap.cc:492] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:27.802964 27695 tablet_bootstrap.cc:492] T 17e0864a32d5452d84ac470b55fbaaf4 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:27.803490 27695 ts_tablet_manager.cc:1397] T 17e0864a32d5452d84ac470b55fbaaf4 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.080s user 0.006s sys 0.002s
I20250903 20:05:27.809023 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcdddd231de24ec2944a2ffac7b8fb58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.812901 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8808c2668707437f9db006c7f9a91e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.813196 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fe1de01bf7a4d418e765ea3e8cbecc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.813620 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 653eade3172344d7aa801edf906ae1d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.813562 27695 raft_consensus.cc:357] T 17e0864a32d5452d84ac470b55fbaaf4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.813817 27695 raft_consensus.cc:383] T 17e0864a32d5452d84ac470b55fbaaf4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:27.813900 27695 raft_consensus.cc:738] T 17e0864a32d5452d84ac470b55fbaaf4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:27.814249 27219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea2d1d9e7e724aa08dfec812dad9589e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.814721 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a09771c026a9474c9f61ed259d843cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.815174 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7c8657489734199b346033bd4afb02e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.825098 27692 tablet_bootstrap.cc:654] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:27.814179 27695 consensus_queue.cc:260] T 17e0864a32d5452d84ac470b55fbaaf4 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.826716 27695 ts_tablet_manager.cc:1428] T 17e0864a32d5452d84ac470b55fbaaf4 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.023s user 0.003s sys 0.000s
I20250903 20:05:27.827095 27695 tablet_bootstrap.cc:492] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:27.834162 27687 ts_tablet_manager.cc:1918] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:27.834640 27687 log.cc:1199] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/a4c582b441f6467c99ba0b090abcf4cc
I20250903 20:05:27.835062 27687 ts_tablet_manager.cc:1939] T a4c582b441f6467c99ba0b090abcf4cc P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:27.837108 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 826) took 1628 ms. Trace:
I20250903 20:05:27.837246 27687 rpcz_store.cc:276] 0903 20:05:26.208512 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.208561 (+ 49us) service_pool.cc:225] Handling call
0903 20:05:27.657682 (+1449121us) ts_tablet_manager.cc:1145] Deleting tablet a4c582b441f6467c99ba0b090abcf4cc
0903 20:05:27.657692 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:27.729256 (+ 71564us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:27.833944 (+104688us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:27.837065 (+ 3121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1448973}
I20250903 20:05:27.838567 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 829) took 1625 ms. Trace:
I20250903 20:05:27.838713 27687 rpcz_store.cc:276] 0903 20:05:26.213492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.213557 (+ 65us) service_pool.cc:225] Handling call
0903 20:05:27.838425 (+1624868us) ts_tablet_manager.cc:1145] Deleting tablet 121a8ba102454cb4807483fac16e3781
0903 20:05:27.838438 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:27.838549 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1624682}
W20250903 20:05:27.839228 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 121a8ba102454cb4807483fac16e3781 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 121a8ba102454cb4807483fac16e3781 already in progress: creating tablet
I20250903 20:05:27.839672 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 835) took 1439 ms. Trace:
I20250903 20:05:27.839816 27687 rpcz_store.cc:276] 0903 20:05:26.400262 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.400337 (+ 75us) service_pool.cc:225] Handling call
0903 20:05:27.839565 (+1439228us) ts_tablet_manager.cc:1145] Deleting tablet 55b2cdee20264ae6b64ebe3607fe1ca6
0903 20:05:27.839574 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:27.839657 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1438998}
W20250903 20:05:27.840301 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 55b2cdee20264ae6b64ebe3607fe1ca6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55b2cdee20264ae6b64ebe3607fe1ca6 already in progress: creating tablet
I20250903 20:05:27.840611 27687 tablet_replica.cc:331] T 17e0864a32d5452d84ac470b55fbaaf4 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:27.840842 27687 raft_consensus.cc:2241] T 17e0864a32d5452d84ac470b55fbaaf4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:27.841071 27687 raft_consensus.cc:2270] T 17e0864a32d5452d84ac470b55fbaaf4 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:27.841884 27687 ts_tablet_manager.cc:1905] T 17e0864a32d5452d84ac470b55fbaaf4 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:27.843410 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5bff419ed31453e80ceec23ea05caad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:27.855935 27684 ts_tablet_manager.cc:1918] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:27.856084 27684 log.cc:1199] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/c6d67cd0c0304c77ba571b414878dad2
I20250903 20:05:27.856479 27684 ts_tablet_manager.cc:1939] T c6d67cd0c0304c77ba571b414878dad2 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:27.861786 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_d40b242552784e45a5de2341a25174fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:27.862735 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d40b242552784e45a5de2341a25174fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:27.868940 28522 consensus_queue.cc:1035] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:27.872181 27695 tablet_bootstrap.cc:654] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:27.878679 28567 raft_consensus.cc:491] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:27.878844 28567 raft_consensus.cc:513] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.879489 28567 leader_election.cc:290] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:27.879630 28567 raft_consensus.cc:491] T a72b85dfb34047528b2b28ff0d40f828 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:27.879715 28567 raft_consensus.cc:513] T a72b85dfb34047528b2b28ff0d40f828 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.880175 28567 leader_election.cc:290] T a72b85dfb34047528b2b28ff0d40f828 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:27.881496 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a72b85dfb34047528b2b28ff0d40f828" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:27.881726 27184 consensus_peers.cc:489] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0293a69f1c246d08d414e95729b4ab4. This is attempt 1: this message will repeat every 5th retry.
W20250903 20:05:27.881933 27321 leader_election.cc:343] T a72b85dfb34047528b2b28ff0d40f828 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:27.883251 27692 tablet_bootstrap.cc:492] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:27.883440 27692 ts_tablet_manager.cc:1397] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.098s user 0.003s sys 0.002s
I20250903 20:05:27.883788 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a72b85dfb34047528b2b28ff0d40f828" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:27.884222 27692 raft_consensus.cc:357] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.884379 27692 raft_consensus.cc:383] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:27.884423 27692 raft_consensus.cc:738] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
W20250903 20:05:27.884440 27320 leader_election.cc:343] T a72b85dfb34047528b2b28ff0d40f828 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:27.884533 27320 leader_election.cc:304] T a72b85dfb34047528b2b28ff0d40f828 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:27.884593 27692 consensus_queue.cc:260] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.885108 28579 raft_consensus.cc:2747] T a72b85dfb34047528b2b28ff0d40f828 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:27.885497 27692 ts_tablet_manager.cc:1428] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:27.885764 27692 tablet_bootstrap.cc:492] T 2d9c1642fec841d497ff3496dd4564db P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:27.885891 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bad3c509460422396a17a3c4ae3d1e8" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:27.887248 27320 leader_election.cc:343] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:27.887629 27692 tablet_bootstrap.cc:654] T 2d9c1642fec841d497ff3496dd4564db P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:27.888108 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bad3c509460422396a17a3c4ae3d1e8" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:27.888375 27240 raft_consensus.cc:2466] T 1bad3c509460422396a17a3c4ae3d1e8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:27.888830 27321 leader_election.cc:304] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:27.889282 28579 raft_consensus.cc:2802] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:27.889350 28579 raft_consensus.cc:491] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:27.889400 27695 tablet_bootstrap.cc:492] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:27.889417 28579 raft_consensus.cc:3058] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:27.889515 27695 ts_tablet_manager.cc:1397] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.063s user 0.004s sys 0.000s
I20250903 20:05:27.890786 27695 raft_consensus.cc:357] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.890987 27695 raft_consensus.cc:383] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:27.891047 27695 raft_consensus.cc:738] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:27.891348 27695 consensus_queue.cc:260] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.891920 27695 ts_tablet_manager.cc:1428] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:27.893904 28567 consensus_queue.cc:1035] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:27.902951 27063 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 9657ef024273400398d5e76f15359c40
I20250903 20:05:27.903193 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_dcc32881015d4f1596b05098a9f86782"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:27.903416 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_9abad42b86c747b68c5d9ef37d8d2788" } new_table_name: "default.Table_1d234b8a96d94c638e1bd46cd352da16" modify_external_catalogs: true
W20250903 20:05:27.903877 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dcc32881015d4f1596b05098a9f86782 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:27.905840 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 475d05d087c34159b093b92cc308bcaf
I20250903 20:05:27.906637 27692 tablet_bootstrap.cc:492] T 2d9c1642fec841d497ff3496dd4564db P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:27.906775 27692 ts_tablet_manager.cc:1397] T 2d9c1642fec841d497ff3496dd4564db P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.003s
I20250903 20:05:27.906812 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_534372788132478792ce2f6e8cc1ae3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:27.907176 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 475d05d087c34159b093b92cc308bcaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:27.907492 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_534372788132478792ce2f6e8cc1ae3c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:27.907662 27692 raft_consensus.cc:357] T 2d9c1642fec841d497ff3496dd4564db P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.907825 27692 raft_consensus.cc:383] T 2d9c1642fec841d497ff3496dd4564db P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:27.907869 27692 raft_consensus.cc:738] T 2d9c1642fec841d497ff3496dd4564db P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:27.909080 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 475d05d087c34159b093b92cc308bcaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:27.909096 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 475d05d087c34159b093b92cc308bcaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:27.909246 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 5fe1de01bf7a4d418e765ea3e8cbecc1
I20250903 20:05:27.909946 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 5fe1de01bf7a4d418e765ea3e8cbecc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:27.910081 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2623cd1b586e40818f657e9fc9f5f570" } new_table_name: "default.Table_f8bbfa6b86694cdab5ec1347c36a69f3" modify_external_catalogs: true
I20250903 20:05:27.910539 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 5fe1de01bf7a4d418e765ea3e8cbecc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:27.909343 27692 consensus_queue.cc:260] T 2d9c1642fec841d497ff3496dd4564db P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.910773 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 5fe1de01bf7a4d418e765ea3e8cbecc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:27.911397 27692 ts_tablet_manager.cc:1428] T 2d9c1642fec841d497ff3496dd4564db P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250903 20:05:27.911527 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 3e930dbaac0e40ee8f175ccf87592d31
I20250903 20:05:27.911723 27692 tablet_bootstrap.cc:492] T 6f8cfe53b91b41d7ba7f9074e1005af3 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:27.912204 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 3e930dbaac0e40ee8f175ccf87592d31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:27.912307 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 3e930dbaac0e40ee8f175ccf87592d31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:27.912945 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 3e930dbaac0e40ee8f175ccf87592d31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:27.915721 27692 tablet_bootstrap.cc:654] T 6f8cfe53b91b41d7ba7f9074e1005af3 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:27.917709 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6f2ad684529148418295c9b463e0a0f7" } new_table_name: "default.Table_16cf18fd6f71409f936e2df972649ad6" modify_external_catalogs: true
W20250903 20:05:27.927269 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4fa886f5be0f489290921e61dcaa4916 consensus state has no leader
I20250903 20:05:27.927978 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_3be712dee7574cd4bdc030111a6d1f1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:27.928802 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3be712dee7574cd4bdc030111a6d1f1a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:27.929564 27184 consensus_peers.cc:489] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1bcb9d5f010f4297bab78c0183889238. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:27.930992 27692 tablet_bootstrap.cc:492] T 6f8cfe53b91b41d7ba7f9074e1005af3 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:27.931147 27692 ts_tablet_manager.cc:1397] T 6f8cfe53b91b41d7ba7f9074e1005af3 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.001s
I20250903 20:05:27.932221 27692 raft_consensus.cc:357] T 6f8cfe53b91b41d7ba7f9074e1005af3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.932471 27692 raft_consensus.cc:383] T 6f8cfe53b91b41d7ba7f9074e1005af3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:27.932523 27692 raft_consensus.cc:738] T 6f8cfe53b91b41d7ba7f9074e1005af3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:27.932765 27692 consensus_queue.cc:260] T 6f8cfe53b91b41d7ba7f9074e1005af3 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.932998 28710 raft_consensus.cc:491] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:27.933096 28710 raft_consensus.cc:513] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.933787 28710 leader_election.cc:290] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:27.934726 27692 ts_tablet_manager.cc:1428] T 6f8cfe53b91b41d7ba7f9074e1005af3 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20250903 20:05:27.935156 27692 tablet_bootstrap.cc:492] T f0efbb7eefb84ab6bee5b9625b91e298 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:27.935714 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "961ade36f7d14eba91bb16d5eae2e254" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:27.936529 28579 raft_consensus.cc:513] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:27.937120 28579 leader_election.cc:290] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:27.938354 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bad3c509460422396a17a3c4ae3d1e8" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:27.938915 27685 ts_tablet_manager.cc:1918] T cd6d6bb514134bdf8e3bc0aaad180732 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250903 20:05:27.938938 27320 leader_election.cc:343] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:27.939065 27685 log.cc:1199] T cd6d6bb514134bdf8e3bc0aaad180732 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/cd6d6bb514134bdf8e3bc0aaad180732
I20250903 20:05:27.939355 27685 ts_tablet_manager.cc:1939] T cd6d6bb514134bdf8e3bc0aaad180732 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:27.941493 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 788) took 2273 ms. Trace:
I20250903 20:05:27.941624 27684 rpcz_store.cc:276] 0903 20:05:25.667850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:25.667903 (+ 53us) service_pool.cc:225] Handling call
0903 20:05:27.604259 (+1936356us) ts_tablet_manager.cc:1145] Deleting tablet c6d67cd0c0304c77ba571b414878dad2
0903 20:05:27.604267 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:27.662288 (+ 58021us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:27.855811 (+193523us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:27.941461 (+ 85650us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1936187}
I20250903 20:05:27.942169 27684 tablet_replica.cc:331] T 45d9b6ff520348628d19671127b9426c P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:27.942446 27684 raft_consensus.cc:2241] T 45d9b6ff520348628d19671127b9426c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:27.942667 27684 raft_consensus.cc:2270] T 45d9b6ff520348628d19671127b9426c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:27.943403 27684 ts_tablet_manager.cc:1905] T 45d9b6ff520348628d19671127b9426c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:27.944070 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bad3c509460422396a17a3c4ae3d1e8" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:27.944381 27240 raft_consensus.cc:3058] T 1bad3c509460422396a17a3c4ae3d1e8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:27.946352 27692 tablet_bootstrap.cc:654] T f0efbb7eefb84ab6bee5b9625b91e298 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:27.946839 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet c6d67cd0c0304c77ba571b414878dad2 (table default.Table_a540eea45dfb46ec92b02a3f9a2b3aed [id=5fe055478b724744a65fcaa2f25bef0d]) successfully deleted
I20250903 20:05:27.947329 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "961ade36f7d14eba91bb16d5eae2e254" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:27.947821 28579 raft_consensus.cc:491] T e0347eab6c8642beabdc2d7e7bbfdc9b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:27.947973 28579 raft_consensus.cc:513] T e0347eab6c8642beabdc2d7e7bbfdc9b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:27.948657 28579 leader_election.cc:290] T e0347eab6c8642beabdc2d7e7bbfdc9b P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:27.949398 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e0347eab6c8642beabdc2d7e7bbfdc9b" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:27.949726 28522 consensus_queue.cc:1035] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250903 20:05:27.949923 27321 leader_election.cc:343] T e0347eab6c8642beabdc2d7e7bbfdc9b P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:27.952078 27357 consensus_meta.cc:287] T 27c21921025a464681bf157b2627d351 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent flushing consensus metadata: real 0.987s user 0.017s sys 0.007s
I20250903 20:05:27.952417 27357 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:54902 (request call id 843) took 1136 ms. Trace:
I20250903 20:05:27.952489 27357 rpcz_store.cc:276] 0903 20:05:26.816165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.816234 (+ 69us) service_pool.cc:225] Handling call
0903 20:05:26.819998 (+ 3764us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0903 20:05:26.820033 (+ 35us) ts_tablet_manager.cc:676] Creating new metadata...
0903 20:05:26.964760 (+144727us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:27.952248 (+987488us) tablet_replica.cc:206] Creating consensus instance
0903 20:05:27.952403 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250903 20:05:27.955437 27462 leader_election.cc:304] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:27.955786 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e0347eab6c8642beabdc2d7e7bbfdc9b" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:27.957500 27320 leader_election.cc:343] T e0347eab6c8642beabdc2d7e7bbfdc9b P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:27.957638 27320 leader_election.cc:304] T e0347eab6c8642beabdc2d7e7bbfdc9b P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:27.958477 28579 raft_consensus.cc:2747] T e0347eab6c8642beabdc2d7e7bbfdc9b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:27.958930 27296 heartbeater.cc:344] Connected to a master server at 127.26.65.125:44059
I20250903 20:05:27.964084 28598 raft_consensus.cc:2747] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:27.991991 27066 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 3b3bfc63aa0b4e56bec189e834dc6a8d (table default.Table_9b20469406c94986ab496ea3e678b586 [id=6d6f5e8cadf9445d8695c075cfabb2ea]). Expected version 1 got 0
I20250903 20:05:27.995777 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_973e08d4f7e34fde936b93bb0b3981b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:27.997184 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 33fb6b357bda43e7abeaed5634183ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:28.004043 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_973e08d4f7e34fde936b93bb0b3981b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:28.008474 27692 tablet_bootstrap.cc:492] T f0efbb7eefb84ab6bee5b9625b91e298 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:28.008695 27692 ts_tablet_manager.cc:1397] T f0efbb7eefb84ab6bee5b9625b91e298 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.074s user 0.000s sys 0.004s
I20250903 20:05:28.009557 27692 raft_consensus.cc:357] T f0efbb7eefb84ab6bee5b9625b91e298 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.009773 27692 raft_consensus.cc:383] T f0efbb7eefb84ab6bee5b9625b91e298 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:28.009840 27692 raft_consensus.cc:738] T f0efbb7eefb84ab6bee5b9625b91e298 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:28.010098 27692 consensus_queue.cc:260] T f0efbb7eefb84ab6bee5b9625b91e298 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.012776 27692 ts_tablet_manager.cc:1428] T f0efbb7eefb84ab6bee5b9625b91e298 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:28.013110 27692 tablet_bootstrap.cc:492] T 1bad3c509460422396a17a3c4ae3d1e8 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:28.014760 27692 tablet_bootstrap.cc:654] T 1bad3c509460422396a17a3c4ae3d1e8 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:28.022055 28655 raft_consensus.cc:491] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:28.022251 28655 raft_consensus.cc:513] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.022886 28655 leader_election.cc:290] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:28.023536 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e2f98f760cf4df4bc2a10530e2157c6" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:28.023727 27386 raft_consensus.cc:2466] T 7e2f98f760cf4df4bc2a10530e2157c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:28.026571 27184 leader_election.cc:304] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:28.027073 28686 raft_consensus.cc:2802] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:28.027143 28686 raft_consensus.cc:491] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:28.027195 28686 raft_consensus.cc:3058] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.027926 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e2f98f760cf4df4bc2a10530e2157c6" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:28.028673 27183 leader_election.cc:343] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:28.030781 27692 tablet_bootstrap.cc:492] T 1bad3c509460422396a17a3c4ae3d1e8 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:28.030982 27692 ts_tablet_manager.cc:1397] T 1bad3c509460422396a17a3c4ae3d1e8 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.001s
I20250903 20:05:28.032462 27692 raft_consensus.cc:357] T 1bad3c509460422396a17a3c4ae3d1e8 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.032688 27692 raft_consensus.cc:383] T 1bad3c509460422396a17a3c4ae3d1e8 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:28.032752 27692 raft_consensus.cc:738] T 1bad3c509460422396a17a3c4ae3d1e8 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:28.033001 27692 consensus_queue.cc:260] T 1bad3c509460422396a17a3c4ae3d1e8 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.033869 27692 ts_tablet_manager.cc:1428] T 1bad3c509460422396a17a3c4ae3d1e8 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250903 20:05:28.034293 27692 tablet_bootstrap.cc:492] T 59a1dd48d76241d7ba817070fbffd853 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:28.035696 28568 consensus_queue.cc:1035] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:28.039968 27687 ts_tablet_manager.cc:1918] T 17e0864a32d5452d84ac470b55fbaaf4 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:28.040082 27687 log.cc:1199] T 17e0864a32d5452d84ac470b55fbaaf4 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/17e0864a32d5452d84ac470b55fbaaf4
I20250903 20:05:28.040470 27687 ts_tablet_manager.cc:1939] T 17e0864a32d5452d84ac470b55fbaaf4 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:28.044600 28568 consensus_queue.cc:1035] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:28.045224 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 836) took 1579 ms. Trace:
I20250903 20:05:28.045311 27687 rpcz_store.cc:276] 0903 20:05:26.466048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.467469 (+ 1421us) service_pool.cc:225] Handling call
0903 20:05:27.840566 (+1373097us) ts_tablet_manager.cc:1145] Deleting tablet 17e0864a32d5452d84ac470b55fbaaf4
0903 20:05:27.840575 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:27.935680 (+ 95105us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:28.039833 (+104153us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:28.045207 (+ 5374us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1372890}
I20250903 20:05:28.045838 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 17e0864a32d5452d84ac470b55fbaaf4 (table default.Table_c21ce98311d847f2aaae2ea2c0549856 [id=263d1af690fb4868a4bd92ae090dfd0a]) successfully deleted
I20250903 20:05:28.047415 28522 raft_consensus.cc:491] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:28.047590 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 837) took 1579 ms. Trace:
I20250903 20:05:28.047689 27687 rpcz_store.cc:276] 0903 20:05:26.468254 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.468334 (+ 80us) service_pool.cc:225] Handling call
0903 20:05:28.045591 (+1577257us) ts_tablet_manager.cc:1145] Deleting tablet ab64604c56914379943c03129ccc2e7c
0903 20:05:28.047465 (+ 1874us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:28.047570 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1577045}
I20250903 20:05:28.047866 28655 consensus_queue.cc:1035] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:28.047755 28522 raft_consensus.cc:513] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.048043 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 838) took 1579 ms. Trace:
I20250903 20:05:28.048134 27687 rpcz_store.cc:276] 0903 20:05:26.468594 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.468828 (+ 234us) service_pool.cc:225] Handling call
0903 20:05:28.047947 (+1579119us) ts_tablet_manager.cc:1145] Deleting tablet ef02f91c0964446daf62ac1dde0330ff
0903 20:05:28.047956 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:28.048029 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1578970}
I20250903 20:05:28.048375 27687 tablet_replica.cc:331] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:28.048554 27687 raft_consensus.cc:2241] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
W20250903 20:05:28.048754 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet ab64604c56914379943c03129ccc2e7c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab64604c56914379943c03129ccc2e7c already in progress: creating tablet
I20250903 20:05:28.048862 27687 raft_consensus.cc:2270] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:28.049031 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet ef02f91c0964446daf62ac1dde0330ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef02f91c0964446daf62ac1dde0330ff already in progress: creating tablet
I20250903 20:05:28.049089 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53881ad30bc843839008ace4012a3f70" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:28.049328 27386 raft_consensus.cc:2466] T 53881ad30bc843839008ace4012a3f70 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:28.049854 27184 leader_election.cc:304] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:28.050176 28655 raft_consensus.cc:2802] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:28.050294 28655 raft_consensus.cc:491] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:28.050361 28655 raft_consensus.cc:3058] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.050933 28522 leader_election.cc:290] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:28.051384 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53881ad30bc843839008ace4012a3f70" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:28.052047 27183 leader_election.cc:343] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:28.055878 27692 tablet_bootstrap.cc:654] T 59a1dd48d76241d7ba817070fbffd853 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:28.061071 27433 heartbeater.cc:344] Connected to a master server at 127.26.65.125:44059
I20250903 20:05:28.061570 28522 raft_consensus.cc:491] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:28.061919 28522 raft_consensus.cc:513] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.063261 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a4f3230f50ec4c95b528fead8c191224" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:28.063521 27386 raft_consensus.cc:2466] T a4f3230f50ec4c95b528fead8c191224 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:28.064047 27184 leader_election.cc:304] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:28.064770 28744 raft_consensus.cc:2802] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:28.064956 28744 raft_consensus.cc:491] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:28.065106 28744 raft_consensus.cc:3058] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.065531 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a4f3230f50ec4c95b528fead8c191224" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:28.066200 27183 leader_election.cc:343] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:28.067763 28522 leader_election.cc:290] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:28.069643 27692 tablet_bootstrap.cc:492] T 59a1dd48d76241d7ba817070fbffd853 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:28.069787 27692 ts_tablet_manager.cc:1397] T 59a1dd48d76241d7ba817070fbffd853 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.002s
I20250903 20:05:28.071017 27692 raft_consensus.cc:357] T 59a1dd48d76241d7ba817070fbffd853 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.071240 27692 raft_consensus.cc:383] T 59a1dd48d76241d7ba817070fbffd853 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:28.071312 27692 raft_consensus.cc:738] T 59a1dd48d76241d7ba817070fbffd853 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:28.071622 27692 consensus_queue.cc:260] T 59a1dd48d76241d7ba817070fbffd853 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.072888 27692 ts_tablet_manager.cc:1428] T 59a1dd48d76241d7ba817070fbffd853 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:28.073251 27692 tablet_bootstrap.cc:492] T 53881ad30bc843839008ace4012a3f70 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:28.074024 28522 consensus_queue.cc:1035] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:28.075915 27692 tablet_bootstrap.cc:654] T 53881ad30bc843839008ace4012a3f70 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:28.081655 27692 tablet_bootstrap.cc:492] T 53881ad30bc843839008ace4012a3f70 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:28.081861 27692 ts_tablet_manager.cc:1397] T 53881ad30bc843839008ace4012a3f70 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20250903 20:05:28.083181 27692 raft_consensus.cc:357] T 53881ad30bc843839008ace4012a3f70 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.083362 27692 raft_consensus.cc:383] T 53881ad30bc843839008ace4012a3f70 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:28.083420 27692 raft_consensus.cc:738] T 53881ad30bc843839008ace4012a3f70 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:28.083631 27687 ts_tablet_manager.cc:1905] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:28.083647 27692 consensus_queue.cc:260] T 53881ad30bc843839008ace4012a3f70 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.084367 27692 ts_tablet_manager.cc:1428] T 53881ad30bc843839008ace4012a3f70 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250903 20:05:28.097012 28579 raft_consensus.cc:491] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:28.097170 28579 raft_consensus.cc:513] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.097831 28579 leader_election.cc:290] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:28.099710 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1fbe54dc40dd47daa109ceeb07546b0a" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:28.100220 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1fbe54dc40dd47daa109ceeb07546b0a" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:28.100437 27239 raft_consensus.cc:2466] T 1fbe54dc40dd47daa109ceeb07546b0a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
W20250903 20:05:28.101140 27320 leader_election.cc:343] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:28.101672 27321 leader_election.cc:304] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:28.102294 28579 raft_consensus.cc:2802] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:28.102399 28579 raft_consensus.cc:491] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:28.102480 28579 raft_consensus.cc:3058] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.116808 28522 raft_consensus.cc:491] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:28.116971 28522 raft_consensus.cc:513] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.117574 28522 leader_election.cc:290] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:28.118796 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59a1dd48d76241d7ba817070fbffd853" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:28.118997 27524 raft_consensus.cc:2466] T 59a1dd48d76241d7ba817070fbffd853 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:28.120440 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59a1dd48d76241d7ba817070fbffd853" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:28.120656 27381 raft_consensus.cc:2466] T 59a1dd48d76241d7ba817070fbffd853 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:28.120651 27183 leader_election.cc:304] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:28.121521 28522 raft_consensus.cc:2802] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:28.121608 28522 raft_consensus.cc:491] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:28.121680 28522 raft_consensus.cc:3058] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.125898 27695 tablet_bootstrap.cc:492] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:28.127769 27524 raft_consensus.cc:3058] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.128744 27695 tablet_bootstrap.cc:654] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:28.132745 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 701) took 5036 ms. Trace:
I20250903 20:05:28.132874 27685 rpcz_store.cc:276] 0903 20:05:23.095831 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:23.097896 (+ 2065us) service_pool.cc:225] Handling call
0903 20:05:27.640285 (+4542389us) ts_tablet_manager.cc:1145] Deleting tablet cd6d6bb514134bdf8e3bc0aaad180732
0903 20:05:27.640296 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:27.866344 (+226048us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:27.938766 (+ 72422us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:28.132725 (+193959us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4542245}
I20250903 20:05:28.133481 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 703) took 4943 ms. Trace:
I20250903 20:05:28.133574 27685 rpcz_store.cc:276] 0903 20:05:23.189686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:23.189744 (+ 58us) service_pool.cc:225] Handling call
0903 20:05:28.133272 (+4943528us) ts_tablet_manager.cc:1145] Deleting tablet cb2f50683e904dd39ee6eb128717d38c
0903 20:05:28.133283 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:28.133464 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4881305}
I20250903 20:05:28.133888 27685 tablet_replica.cc:331] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:28.134675 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet cd6d6bb514134bdf8e3bc0aaad180732 (table default.Table_ef30e0f336384b50bf8293fe525252cf [id=b2f5b8646204428993ce29cf7b2e4caa]) successfully deleted
I20250903 20:05:28.134147 27685 raft_consensus.cc:2241] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
W20250903 20:05:28.134974 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet cb2f50683e904dd39ee6eb128717d38c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb2f50683e904dd39ee6eb128717d38c already in progress: creating tablet
I20250903 20:05:28.135038 27685 raft_consensus.cc:2270] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:28.135772 27685 ts_tablet_manager.cc:1905] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:28.144932 27695 tablet_bootstrap.cc:492] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:28.145105 27695 ts_tablet_manager.cc:1397] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20250903 20:05:28.145882 27695 raft_consensus.cc:357] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.146086 27695 raft_consensus.cc:383] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:28.146140 27695 raft_consensus.cc:738] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:28.146364 27695 consensus_queue.cc:260] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.147162 27695 ts_tablet_manager.cc:1428] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:28.147454 27695 tablet_bootstrap.cc:492] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:28.149812 27695 tablet_bootstrap.cc:654] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:28.154645 27692 tablet_bootstrap.cc:492] T 1fbe54dc40dd47daa109ceeb07546b0a P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:28.156934 27692 tablet_bootstrap.cc:654] T 1fbe54dc40dd47daa109ceeb07546b0a P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:28.159731 28751 consensus_queue.cc:1035] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:28.162642 27684 ts_tablet_manager.cc:1918] T 45d9b6ff520348628d19671127b9426c P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:28.162775 27684 log.cc:1199] T 45d9b6ff520348628d19671127b9426c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/45d9b6ff520348628d19671127b9426c
I20250903 20:05:28.163167 27684 ts_tablet_manager.cc:1939] T 45d9b6ff520348628d19671127b9426c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:28.163959 28744 raft_consensus.cc:513] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.164650 28744 leader_election.cc:290] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:28.165226 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a4f3230f50ec4c95b528fead8c191224" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
W20250903 20:05:28.165829 27183 leader_election.cc:343] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:28.166427 28579 raft_consensus.cc:513] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.167124 28579 leader_election.cc:290] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:28.167960 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1fbe54dc40dd47daa109ceeb07546b0a" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:28.168005 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a4f3230f50ec4c95b528fead8c191224" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:28.168165 27381 raft_consensus.cc:3058] T a4f3230f50ec4c95b528fead8c191224 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.168165 27248 raft_consensus.cc:3058] T 1fbe54dc40dd47daa109ceeb07546b0a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.168540 28686 raft_consensus.cc:513] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.169718 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1fbe54dc40dd47daa109ceeb07546b0a" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
W20250903 20:05:28.170395 27320 leader_election.cc:343] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:28.171142 28598 raft_consensus.cc:491] T 33fb6b357bda43e7abeaed5634183ef6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:28.171329 28598 raft_consensus.cc:513] T 33fb6b357bda43e7abeaed5634183ef6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.172070 28598 leader_election.cc:290] T 33fb6b357bda43e7abeaed5634183ef6 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:28.172719 28744 raft_consensus.cc:491] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:28.172891 28744 raft_consensus.cc:513] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.173086 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e2f98f760cf4df4bc2a10530e2157c6" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:28.173280 27380 raft_consensus.cc:3058] T 7e2f98f760cf4df4bc2a10530e2157c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.173336 27382 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "33fb6b357bda43e7abeaed5634183ef6" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:28.173529 28744 leader_election.cc:290] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:28.173655 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "33fb6b357bda43e7abeaed5634183ef6" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:28.173854 27239 raft_consensus.cc:2391] T 33fb6b357bda43e7abeaed5634183ef6 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
I20250903 20:05:28.174361 27695 tablet_bootstrap.cc:492] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:28.174490 27695 ts_tablet_manager.cc:1397] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.003s
I20250903 20:05:28.174491 27382 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d9c1642fec841d497ff3496dd4564db" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:28.174685 27382 raft_consensus.cc:2466] T 2d9c1642fec841d497ff3496dd4564db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:28.174959 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 789) took 2506 ms. Trace:
I20250903 20:05:28.175086 27184 leader_election.cc:304] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:28.175047 27684 rpcz_store.cc:276] 0903 20:05:25.668425 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:25.668470 (+ 45us) service_pool.cc:225] Handling call
0903 20:05:27.942100 (+2273630us) ts_tablet_manager.cc:1145] Deleting tablet 45d9b6ff520348628d19671127b9426c
0903 20:05:27.942113 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:28.040846 (+ 98733us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:28.162502 (+121656us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:28.174936 (+ 12434us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2273426}
I20250903 20:05:28.175233 27695 raft_consensus.cc:357] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.175376 27695 raft_consensus.cc:383] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:28.175395 27684 tablet_replica.cc:331] T ac9b5d4f39db4f05a15316585040d142 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:28.175428 27695 raft_consensus.cc:738] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:28.175593 27684 raft_consensus.cc:2241] T ac9b5d4f39db4f05a15316585040d142 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:28.175570 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 45d9b6ff520348628d19671127b9426c (table default.Table_509afdab983d4eed97a5d124ff19ab3b [id=7a023e4098354d038d92f90d46fc5125]) successfully deleted
I20250903 20:05:28.175732 27684 raft_consensus.cc:2270] T ac9b5d4f39db4f05a15316585040d142 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:28.175674 27695 consensus_queue.cc:260] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.175911 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d9c1642fec841d497ff3496dd4564db" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:28.175946 28746 raft_consensus.cc:2802] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:28.176038 28746 raft_consensus.cc:491] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:28.176069 27518 raft_consensus.cc:2466] T 2d9c1642fec841d497ff3496dd4564db P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:28.176103 28746 raft_consensus.cc:3058] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.176326 27684 ts_tablet_manager.cc:1905] T ac9b5d4f39db4f05a15316585040d142 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:28.176607 27695 ts_tablet_manager.cc:1428] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:28.176898 27695 tablet_bootstrap.cc:492] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:28.176939 27462 leader_election.cc:304] T 33fb6b357bda43e7abeaed5634183ef6 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:28.177417 28598 raft_consensus.cc:3058] T 33fb6b357bda43e7abeaed5634183ef6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.179809 28686 leader_election.cc:290] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:28.179939 27695 tablet_bootstrap.cc:654] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:28.180377 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e2f98f760cf4df4bc2a10530e2157c6" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
W20250903 20:05:28.182017 27183 leader_election.cc:343] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:28.185513 28744 raft_consensus.cc:491] T 17e0864a32d5452d84ac470b55fbaaf4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:28.185714 28744 raft_consensus.cc:513] T 17e0864a32d5452d84ac470b55fbaaf4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:28.187812 27184 leader_election.cc:343] T 17e0864a32d5452d84ac470b55fbaaf4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 17e0864a32d5452d84ac470b55fbaaf4
I20250903 20:05:28.188187 28744 leader_election.cc:290] T 17e0864a32d5452d84ac470b55fbaaf4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:28.188967 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17e0864a32d5452d84ac470b55fbaaf4" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:28.189783 27183 leader_election.cc:343] T 17e0864a32d5452d84ac470b55fbaaf4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:28.189939 27183 leader_election.cc:304] T 17e0864a32d5452d84ac470b55fbaaf4 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:28.190579 28744 raft_consensus.cc:2747] T 17e0864a32d5452d84ac470b55fbaaf4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:28.191773 27240 raft_consensus.cc:2466] T 1bad3c509460422396a17a3c4ae3d1e8 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:28.192415 27321 leader_election.cc:304] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:28.192881 28751 raft_consensus.cc:2802] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:28.193015 28751 raft_consensus.cc:695] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:28.193250 28751 consensus_queue.cc:237] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.195670 27574 heartbeater.cc:344] Connected to a master server at 127.26.65.125:44059
I20250903 20:05:28.197816 28710 raft_consensus.cc:491] T f2cc35f4244a42eeb4a65de749824c11 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:28.197989 28710 raft_consensus.cc:513] T f2cc35f4244a42eeb4a65de749824c11 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.198704 28710 leader_election.cc:290] T f2cc35f4244a42eeb4a65de749824c11 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:28.204223 27209 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:56514 (request call id 137) took 1290 ms. Trace:
I20250903 20:05:28.204351 27209 rpcz_store.cc:276] 0903 20:05:26.913614 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.913684 (+ 70us) service_pool.cc:225] Handling call
0903 20:05:27.338716 (+425032us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x6160001f2d54. stack: 00007f589abe0dc6 00007f589abe0b80 00007f5899bc83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f58a6639700 00007f58aa3d371f 00007f58a4b04c36 00007f58a4b04965 00007f58a19910be 00007f58a198ff9e 00007f58a19961d5 00007f58a1996aff 00007f58a1996928 0000000000a774dc
0903 20:05:27.589108 (+250392us) spinlock_profiling.cc:243] Waited 249 ms on lock 0x6160000c7054. stack: 00007f589abe0dc6 00007f589abe0b80 00007f5899bc83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f58a6639700 00007f58aa3d371f 00007f58a4b04c36 00007f58a4b04965 00007f58a19910be 00007f58a198ff9e 00007f58a19961d5 00007f58a1996aff 00007f58a1996928 0000000000a774dc
0903 20:05:28.204174 (+615066us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":973197056}
W20250903 20:05:28.206409 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet db3033fc47d946cc9f3e806d223d44cd consensus state has no leader
I20250903 20:05:28.207496 27692 tablet_bootstrap.cc:492] T 1fbe54dc40dd47daa109ceeb07546b0a P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:28.207628 27692 ts_tablet_manager.cc:1397] T 1fbe54dc40dd47daa109ceeb07546b0a P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.002s
I20250903 20:05:28.210580 27692 raft_consensus.cc:357] T 1fbe54dc40dd47daa109ceeb07546b0a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.210781 27692 raft_consensus.cc:383] T 1fbe54dc40dd47daa109ceeb07546b0a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:28.210836 27692 raft_consensus.cc:738] T 1fbe54dc40dd47daa109ceeb07546b0a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:28.211047 27692 consensus_queue.cc:260] T 1fbe54dc40dd47daa109ceeb07546b0a P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:28.211311 27693 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.016s sys 0.046s
W20250903 20:05:28.211409 27693 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.635s user 0.016s sys 0.046s
I20250903 20:05:28.212289 27693 tablet_bootstrap.cc:492] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:28.212423 27693 ts_tablet_manager.cc:1397] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.641s user 0.017s sys 0.048s
I20250903 20:05:28.212855 27246 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2cc35f4244a42eeb4a65de749824c11" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:28.213094 27246 raft_consensus.cc:2391] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
W20250903 20:05:28.213541 27462 leader_election.cc:343] T f2cc35f4244a42eeb4a65de749824c11 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: f2cc35f4244a42eeb4a65de749824c11
I20250903 20:05:28.213745 27462 leader_election.cc:304] T f2cc35f4244a42eeb4a65de749824c11 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:28.214103 27692 ts_tablet_manager.cc:1428] T 1fbe54dc40dd47daa109ceeb07546b0a P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250903 20:05:28.214143 28710 raft_consensus.cc:3058] T f2cc35f4244a42eeb4a65de749824c11 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.214457 27692 tablet_bootstrap.cc:492] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:28.215281 27693 raft_consensus.cc:357] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.215446 27693 raft_consensus.cc:383] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:28.215497 27693 raft_consensus.cc:738] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:28.215725 27693 consensus_queue.cc:260] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.216084 28596 raft_consensus.cc:491] T 6f8cfe53b91b41d7ba7f9074e1005af3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:28.216217 27693 ts_tablet_manager.cc:1428] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250903 20:05:28.216269 28596 raft_consensus.cc:513] T 6f8cfe53b91b41d7ba7f9074e1005af3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.216480 27693 tablet_bootstrap.cc:492] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:28.216873 28596 leader_election.cc:290] T 6f8cfe53b91b41d7ba7f9074e1005af3 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:28.218044 27693 tablet_bootstrap.cc:654] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:28.218061 27382 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cfe53b91b41d7ba7f9074e1005af3" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:28.218456 27382 raft_consensus.cc:2391] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 89f8f0ee68b940db9ce250c0604f934a in this term.
I20250903 20:05:28.221315 27692 tablet_bootstrap.cc:654] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:28.226421 27246 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cfe53b91b41d7ba7f9074e1005af3" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:28.227412 27462 leader_election.cc:304] T 6f8cfe53b91b41d7ba7f9074e1005af3 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:28.227993 28596 raft_consensus.cc:3058] T 6f8cfe53b91b41d7ba7f9074e1005af3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.238095 27695 tablet_bootstrap.cc:492] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:28.238294 27695 ts_tablet_manager.cc:1397] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.061s user 0.008s sys 0.011s
I20250903 20:05:28.239104 27695 raft_consensus.cc:357] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.239305 27695 raft_consensus.cc:383] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:28.239395 27695 raft_consensus.cc:738] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:28.239660 27695 consensus_queue.cc:260] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.240411 27695 ts_tablet_manager.cc:1428] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:28.240825 27695 tablet_bootstrap.cc:492] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
W20250903 20:05:28.243330 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 145c82e8df2c49ea89fd2f02224d8cbf consensus state has no leader
I20250903 20:05:28.243749 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_63f21b5e3dd04258a5d189a80a1da7a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:28.244866 27695 tablet_bootstrap.cc:654] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:28.244975 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_15a4e55b16784ec4b0c3115892bcf650"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:28.245936 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d4e288e7a9f9417288b39f130bab4289" } modify_external_catalogs: true
I20250903 20:05:28.246155 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d4e288e7a9f9417288b39f130bab4289" } modify_external_catalogs: true
I20250903 20:05:28.250429 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_2b9db406a0804f1c8f25adb358424581"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:28.250907 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15a4e55b16784ec4b0c3115892bcf650 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:28.251869 28522 raft_consensus.cc:513] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.252521 28522 leader_election.cc:290] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:28.254613 28598 raft_consensus.cc:2747] T 33fb6b357bda43e7abeaed5634183ef6 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:28.255580 27248 raft_consensus.cc:2466] T 1fbe54dc40dd47daa109ceeb07546b0a P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:28.256120 27321 leader_election.cc:304] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:28.256434 28751 raft_consensus.cc:2802] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:28.256579 28751 raft_consensus.cc:695] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:28.256858 28751 consensus_queue.cc:237] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.256994 27382 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59a1dd48d76241d7ba817070fbffd853" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:28.257148 27382 raft_consensus.cc:3058] T 59a1dd48d76241d7ba817070fbffd853 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.258109 27687 ts_tablet_manager.cc:1918] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:28.258908 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59a1dd48d76241d7ba817070fbffd853" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:28.259187 27524 raft_consensus.cc:3058] T 59a1dd48d76241d7ba817070fbffd853 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:28.259209 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63f21b5e3dd04258a5d189a80a1da7a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:28.259042 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b9db406a0804f1c8f25adb358424581 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:28.261281 27695 tablet_bootstrap.cc:492] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:28.261472 27695 ts_tablet_manager.cc:1397] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.004s
I20250903 20:05:28.262331 27695 raft_consensus.cc:357] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.262560 27695 raft_consensus.cc:383] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:28.262643 27695 raft_consensus.cc:738] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:28.264851 27695 consensus_queue.cc:260] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.265981 27695 ts_tablet_manager.cc:1428] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:28.266834 27695 tablet_bootstrap.cc:492] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:28.267328 27685 ts_tablet_manager.cc:1918] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 20:05:28.267722 27685 log.cc:1199] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/b6b4713936134681aaef1732d46c6ab2
I20250903 20:05:28.268316 27685 ts_tablet_manager.cc:1939] T b6b4713936134681aaef1732d46c6ab2 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:28.268764 27695 tablet_bootstrap.cc:654] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:28.279881 27695 tablet_bootstrap.cc:492] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:28.280078 27695 ts_tablet_manager.cc:1397] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.001s
I20250903 20:05:28.280968 27695 raft_consensus.cc:357] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.281191 27695 raft_consensus.cc:383] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:28.283596 27695 raft_consensus.cc:738] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:28.284190 27695 consensus_queue.cc:260] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.285384 27695 ts_tablet_manager.cc:1428] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250903 20:05:28.286448 27695 tablet_bootstrap.cc:492] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:28.290413 27695 tablet_bootstrap.cc:654] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:28.294097 28522 raft_consensus.cc:491] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:28.294327 28522 raft_consensus.cc:513] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.294963 28522 leader_election.cc:290] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:28.296020 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a223f75045345ae833f52bcc8f239f9" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:28.296671 27183 leader_election.cc:343] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:28.303071 27695 tablet_bootstrap.cc:492] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:28.303261 27695 ts_tablet_manager.cc:1397] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.002s
I20250903 20:05:28.304103 27695 raft_consensus.cc:357] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.304340 27695 raft_consensus.cc:383] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:28.304420 27695 raft_consensus.cc:738] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:28.304909 27695 consensus_queue.cc:260] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.305670 27695 ts_tablet_manager.cc:1428] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:28.308776 27695 tablet_bootstrap.cc:492] T 20c25c84c6fb4609a1964a4ff3c526da P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:28.309044 28522 raft_consensus.cc:491] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:28.309224 28522 raft_consensus.cc:513] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.309952 28522 leader_election.cc:290] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:28.310745 27695 tablet_bootstrap.cc:654] T 20c25c84c6fb4609a1964a4ff3c526da P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:28.310765 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3590506b3fd44a6c84aa469c14ed3e9f consensus state has no leader
I20250903 20:05:28.311240 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_35f8f4785b9f4c459653cff84dfcfed8" } modify_external_catalogs: true
I20250903 20:05:28.311302 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "673c45f3f2a14da9bd306011d396eaca" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:28.311442 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_35f8f4785b9f4c459653cff84dfcfed8" } modify_external_catalogs: true
W20250903 20:05:28.312085 27183 leader_election.cc:343] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:28.312611 27059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0c59083353b34b3c97dd634742350a9b consensus state has no leader
I20250903 20:05:28.313059 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_aa4be8035e9441ac8dc4e936724f5e75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:28.313704 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa4be8035e9441ac8dc4e936724f5e75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:28.323040 27695 tablet_bootstrap.cc:492] T 20c25c84c6fb4609a1964a4ff3c526da P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:28.323257 27695 ts_tablet_manager.cc:1397] T 20c25c84c6fb4609a1964a4ff3c526da P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20250903 20:05:28.324186 27695 raft_consensus.cc:357] T 20c25c84c6fb4609a1964a4ff3c526da P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.324476 27695 raft_consensus.cc:383] T 20c25c84c6fb4609a1964a4ff3c526da P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:28.324618 27695 raft_consensus.cc:738] T 20c25c84c6fb4609a1964a4ff3c526da P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:28.324968 27695 consensus_queue.cc:260] T 20c25c84c6fb4609a1964a4ff3c526da P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.325673 27695 ts_tablet_manager.cc:1428] T 20c25c84c6fb4609a1964a4ff3c526da P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:28.326284 27695 tablet_bootstrap.cc:492] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:28.328227 27695 tablet_bootstrap.cc:654] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:28.338495 28522 raft_consensus.cc:491] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:28.338773 28522 raft_consensus.cc:513] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.339469 28522 leader_election.cc:290] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:28.339783 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_cc45498ec439442797707c4fdd3692bc" } modify_external_catalogs: true
I20250903 20:05:28.340010 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_cc45498ec439442797707c4fdd3692bc" } modify_external_catalogs: true
I20250903 20:05:28.340301 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "466e501e424e46bc98e8bbc64f97005e" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:28.340997 27183 leader_election.cc:343] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:28.345933 27381 raft_consensus.cc:2466] T a4f3230f50ec4c95b528fead8c191224 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:28.347177 28655 raft_consensus.cc:513] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.347627 28710 raft_consensus.cc:2747] T f2cc35f4244a42eeb4a65de749824c11 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:28.348001 28655 leader_election.cc:290] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:28.350622 28746 raft_consensus.cc:513] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.350744 28596 raft_consensus.cc:2747] T 6f8cfe53b91b41d7ba7f9074e1005af3 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:28.351310 28746 leader_election.cc:290] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:28.352604 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53881ad30bc843839008ace4012a3f70" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:28.352782 27518 raft_consensus.cc:3058] T 53881ad30bc843839008ace4012a3f70 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.354523 27684 ts_tablet_manager.cc:1918] T ac9b5d4f39db4f05a15316585040d142 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:28.354619 27684 log.cc:1199] T ac9b5d4f39db4f05a15316585040d142 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/ac9b5d4f39db4f05a15316585040d142
I20250903 20:05:28.354897 27684 ts_tablet_manager.cc:1939] T ac9b5d4f39db4f05a15316585040d142 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:28.355346 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d9c1642fec841d497ff3496dd4564db" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:28.355536 27526 raft_consensus.cc:3058] T 2d9c1642fec841d497ff3496dd4564db P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.357115 27695 tablet_bootstrap.cc:492] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:28.357303 27695 ts_tablet_manager.cc:1397] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.012s
I20250903 20:05:28.357718 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 707) took 4806 ms. Trace:
I20250903 20:05:28.357903 27685 rpcz_store.cc:276] 0903 20:05:23.551529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:23.551592 (+ 63us) service_pool.cc:225] Handling call
0903 20:05:28.133835 (+4582243us) ts_tablet_manager.cc:1145] Deleting tablet b6b4713936134681aaef1732d46c6ab2
0903 20:05:28.133846 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:28.184712 (+ 50866us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:28.267172 (+ 82460us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:28.357669 (+ 90497us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4582040}
I20250903 20:05:28.358429 27695 raft_consensus.cc:357] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.361202 27695 raft_consensus.cc:383] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:28.361243 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet b6b4713936134681aaef1732d46c6ab2 (table default.Table_ac1cd4ca0afa4c3494fa9953c49c838e [id=d0a75545b91c48bba47fb581bb0d1a96]) successfully deleted
I20250903 20:05:28.361295 27695 raft_consensus.cc:738] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:28.361542 27695 consensus_queue.cc:260] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.362102 27695 ts_tablet_manager.cc:1428] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250903 20:05:28.362643 27695 tablet_bootstrap.cc:492] T 90baf83357494a0aae325908c118b471 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:28.367223 27064 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet d859c0222c3647dab3dddd105e4cf0fc (table default.Table_dc24dd98fc4146eda8660576fd53a29d [id=7add4b636f1b4bffae663fdf432de05f]). Expected version 4 got 0
I20250903 20:05:28.372582 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 708) took 4736 ms. Trace:
I20250903 20:05:28.372738 27685 rpcz_store.cc:276] 0903 20:05:23.635808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:23.635863 (+ 55us) service_pool.cc:225] Handling call
0903 20:05:28.367499 (+4731636us) ts_tablet_manager.cc:1145] Deleting tablet 6820c9a583af4bf68f9e63ae06e6ec64
0903 20:05:28.367511 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:28.372527 (+ 5016us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4731457}
I20250903 20:05:28.373162 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 709) took 4732 ms. Trace:
I20250903 20:05:28.373275 27685 rpcz_store.cc:276] 0903 20:05:23.640481 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:23.640541 (+ 60us) service_pool.cc:225] Handling call
0903 20:05:28.373022 (+4732481us) ts_tablet_manager.cc:1145] Deleting tablet f36938c69f6a497f93e9458bdc9c40a0
0903 20:05:28.373031 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:28.373126 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4732299}
I20250903 20:05:28.373631 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 710) took 4724 ms. Trace:
I20250903 20:05:28.373734 27685 rpcz_store.cc:276] 0903 20:05:23.649648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:23.649699 (+ 51us) service_pool.cc:225] Handling call
0903 20:05:28.373498 (+4723799us) ts_tablet_manager.cc:1145] Deleting tablet 02b3e262dd7841dfbb0520377eb94743
0903 20:05:28.373507 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:28.373598 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4723587}
I20250903 20:05:28.374039 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 711) took 4711 ms. Trace:
I20250903 20:05:28.374138 27685 rpcz_store.cc:276] 0903 20:05:23.662699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:23.662960 (+ 261us) service_pool.cc:225] Handling call
0903 20:05:28.373921 (+4710961us) ts_tablet_manager.cc:1145] Deleting tablet 46f49fd718be4a2996d781ab31268217
0903 20:05:28.373928 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:28.374006 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4708492}
I20250903 20:05:28.374462 27685 tablet_replica.cc:331] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:28.374720 27685 raft_consensus.cc:2241] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:28.374892 27685 raft_consensus.cc:2270] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:28.375454 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 6820c9a583af4bf68f9e63ae06e6ec64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6820c9a583af4bf68f9e63ae06e6ec64 already in progress: creating tablet
W20250903 20:05:28.375705 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet f36938c69f6a497f93e9458bdc9c40a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f36938c69f6a497f93e9458bdc9c40a0 already in progress: creating tablet
W20250903 20:05:28.375862 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 02b3e262dd7841dfbb0520377eb94743 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02b3e262dd7841dfbb0520377eb94743 already in progress: creating tablet
W20250903 20:05:28.376031 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 46f49fd718be4a2996d781ab31268217 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46f49fd718be4a2996d781ab31268217 already in progress: creating tablet
I20250903 20:05:28.377004 27219 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bad3c509460422396a17a3c4ae3d1e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 13496efe34944b7280e7a684f180e6e0) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:28.377235 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f8cfe53b91b41d7ba7f9074e1005af3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:28.377429 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 59a1dd48d76241d7ba817070fbffd853 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:28.377566 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet 1fbe54dc40dd47daa109ceeb07546b0a with delete_type TABLET_DATA_DELETED (replaced by tablet 5b46bb34ac214b1ba23fba1a33f746ac) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:28.377754 27221 tablet_service.cc:1515] Processing DeleteTablet for tablet a4f3230f50ec4c95b528fead8c191224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:28.379112 27685 ts_tablet_manager.cc:1905] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:28.379240 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f8cfe53b91b41d7ba7f9074e1005af3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:28.379313 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 4732783236f44467a04fe573653333f9 (DEFAULT_TABLE table=default.Table_d9f8debc2d944aa295651daee0dcb326 [id=048bf27aa393428a875fbd93f61aa61a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.379379 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet 961ade36f7d14eba91bb16d5eae2e254 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:28.379827 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4732783236f44467a04fe573653333f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.379961 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 4732783236f44467a04fe573653333f9 (DEFAULT_TABLE table=default.Table_d9f8debc2d944aa295651daee0dcb326 [id=048bf27aa393428a875fbd93f61aa61a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.381137 27206 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:56514 (request call id 139) took 1089 ms. Trace:
I20250903 20:05:28.381273 27206 rpcz_store.cc:276] 0903 20:05:27.291604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.291746 (+ 142us) service_pool.cc:225] Handling call
0903 20:05:27.777837 (+486091us) spinlock_profiling.cc:243] Waited 466 ms on lock 0x616000183d54. stack: 00007f589abe0dc6 00007f589abe0b80 00007f5899bc83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f58a6639700 00007f58aa3d371f 00007f58a4b04c36 00007f58a4b04965 00007f58a19910be 00007f58a198ff9e 00007f58a19961d5 00007f58a1996aff 00007f58a1996928 0000000000a774dc
0903 20:05:28.351999 (+574162us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x6160001f3954. stack: 00007f589abe0dc6 00007f589abe0b80 00007f5899bc83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f58a6639700 00007f58aa3d371f 00007f58a4b04c36 00007f58a4b04965 00007f58a19910be 00007f58a198ff9e 00007f58a19961d5 00007f58a1996aff 00007f58a1996928 0000000000a774dc
0903 20:05:28.381080 (+ 29081us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1420259840}
I20250903 20:05:28.382526 27499 tablet_service.cc:1468] Processing CreateTablet for tablet cab87148f7b84222915bf8eefc78a5f1 (DEFAULT_TABLE table=default.Table_331a93e34699477294aedde756d58a8f [id=22938e7305dc4d9a9f1112f60715d241]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.383138 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cab87148f7b84222915bf8eefc78a5f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.384524 27507 tablet_service.cc:1468] Processing CreateTablet for tablet eb447d66471a47d0beb03493742a0a18 (DEFAULT_TABLE table=default.Table_711e173c8a754e2b850b887bdb54e5ed [id=df8d44f7fb184c0f9fc8622082583b60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.384866 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb447d66471a47d0beb03493742a0a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.388327 27494 tablet_service.cc:1468] Processing CreateTablet for tablet a5b900e8563a429c957fe199a3471f47 (DEFAULT_TABLE table=default.Table_42e129c0c0c747e281d1dc5eb2862dd9 [id=d20a4f546c1e4821bb5b11a06d270463]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.388725 27494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5b900e8563a429c957fe199a3471f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.393265 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4732783236f44467a04fe573653333f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.394639 27214 tablet_service.cc:1468] Processing CreateTablet for tablet a5b900e8563a429c957fe199a3471f47 (DEFAULT_TABLE table=default.Table_42e129c0c0c747e281d1dc5eb2862dd9 [id=d20a4f546c1e4821bb5b11a06d270463]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.395442 27221 tablet_service.cc:1468] Processing CreateTablet for tablet cab87148f7b84222915bf8eefc78a5f1 (DEFAULT_TABLE table=default.Table_331a93e34699477294aedde756d58a8f [id=22938e7305dc4d9a9f1112f60715d241]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.395977 27215 tablet_service.cc:1468] Processing CreateTablet for tablet eb447d66471a47d0beb03493742a0a18 (DEFAULT_TABLE table=default.Table_711e173c8a754e2b850b887bdb54e5ed [id=df8d44f7fb184c0f9fc8622082583b60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.360014 27205 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:56514 (request call id 140) took 1068 ms. Trace:
I20250903 20:05:28.396548 28522 consensus_queue.cc:1035] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:28.396698 27205 rpcz_store.cc:276] 0903 20:05:27.291699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.298273 (+ 6574us) service_pool.cc:225] Handling call
0903 20:05:27.782852 (+484579us) spinlock_profiling.cc:243] Waited 472 ms on lock 0x616000183d54. stack: 00007f589abe0dc6 00007f589abe0b80 00007f5899bc83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f58a6639700 00007f58aa3d371f 00007f58a4b04c36 00007f58a4b04965 00007f58a19910be 00007f58a198ff9e 00007f58a19961d5 00007f58a1996aff 00007f58a1996928 0000000000a774dc
0903 20:05:27.861020 (+ 78168us) spinlock_profiling.cc:243] Waited 8.6 ms on lock 0x6180000e208c. stack: 00007f589abe0dc6 00007f589abe0b80 00007f5899bc83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f58a7b8c33e 00007f58a7987af1 00007f58aa3d3528 00007f58a4b04c36 00007f58a4b04965 00007f58a19910be 00007f58a198ff9e 00007f58a19961d5 00007f58a1996aff 00007f58a1996928
0903 20:05:28.355153 (+494133us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x6160001f3954. stack: 00007f589abe0dc6 00007f589abe0b80 00007f5899bc83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f58a6639700 00007f58aa3d371f 00007f58a4b04c36 00007f58a4b04965 00007f58a19910be 00007f58a198ff9e 00007f58a19961d5 00007f58a1996aff 00007f58a1996928 0000000000a774dc
0903 20:05:28.359936 (+ 4783us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1477063168}
I20250903 20:05:28.398258 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb447d66471a47d0beb03493742a0a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.398806 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5b900e8563a429c957fe199a3471f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.399235 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cab87148f7b84222915bf8eefc78a5f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.399294 28710 raft_consensus.cc:491] T a4c582b441f6467c99ba0b090abcf4cc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:28.399480 28710 raft_consensus.cc:513] T a4c582b441f6467c99ba0b090abcf4cc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.400231 28710 leader_election.cc:290] T a4c582b441f6467c99ba0b090abcf4cc P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:28.401350 27695 tablet_bootstrap.cc:654] T 90baf83357494a0aae325908c118b471 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:28.401224 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a4c582b441f6467c99ba0b090abcf4cc" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:28.401623 27248 raft_consensus.cc:2391] T a4c582b441f6467c99ba0b090abcf4cc P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
I20250903 20:05:28.403514 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 7ebac71504ec453ea11238205c50611f (DEFAULT_TABLE table=default.Table_52c0d1912aa744ebb10e196e7c81dc2e [id=ed283b2f7dc042a9991357f97f28e983]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.403950 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ebac71504ec453ea11238205c50611f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.405275 27687 log.cc:1199] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/9685cba2f73b412382a69dab2c7f50c1
I20250903 20:05:28.405872 27065 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_260dc7423c364e949e0f0a46af6cf67d" } modify_external_catalogs: true
I20250903 20:05:28.406075 27065 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_260dc7423c364e949e0f0a46af6cf67d" } modify_external_catalogs: true
I20250903 20:05:28.409227 27495 tablet_service.cc:1468] Processing CreateTablet for tablet 9766cea9d5bb407a9900c7e3bb4d2641 (DEFAULT_TABLE table=default.Table_2e1fdc83e24243fca99d90876fb34882 [id=a9aff1260c6a4b4d9d59e5248c65734d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.410188 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9766cea9d5bb407a9900c7e3bb4d2641. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.414863 27384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a223f75045345ae833f52bcc8f239f9" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:28.417019 27066 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 0f5b87c1c4a24f8abdbfd6194f3b2c4a (table default.Table_6e9cf5ead5a24831a6a14d131dd8397e [id=524a27cbe1ad4c10aa2fd4f4074105c6]). Expected version 3 got 0
I20250903 20:05:28.417299 27066 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet b0d79c78e0024b7ba3bb3356be7c8ceb (table default.Table_94e9e32ef1134f07ade1cdf8da69ead6 [id=e248f9c4b9c9406395aaea7ba3feb9ba]). Expected version 1 got 0
I20250903 20:05:28.425583 28579 raft_consensus.cc:491] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:28.426266 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 7ebac71504ec453ea11238205c50611f (DEFAULT_TABLE table=default.Table_52c0d1912aa744ebb10e196e7c81dc2e [id=ed283b2f7dc042a9991357f97f28e983]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.426926 27219 tablet_service.cc:1515] Processing DeleteTablet for tablet 673c45f3f2a14da9bd306011d396eaca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:28.427425 27219 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a223f75045345ae833f52bcc8f239f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:28.427726 27066 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 0f5b87c1c4a24f8abdbfd6194f3b2c4a (table default.Table_6e9cf5ead5a24831a6a14d131dd8397e [id=524a27cbe1ad4c10aa2fd4f4074105c6]). Expected version 3 got 0
I20250903 20:05:28.427937 27219 tablet_service.cc:1515] Processing DeleteTablet for tablet e0347eab6c8642beabdc2d7e7bbfdc9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:28.427996 27066 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet b0d79c78e0024b7ba3bb3356be7c8ceb (table default.Table_94e9e32ef1134f07ade1cdf8da69ead6 [id=e248f9c4b9c9406395aaea7ba3feb9ba]). Expected version 1 got 0
I20250903 20:05:28.428311 27219 tablet_service.cc:1515] Processing DeleteTablet for tablet a72b85dfb34047528b2b28ff0d40f828 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:28.428656 27219 tablet_service.cc:1515] Processing DeleteTablet for tablet 466e501e424e46bc98e8bbc64f97005e with delete_type TABLET_DATA_DELETED (replaced by tablet 06a79706f1184fa49d10182ba669b0ab) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:28.428831 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d9c1642fec841d497ff3496dd4564db" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:28.429138 27219 tablet_service.cc:1515] Processing DeleteTablet for tablet 0a9b7770432942da9373d5100c71bd6a with delete_type TABLET_DATA_DELETED (replaced by tablet 6800e0b6269045f29682704fd847d5d1) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:28.429481 27219 tablet_service.cc:1515] Processing DeleteTablet for tablet 607c9a9cbc5540e5bd0fa436fcba564c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:28.429910 27219 tablet_service.cc:1515] Processing DeleteTablet for tablet 20c25c84c6fb4609a1964a4ff3c526da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:28.430285 27219 tablet_service.cc:1515] Processing DeleteTablet for tablet 53881ad30bc843839008ace4012a3f70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:28.430895 27219 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e2f98f760cf4df4bc2a10530e2157c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:28.431227 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "673c45f3f2a14da9bd306011d396eaca" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:28.431357 27184 leader_election.cc:304] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:28.431778 28746 raft_consensus.cc:2802] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:28.431854 27687 ts_tablet_manager.cc:1939] T 9685cba2f73b412382a69dab2c7f50c1 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:28.431949 28746 raft_consensus.cc:695] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
W20250903 20:05:28.432261 27462 leader_election.cc:343] T a4c582b441f6467c99ba0b090abcf4cc P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: a4c582b441f6467c99ba0b090abcf4cc
I20250903 20:05:28.432255 28746 consensus_queue.cc:237] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.432406 27462 leader_election.cc:304] T a4c582b441f6467c99ba0b090abcf4cc P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:28.433197 27362 tablet_service.cc:1468] Processing CreateTablet for tablet 7ebac71504ec453ea11238205c50611f (DEFAULT_TABLE table=default.Table_52c0d1912aa744ebb10e196e7c81dc2e [id=ed283b2f7dc042a9991357f97f28e983]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.433522 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0955d4bd84b645e3b796679b761bbba7" } new_table_name: "default.Table_7554abe4629c4243bf7d0b531435a22e" modify_external_catalogs: true
I20250903 20:05:28.433808 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53881ad30bc843839008ace4012a3f70" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:28.434016 27384 raft_consensus.cc:2466] T 8a223f75045345ae833f52bcc8f239f9 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:28.435587 27359 tablet_service.cc:1468] Processing CreateTablet for tablet cab87148f7b84222915bf8eefc78a5f1 (DEFAULT_TABLE table=default.Table_331a93e34699477294aedde756d58a8f [id=22938e7305dc4d9a9f1112f60715d241]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.436450 27064 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1d4af1f6213b4b4db3bfaee9e1ede2d0" } modify_external_catalogs: true
I20250903 20:05:28.436636 27064 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1d4af1f6213b4b4db3bfaee9e1ede2d0" } modify_external_catalogs: true
I20250903 20:05:28.437572 27357 tablet_service.cc:1468] Processing CreateTablet for tablet 4732783236f44467a04fe573653333f9 (DEFAULT_TABLE table=default.Table_d9f8debc2d944aa295651daee0dcb326 [id=048bf27aa393428a875fbd93f61aa61a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.437906 28579 raft_consensus.cc:513] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.438772 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 673c45f3f2a14da9bd306011d396eaca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:28.439043 28596 raft_consensus.cc:3058] T a4c582b441f6467c99ba0b090abcf4cc P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.439102 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "466e501e424e46bc98e8bbc64f97005e" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:28.439298 27348 tablet_service.cc:1468] Processing CreateTablet for tablet a5b900e8563a429c957fe199a3471f47 (DEFAULT_TABLE table=default.Table_42e129c0c0c747e281d1dc5eb2862dd9 [id=d20a4f546c1e4821bb5b11a06d270463]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.440352 27386 raft_consensus.cc:3058] T 2d9c1642fec841d497ff3496dd4564db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.441149 27350 tablet_service.cc:1468] Processing CreateTablet for tablet eb447d66471a47d0beb03493742a0a18 (DEFAULT_TABLE table=default.Table_711e173c8a754e2b850b887bdb54e5ed [id=df8d44f7fb184c0f9fc8622082583b60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.441751 27523 raft_consensus.cc:3058] T f0efbb7eefb84ab6bee5b9625b91e298 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.442129 27517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "673c45f3f2a14da9bd306011d396eaca" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:28.442543 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ebac71504ec453ea11238205c50611f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:28.442682 27320 leader_election.cc:343] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:28.444100 27380 raft_consensus.cc:2466] T 7e2f98f760cf4df4bc2a10530e2157c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:28.444684 27382 raft_consensus.cc:2466] T 59a1dd48d76241d7ba817070fbffd853 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:28.445930 27489 tablet_service.cc:1468] Processing CreateTablet for tablet 6ce46d642ad64aa8a5edb5dd465fb9cc (DEFAULT_TABLE table=default.Table_f6d8f4f4883c410c82f937ebeae0141c [id=eea4c18229fb4833b2f8a42411aa7914]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.446338 27489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ce46d642ad64aa8a5edb5dd465fb9cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.447091 27524 raft_consensus.cc:2466] T 59a1dd48d76241d7ba817070fbffd853 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:28.451108 27183 leader_election.cc:304] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:28.451449 28655 raft_consensus.cc:2802] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:28.451601 28655 raft_consensus.cc:695] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:28.451866 28655 consensus_queue.cc:237] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:28.451926 27462 consensus_peers.cc:489] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9e90abef9814338b70a3550ab5686e0. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:28.449951 27379 raft_consensus.cc:3058] T 53881ad30bc843839008ace4012a3f70 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.453351 27184 leader_election.cc:304] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:28.453624 28522 raft_consensus.cc:2802] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:28.453748 28522 raft_consensus.cc:491] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:28.455550 27383 raft_consensus.cc:2466] T 466e501e424e46bc98e8bbc64f97005e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:28.455783 27349 tablet_service.cc:1468] Processing CreateTablet for tablet 9766cea9d5bb407a9900c7e3bb4d2641 (DEFAULT_TABLE table=default.Table_2e1fdc83e24243fca99d90876fb34882 [id=a9aff1260c6a4b4d9d59e5248c65734d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.456120 27385 raft_consensus.cc:2466] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:28.453781 27184 leader_election.cc:304] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:28.456586 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cab87148f7b84222915bf8eefc78a5f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.456912 28744 raft_consensus.cc:2802] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:28.457027 28744 raft_consensus.cc:695] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:28.457341 28744 consensus_queue.cc:237] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.459697 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb447d66471a47d0beb03493742a0a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.463268 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5b900e8563a429c957fe199a3471f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.466396 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4732783236f44467a04fe573653333f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.468082 27489 tablet_service.cc:1468] Processing CreateTablet for tablet 7203b4ba9047430ab782d4c0498466cc (DEFAULT_TABLE table=default.Table_01cecfeda8a247628e40e6d815ddb410 [id=1f0c68829d7b442d91995e377bcfc387]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.468511 27489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7203b4ba9047430ab782d4c0498466cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.469018 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9766cea9d5bb407a9900c7e3bb4d2641. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.472033 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 839) took 1967 ms. Trace:
I20250903 20:05:28.472158 27687 rpcz_store.cc:276] 0903 20:05:26.481146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.484818 (+ 3672us) service_pool.cc:225] Handling call
0903 20:05:28.048332 (+1563514us) ts_tablet_manager.cc:1145] Deleting tablet 9685cba2f73b412382a69dab2c7f50c1
0903 20:05:28.048339 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:28.163510 (+115171us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:28.257981 (+ 94471us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:28.448144 (+190163us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1563245}
I20250903 20:05:28.472595 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 841) took 1710 ms. Trace:
I20250903 20:05:28.472669 27687 rpcz_store.cc:276] 0903 20:05:26.762639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.762711 (+ 72us) service_pool.cc:225] Handling call
0903 20:05:28.472475 (+1709764us) ts_tablet_manager.cc:1145] Deleting tablet f2cc35f4244a42eeb4a65de749824c11
0903 20:05:28.472486 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:28.472561 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1709566}
I20250903 20:05:28.472898 27687 tablet_replica.cc:331] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:28.473070 27687 raft_consensus.cc:2241] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:28.473388 27687 raft_consensus.cc:2270] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:28.473839 28579 leader_election.cc:290] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:28.475018 28686 raft_consensus.cc:491] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:28.475164 28686 raft_consensus.cc:513] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.475209 27493 tablet_service.cc:1468] Processing CreateTablet for tablet e053b613eefc4ea999859847c84b8a3c (DEFAULT_TABLE table=default.Table_90b62da1b4554249986db49d13942d17 [id=1c1a09fd28bd4a1db95c54ab02aeb2c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.475633 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e053b613eefc4ea999859847c84b8a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.475819 28686 leader_election.cc:290] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:28.476382 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a72b85dfb34047528b2b28ff0d40f828" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:28.476631 27219 tablet_service.cc:1468] Processing CreateTablet for tablet 9766cea9d5bb407a9900c7e3bb4d2641 (DEFAULT_TABLE table=default.Table_2e1fdc83e24243fca99d90876fb34882 [id=a9aff1260c6a4b4d9d59e5248c65734d]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:28.476969 27183 leader_election.cc:343] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:28.477995 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 790) took 2808 ms. Trace:
I20250903 20:05:28.478071 27684 rpcz_store.cc:276] 0903 20:05:25.669869 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:25.669916 (+ 47us) service_pool.cc:225] Handling call
0903 20:05:28.175343 (+2505427us) ts_tablet_manager.cc:1145] Deleting tablet ac9b5d4f39db4f05a15316585040d142
0903 20:05:28.175353 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:28.262401 (+ 87048us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:28.354403 (+ 92002us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:28.477979 (+123576us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2505238}
I20250903 20:05:28.478564 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 791) took 2693 ms. Trace:
I20250903 20:05:28.478529 27213 tablet_service.cc:1468] Processing CreateTablet for tablet 6ce46d642ad64aa8a5edb5dd465fb9cc (DEFAULT_TABLE table=default.Table_f6d8f4f4883c410c82f937ebeae0141c [id=eea4c18229fb4833b2f8a42411aa7914]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.478653 27684 rpcz_store.cc:276] 0903 20:05:25.784733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:25.784937 (+ 204us) service_pool.cc:225] Handling call
0903 20:05:28.478435 (+2693498us) ts_tablet_manager.cc:1145] Deleting tablet 45f7068bafaf48d290ded9d1272c0e46
0903 20:05:28.478447 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:28.478546 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2693315}
I20250903 20:05:28.478742 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet ac9b5d4f39db4f05a15316585040d142 (table default.Table_f6b26b9f6ef340989d2250b8d83647c2 [id=74b402d4851f48c2bfd495e82917bb38]) successfully deleted
I20250903 20:05:28.479007 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 792) took 2694 ms. Trace:
I20250903 20:05:28.479010 27222 tablet_service.cc:1468] Processing CreateTablet for tablet 7203b4ba9047430ab782d4c0498466cc (DEFAULT_TABLE table=default.Table_01cecfeda8a247628e40e6d815ddb410 [id=1f0c68829d7b442d91995e377bcfc387]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.479084 27684 rpcz_store.cc:276] 0903 20:05:25.784870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:25.785147 (+ 277us) service_pool.cc:225] Handling call
0903 20:05:28.478902 (+2693755us) ts_tablet_manager.cc:1145] Deleting tablet b6663ba96faa44b88fe8b6def4ff6951
0903 20:05:28.478913 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:28.478992 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2693624}
W20250903 20:05:28.479216 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 45f7068bafaf48d290ded9d1272c0e46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 45f7068bafaf48d290ded9d1272c0e46 already in progress: creating tablet
I20250903 20:05:28.479432 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 799) took 2442 ms. Trace:
I20250903 20:05:28.479502 27684 rpcz_store.cc:276] 0903 20:05:26.037175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.043628 (+ 6453us) service_pool.cc:225] Handling call
0903 20:05:28.479324 (+2435696us) ts_tablet_manager.cc:1145] Deleting tablet d93258c032524b71997389b81846a5be
0903 20:05:28.479335 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:28.479417 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2435474}
W20250903 20:05:28.479637 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet b6663ba96faa44b88fe8b6def4ff6951 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6663ba96faa44b88fe8b6def4ff6951 already in progress: creating tablet
I20250903 20:05:28.479740 27684 tablet_replica.cc:331] T a4c582b441f6467c99ba0b090abcf4cc P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
W20250903 20:05:28.479974 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet d93258c032524b71997389b81846a5be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d93258c032524b71997389b81846a5be already in progress: creating tablet
I20250903 20:05:28.481556 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 9685cba2f73b412382a69dab2c7f50c1 (table default.Table_0cd65e00bd5a40bd9dc9db12fd0bda4d [id=15369700921549be8faa450d2ca98949]) successfully deleted
W20250903 20:05:28.481833 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet f2cc35f4244a42eeb4a65de749824c11 because the tablet was not found. No further retry: Not found: Tablet not found: f2cc35f4244a42eeb4a65de749824c11
I20250903 20:05:28.486581 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 7203b4ba9047430ab782d4c0498466cc (DEFAULT_TABLE table=default.Table_01cecfeda8a247628e40e6d815ddb410 [id=1f0c68829d7b442d91995e377bcfc387]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.488903 27488 tablet_service.cc:1468] Processing CreateTablet for tablet 6ff8d4902e6844d191038a6f4e4af4b1 (DEFAULT_TABLE table=default.Table_1cbf3c0f4048429680ea8fbf4b60a761 [id=733ed72efc6c4959b42da0eba2c86c9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.489315 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ff8d4902e6844d191038a6f4e4af4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.491729 27352 tablet_service.cc:1468] Processing CreateTablet for tablet 6ce46d642ad64aa8a5edb5dd465fb9cc (DEFAULT_TABLE table=default.Table_f6d8f4f4883c410c82f937ebeae0141c [id=eea4c18229fb4833b2f8a42411aa7914]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.492717 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a72b85dfb34047528b2b28ff0d40f828" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:28.493356 27687 ts_tablet_manager.cc:1905] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:28.493546 27365 tablet_service.cc:1468] Processing CreateTablet for tablet e053b613eefc4ea999859847c84b8a3c (DEFAULT_TABLE table=default.Table_90b62da1b4554249986db49d13942d17 [id=1c1a09fd28bd4a1db95c54ab02aeb2c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.500336 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_9b7897119b55487d8054d5b6479b7397"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:28.500999 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b7897119b55487d8054d5b6479b7397 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:28.503598 27503 tablet_service.cc:1468] Processing CreateTablet for tablet 2035344c7f6d478b89a89981d89f22ca (DEFAULT_TABLE table=default.Table_5f424141e5ff4643903a98bfeb011982 [id=71ee48fd20314f778340ef7420ebd16b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.504035 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2035344c7f6d478b89a89981d89f22ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.505256 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7203b4ba9047430ab782d4c0498466cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.505430 28522 raft_consensus.cc:3058] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.506067 27383 raft_consensus.cc:2466] T a72b85dfb34047528b2b28ff0d40f828 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:28.506386 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ce46d642ad64aa8a5edb5dd465fb9cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.508781 27496 tablet_service.cc:1468] Processing CreateTablet for tablet ccb32bb36a6540c39ada3c3cdab3aeaa (DEFAULT_TABLE table=default.Table_da3c2539a98b461abea7427e2b316bc2 [id=5233e14d99a44795b94bdfabfef0ba67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.509156 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccb32bb36a6540c39ada3c3cdab3aeaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.510049 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e053b613eefc4ea999859847c84b8a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.514302 27692 tablet_bootstrap.cc:492] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:28.514438 27692 ts_tablet_manager.cc:1397] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.300s user 0.025s sys 0.043s
I20250903 20:05:28.515199 27692 raft_consensus.cc:357] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.515375 27692 raft_consensus.cc:383] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:28.515422 27692 raft_consensus.cc:738] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:28.515612 27692 consensus_queue.cc:260] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.516196 27692 ts_tablet_manager.cc:1428] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:28.516479 27692 tablet_bootstrap.cc:492] T 7e2f98f760cf4df4bc2a10530e2157c6 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:28.517884 27692 tablet_bootstrap.cc:654] T 7e2f98f760cf4df4bc2a10530e2157c6 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:28.518196 28579 consensus_queue.cc:1035] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20250903 20:05:28.530284 27347 tablet_service.cc:1468] Processing CreateTablet for tablet 2035344c7f6d478b89a89981d89f22ca (DEFAULT_TABLE table=default.Table_5f424141e5ff4643903a98bfeb011982 [id=71ee48fd20314f778340ef7420ebd16b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.530503 27353 tablet_service.cc:1468] Processing CreateTablet for tablet ccb32bb36a6540c39ada3c3cdab3aeaa (DEFAULT_TABLE table=default.Table_da3c2539a98b461abea7427e2b316bc2 [id=5233e14d99a44795b94bdfabfef0ba67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.530139 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 6ff8d4902e6844d191038a6f4e4af4b1 (DEFAULT_TABLE table=default.Table_1cbf3c0f4048429680ea8fbf4b60a761 [id=733ed72efc6c4959b42da0eba2c86c9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.532699 27692 tablet_bootstrap.cc:492] T 7e2f98f760cf4df4bc2a10530e2157c6 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:28.533180 27692 ts_tablet_manager.cc:1397] T 7e2f98f760cf4df4bc2a10530e2157c6 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20250903 20:05:28.535055 27692 raft_consensus.cc:357] T 7e2f98f760cf4df4bc2a10530e2157c6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.535372 27692 raft_consensus.cc:383] T 7e2f98f760cf4df4bc2a10530e2157c6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:28.535457 27692 raft_consensus.cc:738] T 7e2f98f760cf4df4bc2a10530e2157c6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:28.536022 27692 consensus_queue.cc:260] T 7e2f98f760cf4df4bc2a10530e2157c6 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.536726 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2035344c7f6d478b89a89981d89f22ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.537115 27353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccb32bb36a6540c39ada3c3cdab3aeaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.541201 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ff8d4902e6844d191038a6f4e4af4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.544885 28522 raft_consensus.cc:513] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.545830 27685 ts_tablet_manager.cc:1918] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:28.545933 27685 log.cc:1199] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/8605bc329627438998812ed028327f53
I20250903 20:05:28.546258 27685 ts_tablet_manager.cc:1939] T 8605bc329627438998812ed028327f53 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:28.546274 27517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a223f75045345ae833f52bcc8f239f9" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:28.546710 27518 raft_consensus.cc:2466] T 53881ad30bc843839008ace4012a3f70 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
W20250903 20:05:28.546914 27183 leader_election.cc:343] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:28.547299 28522 leader_election.cc:290] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:28.551129 27183 leader_election.cc:304] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:28.551821 28686 raft_consensus.cc:2802] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:28.552001 28686 raft_consensus.cc:695] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:28.552361 28686 consensus_queue.cc:237] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.553481 27379 raft_consensus.cc:2466] T 53881ad30bc843839008ace4012a3f70 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:28.556211 27692 ts_tablet_manager.cc:1428] T 7e2f98f760cf4df4bc2a10530e2157c6 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20250903 20:05:28.556597 27692 tablet_bootstrap.cc:492] T a4f3230f50ec4c95b528fead8c191224 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:28.557922 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7203b4ba9047430ab782d4c0498466cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.558467 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ebac71504ec453ea11238205c50611f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.559008 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ce46d642ad64aa8a5edb5dd465fb9cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.559690 27219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9766cea9d5bb407a9900c7e3bb4d2641. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.568992 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a223f75045345ae833f52bcc8f239f9" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:28.569278 27379 raft_consensus.cc:3058] T 8a223f75045345ae833f52bcc8f239f9 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.572624 27692 tablet_bootstrap.cc:654] T a4f3230f50ec4c95b528fead8c191224 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:28.592226 27222 tablet_service.cc:1468] Processing CreateTablet for tablet e053b613eefc4ea999859847c84b8a3c (DEFAULT_TABLE table=default.Table_90b62da1b4554249986db49d13942d17 [id=1c1a09fd28bd4a1db95c54ab02aeb2c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.592326 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 6ff8d4902e6844d191038a6f4e4af4b1 (DEFAULT_TABLE table=default.Table_1cbf3c0f4048429680ea8fbf4b60a761 [id=733ed72efc6c4959b42da0eba2c86c9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.592728 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e053b613eefc4ea999859847c84b8a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.593413 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 2035344c7f6d478b89a89981d89f22ca (DEFAULT_TABLE table=default.Table_5f424141e5ff4643903a98bfeb011982 [id=71ee48fd20314f778340ef7420ebd16b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.593540 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ff8d4902e6844d191038a6f4e4af4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.593734 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2035344c7f6d478b89a89981d89f22ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.598609 27224 tablet_service.cc:1468] Processing CreateTablet for tablet ccb32bb36a6540c39ada3c3cdab3aeaa (DEFAULT_TABLE table=default.Table_da3c2539a98b461abea7427e2b316bc2 [id=5233e14d99a44795b94bdfabfef0ba67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:28.598999 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccb32bb36a6540c39ada3c3cdab3aeaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:28.608100 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "673c45f3f2a14da9bd306011d396eaca" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:28.608305 27249 raft_consensus.cc:2466] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:28.609025 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 712) took 4830 ms. Trace:
I20250903 20:05:28.609133 27685 rpcz_store.cc:276] 0903 20:05:23.778255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:23.778697 (+ 442us) service_pool.cc:225] Handling call
0903 20:05:28.374396 (+4595699us) ts_tablet_manager.cc:1145] Deleting tablet 8605bc329627438998812ed028327f53
0903 20:05:28.374405 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:28.445668 (+ 71263us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:28.545701 (+100033us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:28.609009 (+ 63308us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4595525}
I20250903 20:05:28.609501 27685 tablet_replica.cc:331] T c6d67cd0c0304c77ba571b414878dad2 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:28.611136 27685 raft_consensus.cc:2241] T c6d67cd0c0304c77ba571b414878dad2 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:28.611299 27685 pending_rounds.cc:70] T c6d67cd0c0304c77ba571b414878dad2 P 89f8f0ee68b940db9ce250c0604f934a: Trying to abort 1 pending ops.
I20250903 20:05:28.611339 27685 pending_rounds.cc:77] T c6d67cd0c0304c77ba571b414878dad2 P 89f8f0ee68b940db9ce250c0604f934a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384978073034752 op_type: NO_OP noop_request { }
I20250903 20:05:28.611435 27693 tablet_bootstrap.cc:492] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:28.611436 27685 raft_consensus.cc:2887] T c6d67cd0c0304c77ba571b414878dad2 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:28.611541 27685 raft_consensus.cc:2270] T c6d67cd0c0304c77ba571b414878dad2 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:28.611613 27693 ts_tablet_manager.cc:1397] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.395s user 0.006s sys 0.007s
I20250903 20:05:28.612169 27685 ts_tablet_manager.cc:1905] T c6d67cd0c0304c77ba571b414878dad2 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:28.612593 27693 raft_consensus.cc:357] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.612811 27693 raft_consensus.cc:383] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:28.612872 27693 raft_consensus.cc:738] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:28.613133 27693 consensus_queue.cc:260] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.613775 27693 ts_tablet_manager.cc:1428] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:28.614141 27693 tablet_bootstrap.cc:492] T 0a9b7770432942da9373d5100c71bd6a P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:28.615298 28655 raft_consensus.cc:491] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:28.615551 28655 raft_consensus.cc:513] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.617616 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0a9b7770432942da9373d5100c71bd6a" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:28.618359 27183 leader_election.cc:343] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:28.618611 28655 leader_election.cc:290] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:28.620518 27693 tablet_bootstrap.cc:654] T 0a9b7770432942da9373d5100c71bd6a P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:28.622769 27321 leader_election.cc:304] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:28.623153 28751 consensus_queue.cc:1035] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:28.623378 28567 raft_consensus.cc:2802] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:28.623458 28567 raft_consensus.cc:491] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:28.623515 28567 raft_consensus.cc:3058] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.623714 28751 consensus_queue.cc:1035] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:28.593425 27184 leader_election.cc:304] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:28.638186 27386 raft_consensus.cc:2466] T 2d9c1642fec841d497ff3496dd4564db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:28.648542 28596 raft_consensus.cc:2747] T a4c582b441f6467c99ba0b090abcf4cc P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:28.650696 27184 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 57337.1us
I20250903 20:05:28.654711 27184 leader_election.cc:304] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:28.666713 27184 leader_election.cc:304] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:28.670897 28745 raft_consensus.cc:2802] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:28.671118 28745 raft_consensus.cc:491] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:28.671286 28745 raft_consensus.cc:3058] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.655272 27695 tablet_bootstrap.cc:492] T 90baf83357494a0aae325908c118b471 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:28.674844 27695 ts_tablet_manager.cc:1397] T 90baf83357494a0aae325908c118b471 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.312s user 0.009s sys 0.000s
I20250903 20:05:28.676164 27695 raft_consensus.cc:357] T 90baf83357494a0aae325908c118b471 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.676458 27695 raft_consensus.cc:383] T 90baf83357494a0aae325908c118b471 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:28.676555 27695 raft_consensus.cc:738] T 90baf83357494a0aae325908c118b471 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:28.676904 27695 consensus_queue.cc:260] T 90baf83357494a0aae325908c118b471 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.677210 28522 consensus_queue.cc:1035] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:28.677837 27695 ts_tablet_manager.cc:1428] T 90baf83357494a0aae325908c118b471 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250903 20:05:28.667029 28744 raft_consensus.cc:2802] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:28.651152 28746 raft_consensus.cc:2802] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:28.685076 27684 raft_consensus.cc:2241] T a4c582b441f6467c99ba0b090abcf4cc P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:28.685464 27684 raft_consensus.cc:2270] T a4c582b441f6467c99ba0b090abcf4cc P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:28.686112 27684 ts_tablet_manager.cc:1905] T a4c582b441f6467c99ba0b090abcf4cc P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:28.678346 28744 raft_consensus.cc:491] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:28.685858 28522 consensus_queue.cc:1035] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250903 20:05:28.689137 28744 raft_consensus.cc:3058] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.678422 28746 raft_consensus.cc:491] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:28.689512 28746 raft_consensus.cc:3058] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.685964 27184 leader_election.cc:304] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:28.690029 28522 raft_consensus.cc:2802] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:28.690135 28522 raft_consensus.cc:695] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:28.690480 28522 consensus_queue.cc:237] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.691066 27693 tablet_bootstrap.cc:492] T 0a9b7770432942da9373d5100c71bd6a P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:28.691195 27693 ts_tablet_manager.cc:1397] T 0a9b7770432942da9373d5100c71bd6a P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.077s user 0.004s sys 0.004s
I20250903 20:05:28.691872 27693 raft_consensus.cc:357] T 0a9b7770432942da9373d5100c71bd6a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.692040 27693 raft_consensus.cc:383] T 0a9b7770432942da9373d5100c71bd6a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:28.692099 27693 raft_consensus.cc:738] T 0a9b7770432942da9373d5100c71bd6a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:28.692276 27693 consensus_queue.cc:260] T 0a9b7770432942da9373d5100c71bd6a P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.692766 27693 ts_tablet_manager.cc:1428] T 0a9b7770432942da9373d5100c71bd6a P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:28.693048 27693 tablet_bootstrap.cc:492] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:28.694576 27693 tablet_bootstrap.cc:654] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:28.702232 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 8605bc329627438998812ed028327f53 (table default.Table_8b0346420a89417b9ae2005be49e9f3a [id=b0b2e2d5798346708995a6b8dc028918]) successfully deleted
I20250903 20:05:28.702632 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0a9b7770432942da9373d5100c71bd6a" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:28.702836 27386 raft_consensus.cc:2466] T 0a9b7770432942da9373d5100c71bd6a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:28.678196 27695 tablet_bootstrap.cc:492] T a9f43781299e47a8aecc37a0b73d7a64 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:28.716023 27059 master_service.cc:948] ReplaceTablet: received request to replace tablet c6d67cd0c0304c77ba571b414878dad2 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:28.716478 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 4739f10128d64e2c986257b1d9deb90f
I20250903 20:05:28.719264 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e58baad01a28404886961ac38943a233" } new_table_name: "default.Table_1de90214f1cd4c43a4c54befcb77e3fe" modify_external_catalogs: true
I20250903 20:05:28.716760 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet b3bb55085af044ec8bb2368c845e7404
I20250903 20:05:28.724654 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_8851a4dad8da4bb8907db1ff956dd28f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:28.725461 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8851a4dad8da4bb8907db1ff956dd28f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:28.717018 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet de20d18d7f264811a0631aa1144aef2c
I20250903 20:05:28.727126 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d3c128a2d7454429bbe0c058c808a9f5" } new_table_name: "default.Table_212106122e7c4c44a50ca8c0d4b381f7" modify_external_catalogs: true
I20250903 20:05:28.731585 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_7eab6c911ff34aa7a4e2e94cd4087de4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:28.734544 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7eab6c911ff34aa7a4e2e94cd4087de4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:28.736672 28686 consensus_queue.cc:1035] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:28.736896 27498 tablet_service.cc:1515] Processing DeleteTablet for tablet 4739f10128d64e2c986257b1d9deb90f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:28.737097 27500 tablet_service.cc:1515] Processing DeleteTablet for tablet b3bb55085af044ec8bb2368c845e7404 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:28.737221 27498 tablet_service.cc:1515] Processing DeleteTablet for tablet de20d18d7f264811a0631aa1144aef2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:28.739639 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 5962685d79044081a75532e19bfa2bce
I20250903 20:05:28.740305 28745 raft_consensus.cc:513] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.740628 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ffe9c64658a24e47b91b43fe8ae41eba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:28.740694 28715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35bdb0090d8747059b5ee103e786e979. Latest consensus state: current_term: 3 leader_uuid: "35bdb0090d8747059b5ee103e786e979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:28.741178 28715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [sys.catalog]: This master's current role is: FOLLOWER
W20250903 20:05:28.741321 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ffe9c64658a24e47b91b43fe8ae41eba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:28.741371 28769 raft_consensus.cc:491] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 35bdb0090d8747059b5ee103e786e979)
I20250903 20:05:28.741619 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a72b85dfb34047528b2b28ff0d40f828" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:28.741601 28769 raft_consensus.cc:513] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } }
W20250903 20:05:28.742249 27183 leader_election.cc:343] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:28.742442 28745 leader_election.cc:290] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:28.742910 27685 ts_tablet_manager.cc:1918] T c6d67cd0c0304c77ba571b414878dad2 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:28.743017 27685 log.cc:1199] T c6d67cd0c0304c77ba571b414878dad2 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/c6d67cd0c0304c77ba571b414878dad2
I20250903 20:05:28.743263 27526 raft_consensus.cc:2466] T 2d9c1642fec841d497ff3496dd4564db P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:28.750689 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a72b85dfb34047528b2b28ff0d40f828" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:28.750937 27383 raft_consensus.cc:3058] T a72b85dfb34047528b2b28ff0d40f828 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.751057 28769 leader_election.cc:290] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8857b5d4ea1e4f45aafb8e029837cca2 (127.26.65.126:39015), 35bdb0090d8747059b5ee103e786e979 (127.26.65.124:40911)
I20250903 20:05:28.752543 27184 leader_election.cc:304] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:28.753420 28745 raft_consensus.cc:2802] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:28.753501 28745 raft_consensus.cc:491] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:28.753557 28745 raft_consensus.cc:3058] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:28.767969 27050 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet b7a6d5d8a97a4769a81ec7224d591f8c (table default.Table_f752a52fa3ba4ac99bc29428fa871318 [id=21b4fd8022f6461691f87200e040e264], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet b7a6d5d8a97a4769a81ec7224d591f8c consensus state has no leader
I20250903 20:05:28.769867 27379 raft_consensus.cc:2466] T 8a223f75045345ae833f52bcc8f239f9 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:28.781123 27488 tablet_service.cc:1515] Processing DeleteTablet for tablet 5962685d79044081a75532e19bfa2bce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:28.782162 27352 tablet_service.cc:1515] Processing DeleteTablet for tablet 4739f10128d64e2c986257b1d9deb90f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:28.782467 27350 tablet_service.cc:1515] Processing DeleteTablet for tablet b3bb55085af044ec8bb2368c845e7404 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:28.782626 27352 tablet_service.cc:1515] Processing DeleteTablet for tablet 5962685d79044081a75532e19bfa2bce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:28.782912 27063 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 0f5b87c1c4a24f8abdbfd6194f3b2c4a (table default.Table_6e9cf5ead5a24831a6a14d131dd8397e [id=524a27cbe1ad4c10aa2fd4f4074105c6]). Expected version 3 got 0
I20250903 20:05:28.783280 27063 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet b0d79c78e0024b7ba3bb3356be7c8ceb (table default.Table_94e9e32ef1134f07ade1cdf8da69ead6 [id=e248f9c4b9c9406395aaea7ba3feb9ba]). Expected version 1 got 0
I20250903 20:05:28.783541 27063 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 3b3bfc63aa0b4e56bec189e834dc6a8d (table default.Table_9b20469406c94986ab496ea3e678b586 [id=6d6f5e8cadf9445d8695c075cfabb2ea]). Expected version 1 got 0
I20250903 20:05:28.783921 28751 raft_consensus.cc:491] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:28.784051 28751 raft_consensus.cc:513] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.784313 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet de20d18d7f264811a0631aa1144aef2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:28.784664 28751 leader_election.cc:290] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:28.784938 27057 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 0ad64c699b294679b6fec37b5ad23528 (table default.Table_24e55a7c820c413f8514bd3ad7bd3269 [id=0904dc5d0c594bc3bbf32de2508c6195]). Expected version 3 got 0
I20250903 20:05:28.785566 28596 consensus_queue.cc:1035] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:28.785851 27057 master_service.cc:948] ReplaceTablet: received request to replace tablet cac38daf002c4ae698d72903a277a587 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:28.786463 27057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_05d58d894486468a8a15aaa4b4c0cabb" } new_table_name: "default.Table_adf05e59a7be478b89666c3162ac83f1" modify_external_catalogs: true
I20250903 20:05:28.786567 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20c25c84c6fb4609a1964a4ff3c526da" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:28.787417 27320 leader_election.cc:343] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:28.788071 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20c25c84c6fb4609a1964a4ff3c526da" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:28.788275 27249 raft_consensus.cc:2466] T 20c25c84c6fb4609a1964a4ff3c526da P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:28.788786 27321 leader_election.cc:304] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:28.789108 28751 raft_consensus.cc:2802] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:28.789191 28751 raft_consensus.cc:491] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:28.789288 28751 raft_consensus.cc:3058] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.791055 28404 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49754 (request call id 0) took 4934 ms. Trace:
I20250903 20:05:28.797132 27692 tablet_bootstrap.cc:492] T a4f3230f50ec4c95b528fead8c191224 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:28.797492 28797 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250903 20:05:28.798719 27184 leader_election.cc:304] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:28.804067 27692 ts_tablet_manager.cc:1397] T a4f3230f50ec4c95b528fead8c191224 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.248s user 0.011s sys 0.009s
I20250903 20:05:28.804905 27692 raft_consensus.cc:357] T a4f3230f50ec4c95b528fead8c191224 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.805106 27692 raft_consensus.cc:383] T a4f3230f50ec4c95b528fead8c191224 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:28.805168 27692 raft_consensus.cc:738] T a4f3230f50ec4c95b528fead8c191224 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:28.805359 27692 consensus_queue.cc:260] T a4f3230f50ec4c95b528fead8c191224 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.805850 27692 ts_tablet_manager.cc:1428] T a4f3230f50ec4c95b528fead8c191224 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:28.806138 27692 tablet_bootstrap.cc:492] T e0347eab6c8642beabdc2d7e7bbfdc9b P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:28.807899 27692 tablet_bootstrap.cc:654] T e0347eab6c8642beabdc2d7e7bbfdc9b P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:28.811156 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 13496efe34944b7280e7a684f180e6e0
I20250903 20:05:28.812062 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet 13496efe34944b7280e7a684f180e6e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:28.812036 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1310291579564dd18d0b47d1050cf905" } new_table_name: "default.Table_b677e454454e465f8717a74bb5a4596b" modify_external_catalogs: true
W20250903 20:05:28.812527 27066 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 53b00774b745462091321ea47fcc2114 consensus state has no leader
I20250903 20:05:28.812940 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_5b06e203567741abb0b8790b44715a44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:28.813213 27488 tablet_service.cc:1515] Processing DeleteTablet for tablet 13496efe34944b7280e7a684f180e6e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:28.813596 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b06e203567741abb0b8790b44715a44 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:28.815744 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_3bcd6f6a18284a278b1467fe1c2374a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:28.816411 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3bcd6f6a18284a278b1467fe1c2374a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:28.828306 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 908e748d08f74ed2a3071312c95ec07c consensus state has no leader
I20250903 20:05:28.828749 27061 master_service.cc:948] ReplaceTablet: received request to replace tablet 86e33bd235c14042a578a4a1c70f11b7 from {username='slave'} at 127.0.0.1:45936
W20250903 20:05:28.830483 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f6df379fb1a54e1cbc2aff4852c3f313 consensus state has no leader
W20250903 20:05:28.830577 27057 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7a9e55f0811345f7afa71645f38ebdaf consensus state has no leader
I20250903 20:05:28.830773 27060 master_service.cc:948] ReplaceTablet: received request to replace tablet 0fef9aa7153a4902abce522e3ff0df3f from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:28.831110 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_67b0c9f0d9164596a34c1f4a5a8cfe2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:28.831874 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67b0c9f0d9164596a34c1f4a5a8cfe2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:28.839258 28596 raft_consensus.cc:491] T 1fbe54dc40dd47daa109ceeb07546b0a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:28.839449 28596 raft_consensus.cc:513] T 1fbe54dc40dd47daa109ceeb07546b0a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.839979 27685 ts_tablet_manager.cc:1939] T c6d67cd0c0304c77ba571b414878dad2 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:28.840129 28596 leader_election.cc:290] T 1fbe54dc40dd47daa109ceeb07546b0a P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:28.841717 27488 tablet_service.cc:1515] Processing DeleteTablet for tablet 53881ad30bc843839008ace4012a3f70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:28.841909 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bad3c509460422396a17a3c4ae3d1e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 13496efe34944b7280e7a684f180e6e0) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:28.842094 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 1fbe54dc40dd47daa109ceeb07546b0a with delete_type TABLET_DATA_DELETED (replaced by tablet 5b46bb34ac214b1ba23fba1a33f746ac) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:28.842254 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet f0efbb7eefb84ab6bee5b9625b91e298 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:28.842423 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 59a1dd48d76241d7ba817070fbffd853 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:28.842818 28567 raft_consensus.cc:513] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.843465 28567 leader_election.cc:290] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:28.843775 28744 raft_consensus.cc:513] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.845479 27246 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1fbe54dc40dd47daa109ceeb07546b0a" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:28.845587 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 713) took 5067 ms. Trace:
I20250903 20:05:28.845685 27246 raft_consensus.cc:2391] T 1fbe54dc40dd47daa109ceeb07546b0a P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
I20250903 20:05:28.845652 27685 rpcz_store.cc:276] 0903 20:05:23.778368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:23.778908 (+ 540us) service_pool.cc:225] Handling call
0903 20:05:28.609462 (+4830554us) ts_tablet_manager.cc:1145] Deleting tablet c6d67cd0c0304c77ba571b414878dad2
0903 20:05:28.609471 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:28.650485 (+ 41014us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:28.742787 (+ 92302us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:28.845571 (+102784us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4830418}
I20250903 20:05:28.845964 27685 tablet_replica.cc:331] T fe902f1b58924be1ac3a5585413f7789 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:28.846189 27685 raft_consensus.cc:2241] T fe902f1b58924be1ac3a5585413f7789 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:28.846387 27685 raft_consensus.cc:2270] T fe902f1b58924be1ac3a5585413f7789 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:28.846628 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "673c45f3f2a14da9bd306011d396eaca" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
W20250903 20:05:28.847201 27320 leader_election.cc:343] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:28.847926 27246 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "673c45f3f2a14da9bd306011d396eaca" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:28.848155 27246 raft_consensus.cc:2391] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 51578fb7bceb4e119b3ab1e58c4f5528 in current term 1: Already voted for candidate 89f8f0ee68b940db9ce250c0604f934a in this term.
I20250903 20:05:28.848450 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1fbe54dc40dd47daa109ceeb07546b0a" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:28.849155 27462 leader_election.cc:304] T 1fbe54dc40dd47daa109ceeb07546b0a P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:28.849692 28596 raft_consensus.cc:3058] T 1fbe54dc40dd47daa109ceeb07546b0a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.850072 27684 ts_tablet_manager.cc:1918] T a4c582b441f6467c99ba0b090abcf4cc P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:28.850188 27684 log.cc:1199] T a4c582b441f6467c99ba0b090abcf4cc P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/a4c582b441f6467c99ba0b090abcf4cc
I20250903 20:05:28.850183 27059 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 3b3bfc63aa0b4e56bec189e834dc6a8d (table default.Table_9b20469406c94986ab496ea3e678b586 [id=6d6f5e8cadf9445d8695c075cfabb2ea]). Expected version 1 got 0
I20250903 20:05:28.850534 27684 ts_tablet_manager.cc:1939] T a4c582b441f6467c99ba0b090abcf4cc P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:28.851377 28567 consensus_queue.cc:1035] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:28.854352 28710 raft_consensus.cc:491] T f0efbb7eefb84ab6bee5b9625b91e298 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:28.854487 28710 raft_consensus.cc:513] T f0efbb7eefb84ab6bee5b9625b91e298 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.855074 28710 leader_election.cc:290] T f0efbb7eefb84ab6bee5b9625b91e298 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:28.855851 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0efbb7eefb84ab6bee5b9625b91e298" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:28.861552 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet a4f3230f50ec4c95b528fead8c191224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:28.862591 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e2f98f760cf4df4bc2a10530e2157c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:28.865085 28371 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" instance_seqno: 1756929907545471) as {username='slave'} at 127.26.65.67:33221; Asking this server to re-register.
I20250903 20:05:28.865936 28780 raft_consensus.cc:2802] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:28.866050 28780 raft_consensus.cc:695] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:28.871301 27574 heartbeater.cc:461] Registering TS with master...
I20250903 20:05:28.872778 27685 ts_tablet_manager.cc:1905] T fe902f1b58924be1ac3a5585413f7789 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:28.873796 28744 leader_election.cc:290] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:28.874761 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "673c45f3f2a14da9bd306011d396eaca" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
W20250903 20:05:28.875886 27183 leader_election.cc:343] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:28.882684 27873 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" candidate_term: 4 candidate_status { last_received { term: 3 index: 933 } } ignore_live_leader: false dest_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" is_pre_election: true
I20250903 20:05:28.883966 28371 ts_manager.cc:194] Registered new tserver with Master: e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:28.886901 27076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" candidate_term: 4 candidate_status { last_received { term: 3 index: 933 } } ignore_live_leader: false dest_uuid: "35bdb0090d8747059b5ee103e786e979" is_pre_election: true
I20250903 20:05:28.887925 28317 leader_election.cc:304] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35dd69f3fa7a4fd8843b97f8a5d68296; no voters: 35bdb0090d8747059b5ee103e786e979, 8857b5d4ea1e4f45aafb8e029837cca2
I20250903 20:05:28.888545 28769 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250903 20:05:28.889124 28780 consensus_queue.cc:237] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.908864 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b4d6ce0289db42f8aa51e69555580156"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:28.909838 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4d6ce0289db42f8aa51e69555580156 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:28.911722 27695 tablet_bootstrap.cc:654] T a9f43781299e47a8aecc37a0b73d7a64 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:28.912699 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_7197fb56c22c40948ede1c2959901507"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:28.913448 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7197fb56c22c40948ede1c2959901507 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:28.914791 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_158bc2760d3c4441840cb7f7eae71ff0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:28.915421 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_158bc2760d3c4441840cb7f7eae71ff0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:28.922552 27321 leader_election.cc:304] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:28.923028 28567 raft_consensus.cc:2747] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250903 20:05:28.924871 28744 raft_consensus.cc:491] T 90baf83357494a0aae325908c118b471 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:28.925009 28744 raft_consensus.cc:513] T 90baf83357494a0aae325908c118b471 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.925575 28744 leader_election.cc:290] T 90baf83357494a0aae325908c118b471 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:28.926388 27523 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90baf83357494a0aae325908c118b471" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:28.926910 27183 leader_election.cc:343] T 90baf83357494a0aae325908c118b471 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:28.929255 27692 tablet_bootstrap.cc:492] T e0347eab6c8642beabdc2d7e7bbfdc9b P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:28.929409 27692 ts_tablet_manager.cc:1397] T e0347eab6c8642beabdc2d7e7bbfdc9b P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.123s user 0.003s sys 0.000s
I20250903 20:05:28.930254 27692 raft_consensus.cc:357] T e0347eab6c8642beabdc2d7e7bbfdc9b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.930423 27692 raft_consensus.cc:383] T e0347eab6c8642beabdc2d7e7bbfdc9b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:28.930472 27692 raft_consensus.cc:738] T e0347eab6c8642beabdc2d7e7bbfdc9b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:28.930662 27692 consensus_queue.cc:260] T e0347eab6c8642beabdc2d7e7bbfdc9b P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.931166 27692 ts_tablet_manager.cc:1428] T e0347eab6c8642beabdc2d7e7bbfdc9b P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250903 20:05:28.931463 27692 tablet_bootstrap.cc:492] T 8a223f75045345ae833f52bcc8f239f9 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
W20250903 20:05:28.933359 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8292ae4c5bdd469681f3a3b295b24349 consensus state has no leader
I20250903 20:05:28.939082 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7554abe4629c4243bf7d0b531435a22e" } modify_external_catalogs: true
I20250903 20:05:28.939169 28710 raft_consensus.cc:491] T 1bad3c509460422396a17a3c4ae3d1e8 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:28.939270 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7554abe4629c4243bf7d0b531435a22e" } modify_external_catalogs: true
I20250903 20:05:28.939337 28710 raft_consensus.cc:513] T 1bad3c509460422396a17a3c4ae3d1e8 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:28.939002 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_973e08d4f7e34fde936b93bb0b3981b4" } modify_external_catalogs: true
I20250903 20:05:28.939813 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_973e08d4f7e34fde936b93bb0b3981b4" } modify_external_catalogs: true
I20250903 20:05:28.939957 28710 leader_election.cc:290] T 1bad3c509460422396a17a3c4ae3d1e8 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:28.941089 27064 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2e1fdc83e24243fca99d90876fb34882" } modify_external_catalogs: true
I20250903 20:05:28.941262 27064 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2e1fdc83e24243fca99d90876fb34882" } modify_external_catalogs: true
I20250903 20:05:28.944034 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 804) took 2829 ms. Trace:
I20250903 20:05:28.944149 27684 rpcz_store.cc:276] 0903 20:05:26.114648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.114713 (+ 65us) service_pool.cc:225] Handling call
0903 20:05:28.479708 (+2364995us) ts_tablet_manager.cc:1145] Deleting tablet a4c582b441f6467c99ba0b090abcf4cc
0903 20:05:28.479715 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:28.685340 (+205625us) spinlock_profiling.cc:243] Waited 205 ms on lock 0x6160001b0a54. stack: 00007f71bc6f7dc6 00007f71bc6f7b80 00007f71bb6df3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f71c8181600 00007f71c9494b13 00007f71cbf772b7 00007f71cbfa9344 00007f71cbf8ada0 00007f71cbf8ab2d 0000000000a774dc 00007f71bc73e14f 00007f71bc74141b 00007f71bc741298
0903 20:05:28.745560 (+ 60220us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:28.849952 (+104392us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:28.944004 (+ 94052us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":471675264,"tablet-delete.queue_time_us":2364798}
I20250903 20:05:28.944621 27684 tablet_replica.cc:331] T cd6d6bb514134bdf8e3bc0aaad180732 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:28.944834 27684 raft_consensus.cc:2241] T cd6d6bb514134bdf8e3bc0aaad180732 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:28.944981 27684 raft_consensus.cc:2270] T cd6d6bb514134bdf8e3bc0aaad180732 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:28.945668 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bad3c509460422396a17a3c4ae3d1e8" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:28.945940 27239 raft_consensus.cc:2391] T 1bad3c509460422396a17a3c4ae3d1e8 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
I20250903 20:05:28.946103 28745 raft_consensus.cc:513] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.946756 28745 leader_election.cc:290] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:28.948208 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0efbb7eefb84ab6bee5b9625b91e298" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:28.948657 27685 ts_tablet_manager.cc:1918] T fe902f1b58924be1ac3a5585413f7789 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:28.948747 27685 log.cc:1199] T fe902f1b58924be1ac3a5585413f7789 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/fe902f1b58924be1ac3a5585413f7789
I20250903 20:05:28.948751 27523 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0a9b7770432942da9373d5100c71bd6a" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:28.949122 27685 ts_tablet_manager.cc:1939] T fe902f1b58924be1ac3a5585413f7789 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
W20250903 20:05:28.949285 27183 leader_election.cc:343] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:28.949636 28596 raft_consensus.cc:2747] T 1fbe54dc40dd47daa109ceeb07546b0a P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:28.950433 27462 leader_election.cc:304] T f0efbb7eefb84ab6bee5b9625b91e298 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:28.950851 28751 raft_consensus.cc:513] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.945746 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bad3c509460422396a17a3c4ae3d1e8" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:28.951453 28596 raft_consensus.cc:2747] T f0efbb7eefb84ab6bee5b9625b91e298 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:28.951608 28751 leader_election.cc:290] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:28.952374 27462 leader_election.cc:304] T 1bad3c509460422396a17a3c4ae3d1e8 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:28.952780 27524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20c25c84c6fb4609a1964a4ff3c526da" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:28.953148 28710 raft_consensus.cc:3058] T 1bad3c509460422396a17a3c4ae3d1e8 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:28.955919 27320 leader_election.cc:343] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:28.956251 27684 ts_tablet_manager.cc:1905] T cd6d6bb514134bdf8e3bc0aaad180732 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:28.961062 27060 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 3b3bfc63aa0b4e56bec189e834dc6a8d (table default.Table_9b20469406c94986ab496ea3e678b586 [id=6d6f5e8cadf9445d8695c075cfabb2ea]). Expected version 1 got 0
I20250903 20:05:28.962757 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 20c25c84c6fb4609a1964a4ff3c526da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:28.962810 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20c25c84c6fb4609a1964a4ff3c526da" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:28.962993 27239 raft_consensus.cc:3058] T 20c25c84c6fb4609a1964a4ff3c526da P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:28.964871 27695 tablet_bootstrap.cc:492] T a9f43781299e47a8aecc37a0b73d7a64 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:28.965013 27695 ts_tablet_manager.cc:1397] T a9f43781299e47a8aecc37a0b73d7a64 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.287s user 0.000s sys 0.011s
I20250903 20:05:28.966243 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet e0347eab6c8642beabdc2d7e7bbfdc9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:28.966153 27695 raft_consensus.cc:357] T a9f43781299e47a8aecc37a0b73d7a64 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.966363 27695 raft_consensus.cc:383] T a9f43781299e47a8aecc37a0b73d7a64 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:28.966426 27695 raft_consensus.cc:738] T a9f43781299e47a8aecc37a0b73d7a64 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:28.966606 27695 consensus_queue.cc:260] T a9f43781299e47a8aecc37a0b73d7a64 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.967113 27695 ts_tablet_manager.cc:1428] T a9f43781299e47a8aecc37a0b73d7a64 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250903 20:05:28.967418 27695 tablet_bootstrap.cc:492] T f9709dfd18654c5bb071e39da7c8889f P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:28.974771 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 0a9b7770432942da9373d5100c71bd6a with delete_type TABLET_DATA_DELETED (replaced by tablet 6800e0b6269045f29682704fd847d5d1) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:28.975502 28371 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" instance_seqno: 1756929906558199) as {username='slave'} at 127.26.65.65:57187; Asking this server to re-register.
I20250903 20:05:28.979166 27296 heartbeater.cc:461] Registering TS with master...
I20250903 20:05:28.980736 27695 tablet_bootstrap.cc:654] T f9709dfd18654c5bb071e39da7c8889f P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:28.984577 28371 ts_manager.cc:194] Registered new tserver with Master: 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:28.985833 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_9e9e434350e54dc4a9397cee96d16383" } modify_external_catalogs: true
I20250903 20:05:28.986099 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_9e9e434350e54dc4a9397cee96d16383" } modify_external_catalogs: true
I20250903 20:05:28.991990 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 86e33bd235c14042a578a4a1c70f11b7
I20250903 20:05:28.992259 27693 tablet_bootstrap.cc:492] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:28.992790 27061 catalog_manager.cc:6652] ReplaceTablet: tablet 86e33bd235c14042a578a4a1c70f11b7 deleted and replaced by tablet 827d186faaa140bdab2f87cf421ca64c
I20250903 20:05:28.992858 27693 ts_tablet_manager.cc:1397] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.300s user 0.002s sys 0.011s
I20250903 20:05:28.992945 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet 86e33bd235c14042a578a4a1c70f11b7 with delete_type TABLET_DATA_DELETED (replaced by tablet 827d186faaa140bdab2f87cf421ca64c) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:28.994823 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 86e33bd235c14042a578a4a1c70f11b7 with delete_type TABLET_DATA_DELETED (replaced by tablet 827d186faaa140bdab2f87cf421ca64c) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:28.995267 27693 raft_consensus.cc:357] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:28.995476 27693 raft_consensus.cc:383] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:28.995523 27693 raft_consensus.cc:738] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.006290 27693 consensus_queue.cc:260] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.006989 27693 ts_tablet_manager.cc:1428] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250903 20:05:29.010990 27693 tablet_bootstrap.cc:492] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:29.015283 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f6ddd620f29a4333a8f71f99316e1a8d" } modify_external_catalogs: true
I20250903 20:05:29.015486 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f6ddd620f29a4333a8f71f99316e1a8d" } modify_external_catalogs: true
I20250903 20:05:29.017532 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet f9709dfd18654c5bb071e39da7c8889f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:17 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:29.019426 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1de90214f1cd4c43a4c54befcb77e3fe" } new_table_name: "default.Table_9c69923fa15e405690f9dd166a271493" modify_external_catalogs: true
I20250903 20:05:29.021142 27687 ts_tablet_manager.cc:1918] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:29.021255 27687 log.cc:1199] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/961ade36f7d14eba91bb16d5eae2e254
I20250903 20:05:29.021600 27687 ts_tablet_manager.cc:1939] T 961ade36f7d14eba91bb16d5eae2e254 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
W20250903 20:05:29.022688 28809 log.cc:927] Time spent T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20250903 20:05:29.022742 27066 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 5d6671c904e44db99a8ce7d00e29d9fd
I20250903 20:05:29.023464 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 9766cea9d5bb407a9900c7e3bb4d2641
I20250903 20:05:29.024456 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet 9766cea9d5bb407a9900c7e3bb4d2641 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:29.025616 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_c51dfa63afbc4a9aabd9a43355859503"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:29.026408 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c51dfa63afbc4a9aabd9a43355859503 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:29.028632 28568 consensus_queue.cc:1035] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:29.030920 28761 raft_consensus.cc:491] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250903 20:05:29.031050 27184 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 260332us
I20250903 20:05:29.031250 28761 raft_consensus.cc:513] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:29.031472 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet c6d67cd0c0304c77ba571b414878dad2 (table default.Table_a540eea45dfb46ec92b02a3f9a2b3aed [id=5fe055478b724744a65fcaa2f25bef0d]) successfully deleted
I20250903 20:05:29.032198 28761 leader_election.cc:290] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:29.032421 27219 tablet_service.cc:1515] Processing DeleteTablet for tablet 4739f10128d64e2c986257b1d9deb90f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:29.033402 27219 tablet_service.cc:1515] Processing DeleteTablet for tablet b3bb55085af044ec8bb2368c845e7404 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:29.033728 27219 tablet_service.cc:1515] Processing DeleteTablet for tablet de20d18d7f264811a0631aa1144aef2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:29.033926 27462 leader_election.cc:343] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 17e0864a32d5452d84ac470b55fbaaf4
I20250903 20:05:29.034022 27219 tablet_service.cc:1515] Processing DeleteTablet for tablet 5962685d79044081a75532e19bfa2bce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:29.034713 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 86e33bd235c14042a578a4a1c70f11b7 with delete_type TABLET_DATA_DELETED (replaced by tablet 827d186faaa140bdab2f87cf421ca64c) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:29.034790 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet 13496efe34944b7280e7a684f180e6e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:29.035619 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet 9766cea9d5bb407a9900c7e3bb4d2641 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:29.036381 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 714) took 5257 ms. Trace:
I20250903 20:05:29.036469 27685 rpcz_store.cc:276] 0903 20:05:23.778455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:23.779102 (+ 647us) service_pool.cc:225] Handling call
0903 20:05:28.845922 (+5066820us) ts_tablet_manager.cc:1145] Deleting tablet fe902f1b58924be1ac3a5585413f7789
0903 20:05:28.845930 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:28.945553 (+ 99623us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:28.948557 (+ 3004us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:29.036361 (+ 87804us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5066709}
I20250903 20:05:29.036789 27685 tablet_replica.cc:331] T e78e6d212daf4589a6b4b65711e356cb P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:29.036975 27685 raft_consensus.cc:2241] T e78e6d212daf4589a6b4b65711e356cb P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:29.037158 27685 raft_consensus.cc:2270] T e78e6d212daf4589a6b4b65711e356cb P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:29.037305 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90baf83357494a0aae325908c118b471" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:29.037909 27377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0a9b7770432942da9373d5100c71bd6a" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:29.038067 27377 raft_consensus.cc:3058] T 0a9b7770432942da9373d5100c71bd6a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:29.038465 27685 ts_tablet_manager.cc:1905] T e78e6d212daf4589a6b4b65711e356cb P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:29.039783 27693 tablet_bootstrap.cc:654] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.040518 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 9766cea9d5bb407a9900c7e3bb4d2641 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:29.040787 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "673c45f3f2a14da9bd306011d396eaca" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:29.041003 27386 raft_consensus.cc:2391] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 89f8f0ee68b940db9ce250c0604f934a in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
I20250903 20:05:29.041998 27692 tablet_bootstrap.cc:654] T 8a223f75045345ae833f52bcc8f239f9 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.043648 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet bf459db3327640f88574fca7b5f2ca98
I20250903 20:05:29.045055 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet bf459db3327640f88574fca7b5f2ca98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:29 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:29.046355 27383 raft_consensus.cc:2466] T a72b85dfb34047528b2b28ff0d40f828 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:29.051354 28746 raft_consensus.cc:513] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.051991 28746 leader_election.cc:290] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:29.050177 28710 raft_consensus.cc:2747] T 1bad3c509460422396a17a3c4ae3d1e8 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:29.052409 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 857) took 2038 ms. Trace:
I20250903 20:05:29.052503 27687 rpcz_store.cc:276] 0903 20:05:27.013536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.013584 (+ 48us) service_pool.cc:225] Handling call
0903 20:05:28.472840 (+1459256us) ts_tablet_manager.cc:1145] Deleting tablet 961ade36f7d14eba91bb16d5eae2e254
0903 20:05:28.472847 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:28.815453 (+342606us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:29.021011 (+205558us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:29.052384 (+ 31373us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1458905}
I20250903 20:05:29.053112 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c27464eb362a44268d9c085034dd3ff4" } new_table_name: "default.Table_dc6f6f88ae4b4e129dc155a6d583fd9d" modify_external_catalogs: true
I20250903 20:05:29.053161 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_8a37507f7b5a4aa6867f2209f9c94680"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:29.053298 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_711e173c8a754e2b850b887bdb54e5ed" } new_table_name: "default.Table_24677e93e1624ada977cc29bf0c8f76d" modify_external_catalogs: true
W20250903 20:05:29.053905 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a37507f7b5a4aa6867f2209f9c94680 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:29.053944 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 961ade36f7d14eba91bb16d5eae2e254 (table default.Table_459903503b5148d09d3cb8f2f5519fab [id=3c69612aaf164dc899dcc252d85636f6]) successfully deleted
I20250903 20:05:29.055328 27057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3aecc9b035a24287af94e9c31d5d0714" } new_table_name: "default.Table_f2505e03caaa40c4b17d2e60b0196843" modify_external_catalogs: true
I20250903 20:05:29.056978 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet bf459db3327640f88574fca7b5f2ca98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:29 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:29.061197 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17e0864a32d5452d84ac470b55fbaaf4" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:29.061419 27240 raft_consensus.cc:2466] T 17e0864a32d5452d84ac470b55fbaaf4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:29.061621 27523 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "466e501e424e46bc98e8bbc64f97005e" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:29.061875 27462 leader_election.cc:304] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
W20250903 20:05:29.062290 27183 leader_election.cc:343] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:29.062377 28710 raft_consensus.cc:2802] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:29.062453 28710 raft_consensus.cc:491] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:29.062521 28710 raft_consensus.cc:3058] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:29.062537 28746 raft_consensus.cc:491] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:29.062647 28746 raft_consensus.cc:513] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.063210 28746 leader_election.cc:290] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:29.066313 27523 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e0347eab6c8642beabdc2d7e7bbfdc9b" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:29.073499 28371 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" instance_seqno: 1756929907080274) as {username='slave'} at 127.26.65.66:56753; Asking this server to re-register.
I20250903 20:05:29.067193 27523 raft_consensus.cc:2466] T e0347eab6c8642beabdc2d7e7bbfdc9b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:29.074962 27183 leader_election.cc:304] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:29.075273 28522 raft_consensus.cc:2802] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:29.075366 27433 heartbeater.cc:461] Registering TS with master...
I20250903 20:05:29.075502 28522 raft_consensus.cc:491] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:29.075639 28522 raft_consensus.cc:3058] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:29.076011 28568 consensus_queue.cc:1035] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:29.088972 27687 tablet_replica.cc:331] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:29.089237 27687 raft_consensus.cc:2241] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:29.091341 27692 tablet_bootstrap.cc:492] T 8a223f75045345ae833f52bcc8f239f9 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:29.091615 27692 ts_tablet_manager.cc:1397] T 8a223f75045345ae833f52bcc8f239f9 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.160s user 0.004s sys 0.001s
I20250903 20:05:29.092443 27687 raft_consensus.cc:2270] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:29.092504 27692 raft_consensus.cc:357] T 8a223f75045345ae833f52bcc8f239f9 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.092842 27692 raft_consensus.cc:383] T 8a223f75045345ae833f52bcc8f239f9 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.092941 27692 raft_consensus.cc:738] T 8a223f75045345ae833f52bcc8f239f9 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.093250 27692 consensus_queue.cc:260] T 8a223f75045345ae833f52bcc8f239f9 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.093894 27692 ts_tablet_manager.cc:1428] T 8a223f75045345ae833f52bcc8f239f9 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:29.094291 27692 tablet_bootstrap.cc:492] T f2629f46704e4a0fa7922533928ca2ed P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:29.094668 27687 ts_tablet_manager.cc:1905] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:29.101727 28568 consensus_queue.cc:1035] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:29.104557 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet fe902f1b58924be1ac3a5585413f7789 (table default.Table_1840897b7e7f4f1e8052d16ca5e77a4f [id=78785eeb2c464e6d9f0cc5c8792376ec]) successfully deleted
I20250903 20:05:29.105697 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "466e501e424e46bc98e8bbc64f97005e" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:29.105907 27383 raft_consensus.cc:3058] T 466e501e424e46bc98e8bbc64f97005e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:29.106892 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e0347eab6c8642beabdc2d7e7bbfdc9b" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:29.107086 27386 raft_consensus.cc:2466] T e0347eab6c8642beabdc2d7e7bbfdc9b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:29.123704 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet bf459db3327640f88574fca7b5f2ca98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:29 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:29.124101 27184 leader_election.cc:343] T 90baf83357494a0aae325908c118b471 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:29.124274 27184 leader_election.cc:304] T 90baf83357494a0aae325908c118b471 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:29.124534 28808 raft_consensus.cc:2747] T 90baf83357494a0aae325908c118b471 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:29.124812 27184 leader_election.cc:304] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:29.125025 28808 raft_consensus.cc:2747] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250903 20:05:29.125181 27184 leader_election.cc:304] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:29.125425 28808 raft_consensus.cc:2802] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:29.125645 28808 raft_consensus.cc:695] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:29.126108 28808 consensus_queue.cc:237] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:29.145246 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 908e748d08f74ed2a3071312c95ec07c consensus state has no leader
I20250903 20:05:29.147847 27692 tablet_bootstrap.cc:654] T f2629f46704e4a0fa7922533928ca2ed P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.148962 27684 ts_tablet_manager.cc:1918] T cd6d6bb514134bdf8e3bc0aaad180732 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:29.149077 27684 log.cc:1199] T cd6d6bb514134bdf8e3bc0aaad180732 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/cd6d6bb514134bdf8e3bc0aaad180732
I20250903 20:05:29.149374 27684 ts_tablet_manager.cc:1939] T cd6d6bb514134bdf8e3bc0aaad180732 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:29.150895 28710 raft_consensus.cc:513] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:29.151656 28710 leader_election.cc:290] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:29.154258 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 807) took 3010 ms. Trace:
I20250903 20:05:29.154379 27684 rpcz_store.cc:276] 0903 20:05:26.143761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.145373 (+ 1612us) service_pool.cc:225] Handling call
0903 20:05:28.944569 (+2799196us) ts_tablet_manager.cc:1145] Deleting tablet cd6d6bb514134bdf8e3bc0aaad180732
0903 20:05:28.944582 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:29.052297 (+107715us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:29.148840 (+ 96543us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:29.154229 (+ 5389us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2798994}
I20250903 20:05:29.154742 27684 tablet_replica.cc:331] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:29.154930 27684 raft_consensus.cc:2241] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:29.155333 27684 raft_consensus.cc:2270] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:29.155938 27684 ts_tablet_manager.cc:1905] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:29.156080 27377 raft_consensus.cc:2466] T 0a9b7770432942da9373d5100c71bd6a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:29.158427 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet cd6d6bb514134bdf8e3bc0aaad180732 (table default.Table_ef30e0f336384b50bf8293fe525252cf [id=b2f5b8646204428993ce29cf7b2e4caa]) successfully deleted
W20250903 20:05:29.158692 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eb447d66471a47d0beb03493742a0a18 consensus state has no leader
I20250903 20:05:29.158773 28780 raft_consensus.cc:491] T a9f43781299e47a8aecc37a0b73d7a64 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:29.158921 28780 raft_consensus.cc:513] T a9f43781299e47a8aecc37a0b73d7a64 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.159538 28780 leader_election.cc:290] T a9f43781299e47a8aecc37a0b73d7a64 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:29.160107 27523 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9f43781299e47a8aecc37a0b73d7a64" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:29.161038 27462 leader_election.cc:343] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 17e0864a32d5452d84ac470b55fbaaf4
W20250903 20:05:29.161629 27183 leader_election.cc:343] T a9f43781299e47a8aecc37a0b73d7a64 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:29.162304 27377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9f43781299e47a8aecc37a0b73d7a64" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:29.162393 27184 leader_election.cc:304] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:29.163283 27184 leader_election.cc:343] T a9f43781299e47a8aecc37a0b73d7a64 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:29.163409 27184 leader_election.cc:304] T a9f43781299e47a8aecc37a0b73d7a64 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:29.163664 28746 raft_consensus.cc:2802] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:29.163796 28746 raft_consensus.cc:695] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:29.164065 28746 consensus_queue.cc:237] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:29.164898 27066 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 31edb50c250f42739f82cdaaf1525266 consensus state has no leader
I20250903 20:05:29.165000 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_77f24f93527744b5b55d7c2243ef8774"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:29.165225 27066 master_service.cc:948] ReplaceTablet: received request to replace tablet 02b3e262dd7841dfbb0520377eb94743 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:29.165279 28655 raft_consensus.cc:2747] T a9f43781299e47a8aecc37a0b73d7a64 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:29.165586 27687 ts_tablet_manager.cc:1918] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:29.165720 27687 log.cc:1199] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/33fb6b357bda43e7abeaed5634183ef6
W20250903 20:05:29.165724 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77f24f93527744b5b55d7c2243ef8774 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:29.165920 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6181f8539fd34670ad894eae5e3ac4d3" } new_table_name: "default.Table_6b3c55139f2f4a4f8129be9b27c7ae96" modify_external_catalogs: true
I20250903 20:05:29.166150 27687 ts_tablet_manager.cc:1939] T 33fb6b357bda43e7abeaed5634183ef6 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:29.167013 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a223f75045345ae833f52bcc8f239f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:29.168605 28371 ts_manager.cc:194] Registered new tserver with Master: 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:29.168577 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_490a5f122961450f90ac24f19af39dac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:29.168733 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_d7af80d1b68748c99d85f6914f57b251"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:29.168970 27239 raft_consensus.cc:2466] T 20c25c84c6fb4609a1964a4ff3c526da P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
W20250903 20:05:29.169500 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_490a5f122961450f90ac24f19af39dac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:29.171015 27353 tablet_service.cc:1468] Processing CreateTablet for tablet a4eac88097954730a735f555c2bf7b0f (DEFAULT_TABLE table=default.Table_3be712dee7574cd4bdc030111a6d1f1a [id=1f0fc7b4c128427c9f4a323051630929]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:29.171491 27353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4eac88097954730a735f555c2bf7b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:29.163475 27059 catalog_manager.cc:5582] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a reported cstate change: term changed from 0 to 1, leader changed from <none> to 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65). New cstate: current_term: 1 leader_uuid: "89f8f0ee68b940db9ce250c0604f934a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: HEALTHY } } }
I20250903 20:05:29.173004 27059 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 3b3bfc63aa0b4e56bec189e834dc6a8d (table default.Table_9b20469406c94986ab496ea3e678b586 [id=6d6f5e8cadf9445d8695c075cfabb2ea]). Expected version 1 got 0
W20250903 20:05:29.172993 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7af80d1b68748c99d85f6914f57b251 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:29.175336 27495 tablet_service.cc:1468] Processing CreateTablet for tablet a4eac88097954730a735f555c2bf7b0f (DEFAULT_TABLE table=default.Table_3be712dee7574cd4bdc030111a6d1f1a [id=1f0fc7b4c128427c9f4a323051630929]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:29.175385 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17e0864a32d5452d84ac470b55fbaaf4" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:29.175599 27240 raft_consensus.cc:3058] T 17e0864a32d5452d84ac470b55fbaaf4 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:29.177040 27057 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4e34d6d5d68e4395aed78821ea077c1b consensus state has no leader
I20250903 20:05:29.178689 27321 leader_election.cc:304] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:29.179164 28751 raft_consensus.cc:2802] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:29.179282 28751 raft_consensus.cc:695] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:29.179522 28751 consensus_queue.cc:237] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.181422 27695 tablet_bootstrap.cc:492] T f9709dfd18654c5bb071e39da7c8889f P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:29.181362 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 8a75296d50cb46ffae87800613dc9a49 (DEFAULT_TABLE table=default.Table_dcc32881015d4f1596b05098a9f86782 [id=5d39188d242741a081cee0fe1904c02d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:29.181401 27499 tablet_service.cc:1468] Processing CreateTablet for tablet 8a75296d50cb46ffae87800613dc9a49 (DEFAULT_TABLE table=default.Table_dcc32881015d4f1596b05098a9f86782 [id=5d39188d242741a081cee0fe1904c02d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:29.181398 27348 tablet_service.cc:1468] Processing CreateTablet for tablet 60337a8e50ef4a26bbcbe32f83c5d2b5 (DEFAULT_TABLE table=default.Table_534372788132478792ce2f6e8cc1ae3c [id=0c6acf68ada9439fbc150e44a90ec18e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:29.181560 27695 ts_tablet_manager.cc:1397] T f9709dfd18654c5bb071e39da7c8889f P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.214s user 0.003s sys 0.003s
I20250903 20:05:29.181778 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a75296d50cb46ffae87800613dc9a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:29.181816 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 60337a8e50ef4a26bbcbe32f83c5d2b5 (DEFAULT_TABLE table=default.Table_534372788132478792ce2f6e8cc1ae3c [id=0c6acf68ada9439fbc150e44a90ec18e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:29.181871 27489 tablet_service.cc:1468] Processing CreateTablet for tablet fab456384e7242779690eb19bf41afac (DEFAULT_TABLE table=default.Table_d40b242552784e45a5de2341a25174fc [id=c2c68e2d20b740899cbd07b89f1d62c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:29.182314 27352 tablet_service.cc:1468] Processing CreateTablet for tablet fab456384e7242779690eb19bf41afac (DEFAULT_TABLE table=default.Table_d40b242552784e45a5de2341a25174fc [id=c2c68e2d20b740899cbd07b89f1d62c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:29.182454 27695 raft_consensus.cc:357] T f9709dfd18654c5bb071e39da7c8889f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.182619 27695 raft_consensus.cc:383] T f9709dfd18654c5bb071e39da7c8889f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.182662 27695 raft_consensus.cc:738] T f9709dfd18654c5bb071e39da7c8889f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.182904 27695 consensus_queue.cc:260] T f9709dfd18654c5bb071e39da7c8889f P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.182986 27350 tablet_service.cc:1468] Processing CreateTablet for tablet 38de36364c964546ac8dee8c468ad2aa (DEFAULT_TABLE table=default.Table_3ea0dde685ef4d89b831c292658d9c51 [id=26c338e325ac43d0bc7b5b34d7d8ca44]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:29.183463 27695 ts_tablet_manager.cc:1428] T f9709dfd18654c5bb071e39da7c8889f P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:29.183738 27695 tablet_bootstrap.cc:492] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:29.184607 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 38de36364c964546ac8dee8c468ad2aa (DEFAULT_TABLE table=default.Table_3ea0dde685ef4d89b831c292658d9c51 [id=26c338e325ac43d0bc7b5b34d7d8ca44]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:29.185108 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fab456384e7242779690eb19bf41afac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:29.188082 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_f6fb411b2d7949b4a424655ba7ff4921"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:29.189065 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6fb411b2d7949b4a424655ba7ff4921 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:29.190028 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38de36364c964546ac8dee8c468ad2aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:29.193559 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60337a8e50ef4a26bbcbe32f83c5d2b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:29.193918 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ee99c37551924b8fb11237c20dc777de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:29.194643 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee99c37551924b8fb11237c20dc777de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:29.198727 27693 tablet_bootstrap.cc:492] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:29.198904 27693 ts_tablet_manager.cc:1397] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.188s user 0.007s sys 0.011s
I20250903 20:05:29.199676 27693 raft_consensus.cc:357] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.199852 27693 raft_consensus.cc:383] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.200052 27693 raft_consensus.cc:738] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.200353 27693 consensus_queue.cc:260] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.200263 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_7b59b780063640dd8ef72c72e5e5cf7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:29.200933 28770 consensus_queue.cc:1035] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:29.201206 27693 ts_tablet_manager.cc:1428] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:29.201484 27693 tablet_bootstrap.cc:492] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
W20250903 20:05:29.202296 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b59b780063640dd8ef72c72e5e5cf7a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:29.203240 27693 tablet_bootstrap.cc:654] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.203895 27349 tablet_service.cc:1468] Processing CreateTablet for tablet d67c1780ea644170a8126da5bbfa9ced (DEFAULT_TABLE table=default.Table_15a4e55b16784ec4b0c3115892bcf650 [id=23f3eae6301440eca3354fc2d1e8f1cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:29.204334 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d67c1780ea644170a8126da5bbfa9ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:29.206559 27359 tablet_service.cc:1468] Processing CreateTablet for tablet 5985042cdd274b0ca5815e697e035a45 (DEFAULT_TABLE table=default.Table_ecc03aa0a01940e284383daf011ddc7b [id=267a80e8a2664ff4bb4d346997e0d2d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:29.206945 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5985042cdd274b0ca5815e697e035a45. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:29.207109 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 5985042cdd274b0ca5815e697e035a45 (DEFAULT_TABLE table=default.Table_ecc03aa0a01940e284383daf011ddc7b [id=267a80e8a2664ff4bb4d346997e0d2d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:29.207594 27494 tablet_service.cc:1468] Processing CreateTablet for tablet d67c1780ea644170a8126da5bbfa9ced (DEFAULT_TABLE table=default.Table_15a4e55b16784ec4b0c3115892bcf650 [id=23f3eae6301440eca3354fc2d1e8f1cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:29.216868 27693 tablet_bootstrap.cc:492] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:29.217608 27693 ts_tablet_manager.cc:1397] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.006s
I20250903 20:05:29.218652 27693 raft_consensus.cc:357] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.219720 27695 tablet_bootstrap.cc:654] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.219894 27693 raft_consensus.cc:383] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.220120 27693 raft_consensus.cc:738] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.220578 27693 consensus_queue.cc:260] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.221869 27693 ts_tablet_manager.cc:1428] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250903 20:05:29.222358 27693 tablet_bootstrap.cc:492] T 607c9a9cbc5540e5bd0fa436fcba564c P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:29.224458 27693 tablet_bootstrap.cc:654] T 607c9a9cbc5540e5bd0fa436fcba564c P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.233958 27693 tablet_bootstrap.cc:492] T 607c9a9cbc5540e5bd0fa436fcba564c P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:29.234835 27693 ts_tablet_manager.cc:1397] T 607c9a9cbc5540e5bd0fa436fcba564c P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.005s
I20250903 20:05:29.236003 27693 raft_consensus.cc:357] T 607c9a9cbc5540e5bd0fa436fcba564c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.237315 27693 raft_consensus.cc:383] T 607c9a9cbc5540e5bd0fa436fcba564c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.237558 27693 raft_consensus.cc:738] T 607c9a9cbc5540e5bd0fa436fcba564c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.238029 27693 consensus_queue.cc:260] T 607c9a9cbc5540e5bd0fa436fcba564c P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.239176 27693 ts_tablet_manager.cc:1428] T 607c9a9cbc5540e5bd0fa436fcba564c P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250903 20:05:29.240077 27693 tablet_bootstrap.cc:492] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:29.243018 27693 tablet_bootstrap.cc:654] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.245095 27383 raft_consensus.cc:2466] T 466e501e424e46bc98e8bbc64f97005e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:29.245884 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4eac88097954730a735f555c2bf7b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:29.246389 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a75296d50cb46ffae87800613dc9a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:29.246709 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 858) took 2232 ms. Trace:
I20250903 20:05:29.246843 27489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fab456384e7242779690eb19bf41afac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:29.246798 27687 rpcz_store.cc:276] 0903 20:05:27.014060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.014102 (+ 42us) service_pool.cc:225] Handling call
0903 20:05:29.088898 (+2074796us) ts_tablet_manager.cc:1145] Deleting tablet 33fb6b357bda43e7abeaed5634183ef6
0903 20:05:29.088915 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:29.149614 (+ 60699us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:29.165436 (+ 15822us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:29.246684 (+ 81248us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2074559}
I20250903 20:05:29.247251 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 859) took 2216 ms. Trace:
I20250903 20:05:29.247319 27687 rpcz_store.cc:276] 0903 20:05:27.030343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.030394 (+ 51us) service_pool.cc:225] Handling call
0903 20:05:29.247102 (+2216708us) ts_tablet_manager.cc:1145] Deleting tablet e0b9172ebb484382adb86e597f5cbc3b
0903 20:05:29.247112 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:29.247233 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2216469}
I20250903 20:05:29.247632 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 860) took 2216 ms. Trace:
I20250903 20:05:29.247692 27687 rpcz_store.cc:276] 0903 20:05:27.030772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.030814 (+ 42us) service_pool.cc:225] Handling call
0903 20:05:29.247527 (+2216713us) ts_tablet_manager.cc:1145] Deleting tablet 553443eda16c44fa84b1db22b1363808
0903 20:05:29.247535 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:29.247616 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2216535}
I20250903 20:05:29.248036 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 861) took 2216 ms. Trace:
I20250903 20:05:29.248113 27687 rpcz_store.cc:276] 0903 20:05:27.031128 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.031170 (+ 42us) service_pool.cc:225] Handling call
0903 20:05:29.247899 (+2216729us) ts_tablet_manager.cc:1145] Deleting tablet 44fef873a0644dd988322847f333ec95
0903 20:05:29.247910 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:29.248019 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2216540}
I20250903 20:05:29.248406 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 862) took 2216 ms. Trace:
I20250903 20:05:29.248462 27687 rpcz_store.cc:276] 0903 20:05:27.031526 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.031565 (+ 39us) service_pool.cc:225] Handling call
0903 20:05:29.248310 (+2216745us) ts_tablet_manager.cc:1145] Deleting tablet ea3f82f4e4244219ab0919f3e341aead
0903 20:05:29.248318 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:29.248394 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2216609}
I20250903 20:05:29.248695 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 863) took 2216 ms. Trace:
I20250903 20:05:29.248742 27687 rpcz_store.cc:276] 0903 20:05:27.031899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.031939 (+ 40us) service_pool.cc:225] Handling call
0903 20:05:29.248623 (+2216684us) ts_tablet_manager.cc:1145] Deleting tablet 81852d8e06ae4c6bb715a88d8c398bfc
0903 20:05:29.248629 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:29.248685 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2216399}
I20250903 20:05:29.249002 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 864) took 2034 ms. Trace:
I20250903 20:05:29.249065 27687 rpcz_store.cc:276] 0903 20:05:27.214447 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.214512 (+ 65us) service_pool.cc:225] Handling call
0903 20:05:29.248902 (+2034390us) ts_tablet_manager.cc:1145] Deleting tablet ce3be9d44147420d953e716a347352c9
0903 20:05:29.248910 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:29.248989 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2034200}
I20250903 20:05:29.249280 27687 tablet_replica.cc:331] T 53881ad30bc843839008ace4012a3f70 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:29.249460 27687 raft_consensus.cc:2241] T 53881ad30bc843839008ace4012a3f70 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:29.249620 27687 raft_consensus.cc:2270] T 53881ad30bc843839008ace4012a3f70 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:29.249998 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60337a8e50ef4a26bbcbe32f83c5d2b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:29.250777 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 33fb6b357bda43e7abeaed5634183ef6 (table default.Table_7a6b693ddf594ff6821359a4a7fbee01 [id=fb19a730ea6343fbba98571261d7a894]) successfully deleted
I20250903 20:05:29.251158 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38de36364c964546ac8dee8c468ad2aa. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:29.251156 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet e0b9172ebb484382adb86e597f5cbc3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0b9172ebb484382adb86e597f5cbc3b already in progress: creating tablet
W20250903 20:05:29.251399 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 553443eda16c44fa84b1db22b1363808 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 553443eda16c44fa84b1db22b1363808 already in progress: creating tablet
I20250903 20:05:29.251638 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5985042cdd274b0ca5815e697e035a45. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:29.251631 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 44fef873a0644dd988322847f333ec95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44fef873a0644dd988322847f333ec95 already in progress: creating tablet
W20250903 20:05:29.251837 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet ea3f82f4e4244219ab0919f3e341aead because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea3f82f4e4244219ab0919f3e341aead already in progress: creating tablet
I20250903 20:05:29.252097 27494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d67c1780ea644170a8126da5bbfa9ced. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:29.252091 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 81852d8e06ae4c6bb715a88d8c398bfc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81852d8e06ae4c6bb715a88d8c398bfc already in progress: creating tablet
I20250903 20:05:29.252269 28780 consensus_queue.cc:1035] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250903 20:05:29.252355 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet ce3be9d44147420d953e716a347352c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce3be9d44147420d953e716a347352c9 already in progress: creating tablet
I20250903 20:05:29.256096 27695 tablet_bootstrap.cc:492] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:29.256250 27695 ts_tablet_manager.cc:1397] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.073s user 0.002s sys 0.001s
I20250903 20:05:29.257431 28780 raft_consensus.cc:491] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:29.257563 28780 raft_consensus.cc:513] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.258173 28780 leader_election.cc:290] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:29.258415 27695 raft_consensus.cc:357] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.258553 27685 ts_tablet_manager.cc:1918] T e78e6d212daf4589a6b4b65711e356cb P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:29.258682 27685 log.cc:1199] T e78e6d212daf4589a6b4b65711e356cb P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/e78e6d212daf4589a6b4b65711e356cb
I20250903 20:05:29.258813 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cc6cf579e2642e99fefed723b0ee900" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:29.258999 27522 raft_consensus.cc:2466] T 7cc6cf579e2642e99fefed723b0ee900 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:29.259055 27685 ts_tablet_manager.cc:1939] T e78e6d212daf4589a6b4b65711e356cb P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:29.259717 27183 leader_election.cc:304] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:29.258570 27695 raft_consensus.cc:383] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.260071 28780 raft_consensus.cc:2802] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250903 20:05:29.260114 27695 raft_consensus.cc:738] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.260129 28780 raft_consensus.cc:491] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:29.260355 28780 raft_consensus.cc:3058] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Advancing to term 2
I20250903 20:05:29.260350 27695 consensus_queue.cc:260] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.260711 27692 tablet_bootstrap.cc:492] T f2629f46704e4a0fa7922533928ca2ed P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:29.260864 27692 ts_tablet_manager.cc:1397] T f2629f46704e4a0fa7922533928ca2ed P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.167s user 0.010s sys 0.019s
I20250903 20:05:29.260910 27695 ts_tablet_manager.cc:1428] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250903 20:05:29.261272 27695 tablet_bootstrap.cc:492] T 5f27f525fdab407280aa23df1ae71b43 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:29.261639 27692 raft_consensus.cc:357] T f2629f46704e4a0fa7922533928ca2ed P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.261802 27692 raft_consensus.cc:383] T f2629f46704e4a0fa7922533928ca2ed P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.261852 27692 raft_consensus.cc:738] T f2629f46704e4a0fa7922533928ca2ed P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.262074 27692 consensus_queue.cc:260] T f2629f46704e4a0fa7922533928ca2ed P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.250206 27687 ts_tablet_manager.cc:1905] T 53881ad30bc843839008ace4012a3f70 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:29.286423 27695 tablet_bootstrap.cc:654] T 5f27f525fdab407280aa23df1ae71b43 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.288013 27240 raft_consensus.cc:2466] T 17e0864a32d5452d84ac470b55fbaaf4 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:29.289225 27692 ts_tablet_manager.cc:1428] T f2629f46704e4a0fa7922533928ca2ed P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.028s user 0.000s sys 0.003s
I20250903 20:05:29.289544 27692 tablet_bootstrap.cc:492] T 466e501e424e46bc98e8bbc64f97005e P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:29.288522 27462 leader_election.cc:304] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:29.291258 28710 raft_consensus.cc:2802] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:29.291399 28710 raft_consensus.cc:695] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:29.291669 28710 consensus_queue.cc:237] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:29.298192 27692 tablet_bootstrap.cc:654] T 466e501e424e46bc98e8bbc64f97005e P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.299482 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 53b00774b745462091321ea47fcc2114
I20250903 20:05:29.301124 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet 53b00774b745462091321ea47fcc2114 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:29.301501 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 53b00774b745462091321ea47fcc2114 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:29.306228 28568 raft_consensus.cc:491] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:29.306842 28568 raft_consensus.cc:513] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.309180 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9709dfd18654c5bb071e39da7c8889f" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:29.309485 27240 raft_consensus.cc:2466] T f9709dfd18654c5bb071e39da7c8889f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:29.310077 27321 leader_election.cc:304] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:29.310204 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9709dfd18654c5bb071e39da7c8889f" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:29.310407 28567 raft_consensus.cc:2802] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:29.310478 28567 raft_consensus.cc:491] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:29.310539 28567 raft_consensus.cc:3058] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:29.310894 27320 leader_election.cc:343] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:29.314184 27692 tablet_bootstrap.cc:492] T 466e501e424e46bc98e8bbc64f97005e P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:29.314184 27693 tablet_bootstrap.cc:492] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:29.314364 27692 ts_tablet_manager.cc:1397] T 466e501e424e46bc98e8bbc64f97005e P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.004s
I20250903 20:05:29.314548 27693 ts_tablet_manager.cc:1397] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.075s user 0.002s sys 0.007s
I20250903 20:05:29.314548 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 31186292b25f4c3cb23597f66dcbd282
I20250903 20:05:29.315178 27692 raft_consensus.cc:357] T 466e501e424e46bc98e8bbc64f97005e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.315359 27692 raft_consensus.cc:383] T 466e501e424e46bc98e8bbc64f97005e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.315411 27692 raft_consensus.cc:738] T 466e501e424e46bc98e8bbc64f97005e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.315476 27693 raft_consensus.cc:357] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.315626 27693 raft_consensus.cc:383] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.315678 27693 raft_consensus.cc:738] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.315905 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet 31186292b25f4c3cb23597f66dcbd282 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:29.316144 27692 consensus_queue.cc:260] T 466e501e424e46bc98e8bbc64f97005e P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.316411 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 31186292b25f4c3cb23597f66dcbd282 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:29.316540 27692 ts_tablet_manager.cc:1428] T 466e501e424e46bc98e8bbc64f97005e P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250903 20:05:29.315925 27693 consensus_queue.cc:260] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.316802 27692 tablet_bootstrap.cc:492] T 673c45f3f2a14da9bd306011d396eaca P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:29.317165 27693 ts_tablet_manager.cc:1428] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250903 20:05:29.317200 27695 tablet_bootstrap.cc:492] T 5f27f525fdab407280aa23df1ae71b43 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:29.317341 27695 ts_tablet_manager.cc:1397] T 5f27f525fdab407280aa23df1ae71b43 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.056s user 0.000s sys 0.003s
I20250903 20:05:29.317426 27693 tablet_bootstrap.cc:492] T f2629f46704e4a0fa7922533928ca2ed P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:29.318048 27695 raft_consensus.cc:357] T 5f27f525fdab407280aa23df1ae71b43 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.318198 27695 raft_consensus.cc:383] T 5f27f525fdab407280aa23df1ae71b43 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.318279 27695 raft_consensus.cc:738] T 5f27f525fdab407280aa23df1ae71b43 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.318518 27695 consensus_queue.cc:260] T 5f27f525fdab407280aa23df1ae71b43 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.318981 27695 ts_tablet_manager.cc:1428] T 5f27f525fdab407280aa23df1ae71b43 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250903 20:05:29.319262 27695 tablet_bootstrap.cc:492] T 9e98cd1d9bd246e182e2969d147e68bf P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:29.319960 27693 tablet_bootstrap.cc:654] T f2629f46704e4a0fa7922533928ca2ed P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:29.320009 27066 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 57943651b6374843ade249d11d6183cf consensus state has no leader
I20250903 20:05:29.320966 27695 tablet_bootstrap.cc:654] T 9e98cd1d9bd246e182e2969d147e68bf P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.322749 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_f9124043f4d84586858e6bdeb679df55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:29.323493 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9124043f4d84586858e6bdeb679df55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:29.325529 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_8b2d83116755450387b770b591314010"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:29.326119 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b2d83116755450387b770b591314010 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:29.327623 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f6658302f6a546d0a9a2348e78572b99" } modify_external_catalogs: true
I20250903 20:05:29.327827 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f6658302f6a546d0a9a2348e78572b99" } modify_external_catalogs: true
I20250903 20:05:29.328029 27693 tablet_bootstrap.cc:492] T f2629f46704e4a0fa7922533928ca2ed P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:29.328166 27693 ts_tablet_manager.cc:1397] T f2629f46704e4a0fa7922533928ca2ed P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20250903 20:05:29.328334 27692 tablet_bootstrap.cc:654] T 673c45f3f2a14da9bd306011d396eaca P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.329072 27693 raft_consensus.cc:357] T f2629f46704e4a0fa7922533928ca2ed P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.329332 27693 raft_consensus.cc:383] T f2629f46704e4a0fa7922533928ca2ed P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.329425 27693 raft_consensus.cc:738] T f2629f46704e4a0fa7922533928ca2ed P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.329355 27065 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b0a0a2dfe31e4961871f334372541d05" } modify_external_catalogs: true
I20250903 20:05:29.329552 27065 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b0a0a2dfe31e4961871f334372541d05" } modify_external_catalogs: true
I20250903 20:05:29.329751 27693 consensus_queue.cc:260] T f2629f46704e4a0fa7922533928ca2ed P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.330701 28568 leader_election.cc:290] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:29.332585 27693 ts_tablet_manager.cc:1428] T f2629f46704e4a0fa7922533928ca2ed P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250903 20:05:29.332926 27693 tablet_bootstrap.cc:492] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:29.334764 27693 tablet_bootstrap.cc:654] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.337852 27692 tablet_bootstrap.cc:492] T 673c45f3f2a14da9bd306011d396eaca P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:29.338022 27692 ts_tablet_manager.cc:1397] T 673c45f3f2a14da9bd306011d396eaca P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20250903 20:05:29.339073 27692 raft_consensus.cc:357] T 673c45f3f2a14da9bd306011d396eaca P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.339308 27692 raft_consensus.cc:383] T 673c45f3f2a14da9bd306011d396eaca P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.339360 27692 raft_consensus.cc:738] T 673c45f3f2a14da9bd306011d396eaca P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.339625 27692 consensus_queue.cc:260] T 673c45f3f2a14da9bd306011d396eaca P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.340130 27692 ts_tablet_manager.cc:1428] T 673c45f3f2a14da9bd306011d396eaca P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250903 20:05:29.340421 27692 tablet_bootstrap.cc:492] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:29.340780 28770 raft_consensus.cc:491] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:29.340902 28770 raft_consensus.cc:513] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.341570 28770 leader_election.cc:290] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:29.342119 27064 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_66babebee1c9462ea057c8489d43846d" } modify_external_catalogs: true
I20250903 20:05:29.342329 27064 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_66babebee1c9462ea057c8489d43846d" } modify_external_catalogs: true
I20250903 20:05:29.342417 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90baf83357494a0aae325908c118b471" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:29.343067 27320 leader_election.cc:343] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:29.343719 27693 tablet_bootstrap.cc:492] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:29.343883 27693 ts_tablet_manager.cc:1397] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.001s
I20250903 20:05:29.344483 27692 tablet_bootstrap.cc:654] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.345005 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90baf83357494a0aae325908c118b471" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:29.345227 27240 raft_consensus.cc:2466] T 90baf83357494a0aae325908c118b471 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:29.345695 27321 leader_election.cc:304] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:29.346477 27684 ts_tablet_manager.cc:1918] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:29.346577 27684 log.cc:1199] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/e78e6d212daf4589a6b4b65711e356cb
I20250903 20:05:29.346899 27684 ts_tablet_manager.cc:1939] T e78e6d212daf4589a6b4b65711e356cb P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:29.350509 28770 raft_consensus.cc:2802] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:29.350605 28770 raft_consensus.cc:491] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:29.350661 28770 raft_consensus.cc:3058] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:29.351118 28567 raft_consensus.cc:513] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.351732 28567 leader_election.cc:290] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:29.352396 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9709dfd18654c5bb071e39da7c8889f" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:29.352854 28655 raft_consensus.cc:491] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:29.352993 28655 raft_consensus.cc:513] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.353555 28655 leader_election.cc:290] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:29.359246 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2629f46704e4a0fa7922533928ca2ed" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:29.359447 27522 raft_consensus.cc:2466] T f2629f46704e4a0fa7922533928ca2ed P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:29.359957 27183 leader_election.cc:304] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
W20250903 20:05:29.362735 27320 leader_election.cc:343] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:29.365304 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 808) took 3221 ms. Trace:
I20250903 20:05:29.365425 27684 rpcz_store.cc:276] 0903 20:05:26.143853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.145615 (+ 1762us) service_pool.cc:225] Handling call
0903 20:05:29.154684 (+3009069us) ts_tablet_manager.cc:1145] Deleting tablet e78e6d212daf4589a6b4b65711e356cb
0903 20:05:29.154697 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:29.245543 (+ 90846us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:29.346346 (+100803us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:29.365273 (+ 18927us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3008933}
I20250903 20:05:29.365803 27684 tablet_replica.cc:331] T 283a3afed7414a958b2dc4176f00bb01 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:29.366029 27684 raft_consensus.cc:2241] T 283a3afed7414a958b2dc4176f00bb01 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:29.366919 27695 tablet_bootstrap.cc:492] T 9e98cd1d9bd246e182e2969d147e68bf P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:29.367004 28655 raft_consensus.cc:2802] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:29.367141 28655 raft_consensus.cc:491] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:29.367256 28655 raft_consensus.cc:3058] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:29.367051 27695 ts_tablet_manager.cc:1397] T 9e98cd1d9bd246e182e2969d147e68bf P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.048s user 0.001s sys 0.002s
I20250903 20:05:29.368548 27695 raft_consensus.cc:357] T 9e98cd1d9bd246e182e2969d147e68bf P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.368714 27695 raft_consensus.cc:383] T 9e98cd1d9bd246e182e2969d147e68bf P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.369410 27695 raft_consensus.cc:738] T 9e98cd1d9bd246e182e2969d147e68bf P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.369628 27695 consensus_queue.cc:260] T 9e98cd1d9bd246e182e2969d147e68bf P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.370057 27695 ts_tablet_manager.cc:1428] T 9e98cd1d9bd246e182e2969d147e68bf P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250903 20:05:29.370339 27695 tablet_bootstrap.cc:492] T 4af8c03c9d834405b4e25d35441db961 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:29.371672 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet e78e6d212daf4589a6b4b65711e356cb (table default.Table_7f07933eeaf8443384edb1e59396e36e [id=6b3f6e948024464fa28fe985534395ff]) successfully deleted
I20250903 20:05:29.371912 27695 tablet_bootstrap.cc:654] T 4af8c03c9d834405b4e25d35441db961 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.366191 27684 raft_consensus.cc:2270] T 283a3afed7414a958b2dc4176f00bb01 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:29.372979 27684 ts_tablet_manager.cc:1905] T 283a3afed7414a958b2dc4176f00bb01 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:29.370316 28710 raft_consensus.cc:491] T 8a223f75045345ae833f52bcc8f239f9 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:29.374380 28710 raft_consensus.cc:513] T 8a223f75045345ae833f52bcc8f239f9 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.374984 28710 leader_election.cc:290] T 8a223f75045345ae833f52bcc8f239f9 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:29.375826 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a223f75045345ae833f52bcc8f239f9" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:29.376793 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a223f75045345ae833f52bcc8f239f9" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:29.377082 27383 raft_consensus.cc:2391] T 8a223f75045345ae833f52bcc8f239f9 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 89f8f0ee68b940db9ce250c0604f934a in this term.
I20250903 20:05:29.377040 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_444ca50a57904488bdecef2ff5522be1" } new_table_name: "default.Table_679cf2a9355f4218b5edb4339e48dab7" modify_external_catalogs: true
I20250903 20:05:29.378074 27462 leader_election.cc:304] T 8a223f75045345ae833f52bcc8f239f9 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:29.379480 28710 raft_consensus.cc:3058] T 8a223f75045345ae833f52bcc8f239f9 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:29.382659 27687 ts_tablet_manager.cc:1918] T 53881ad30bc843839008ace4012a3f70 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:29.382781 27687 log.cc:1199] T 53881ad30bc843839008ace4012a3f70 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/53881ad30bc843839008ace4012a3f70
I20250903 20:05:29.383064 27687 ts_tablet_manager.cc:1939] T 53881ad30bc843839008ace4012a3f70 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:29.387423 27692 tablet_bootstrap.cc:492] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:29.387598 27692 ts_tablet_manager.cc:1397] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.001s
I20250903 20:05:29.388350 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 715) took 5609 ms. Trace:
I20250903 20:05:29.388449 27685 rpcz_store.cc:276] 0903 20:05:23.778545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:23.779461 (+ 916us) service_pool.cc:225] Handling call
0903 20:05:29.036736 (+5257275us) ts_tablet_manager.cc:1145] Deleting tablet e78e6d212daf4589a6b4b65711e356cb
0903 20:05:29.036748 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:29.154192 (+117444us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:29.258429 (+104237us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:29.388328 (+129899us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5257123}
I20250903 20:05:29.388826 27685 tablet_replica.cc:331] T 45d9b6ff520348628d19671127b9426c P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:29.389087 27685 raft_consensus.cc:2241] T 45d9b6ff520348628d19671127b9426c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:29.389087 27692 raft_consensus.cc:357] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.389254 27692 raft_consensus.cc:383] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.389251 27685 raft_consensus.cc:2270] T 45d9b6ff520348628d19671127b9426c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:29.389307 27692 raft_consensus.cc:738] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.389554 27692 consensus_queue.cc:260] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.389799 27685 ts_tablet_manager.cc:1905] T 45d9b6ff520348628d19671127b9426c P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:29.390071 27692 ts_tablet_manager.cc:1428] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:29.390383 27692 tablet_bootstrap.cc:492] T 90baf83357494a0aae325908c118b471 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:29.391151 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet e78e6d212daf4589a6b4b65711e356cb (table default.Table_7f07933eeaf8443384edb1e59396e36e [id=6b3f6e948024464fa28fe985534395ff]) successfully deleted
I20250903 20:05:29.391808 27184 leader_election.cc:304] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:29.391917 27692 tablet_bootstrap.cc:654] T 90baf83357494a0aae325908c118b471 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.393098 28596 raft_consensus.cc:491] T 7e2f98f760cf4df4bc2a10530e2157c6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:29.393224 28596 raft_consensus.cc:513] T 7e2f98f760cf4df4bc2a10530e2157c6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:29.393795 28596 leader_election.cc:290] T 7e2f98f760cf4df4bc2a10530e2157c6 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:29.394666 27377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e2f98f760cf4df4bc2a10530e2157c6" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:29.394737 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2629f46704e4a0fa7922533928ca2ed" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:29.394891 27386 raft_consensus.cc:2466] T f2629f46704e4a0fa7922533928ca2ed P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:29.395331 28745 raft_consensus.cc:2802] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:29.395455 28745 raft_consensus.cc:695] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:29.395639 27214 tablet_service.cc:1468] Processing CreateTablet for tablet a4eac88097954730a735f555c2bf7b0f (DEFAULT_TABLE table=default.Table_3be712dee7574cd4bdc030111a6d1f1a [id=1f0fc7b4c128427c9f4a323051630929]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:29.395743 28745 consensus_queue.cc:237] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.396062 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4eac88097954730a735f555c2bf7b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:29.396183 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 38de36364c964546ac8dee8c468ad2aa (DEFAULT_TABLE table=default.Table_3ea0dde685ef4d89b831c292658d9c51 [id=26c338e325ac43d0bc7b5b34d7d8ca44]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:29.396492 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38de36364c964546ac8dee8c468ad2aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:29.395640 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 60337a8e50ef4a26bbcbe32f83c5d2b5 (DEFAULT_TABLE table=default.Table_534372788132478792ce2f6e8cc1ae3c [id=0c6acf68ada9439fbc150e44a90ec18e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:29.398731 27221 tablet_service.cc:1468] Processing CreateTablet for tablet 5985042cdd274b0ca5815e697e035a45 (DEFAULT_TABLE table=default.Table_ecc03aa0a01940e284383daf011ddc7b [id=267a80e8a2664ff4bb4d346997e0d2d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:29.398907 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60337a8e50ef4a26bbcbe32f83c5d2b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:29.399048 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5985042cdd274b0ca5815e697e035a45. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:29.400547 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet 90baf83357494a0aae325908c118b471 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:29.400749 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d9276aa6946f41d7ac0fabc39338c23d" } new_table_name: "default.Table_6d2d1ff2291b43f8ae4c0a6a456f5014" modify_external_catalogs: true
I20250903 20:05:29.395843 27213 tablet_service.cc:1468] Processing CreateTablet for tablet fab456384e7242779690eb19bf41afac (DEFAULT_TABLE table=default.Table_d40b242552784e45a5de2341a25174fc [id=c2c68e2d20b740899cbd07b89f1d62c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:29.401212 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fab456384e7242779690eb19bf41afac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:29.398452 27225 tablet_service.cc:1468] Processing CreateTablet for tablet d67c1780ea644170a8126da5bbfa9ced (DEFAULT_TABLE table=default.Table_15a4e55b16784ec4b0c3115892bcf650 [id=23f3eae6301440eca3354fc2d1e8f1cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:29.402307 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 53b00774b745462091321ea47fcc2114 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:29.402474 27224 tablet_service.cc:1515] Processing DeleteTablet for tablet 31186292b25f4c3cb23597f66dcbd282 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:29.402554 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d67c1780ea644170a8126da5bbfa9ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:29.398471 27219 tablet_service.cc:1468] Processing CreateTablet for tablet 8a75296d50cb46ffae87800613dc9a49 (DEFAULT_TABLE table=default.Table_dcc32881015d4f1596b05098a9f86782 [id=5d39188d242741a081cee0fe1904c02d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:29.406780 27219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a75296d50cb46ffae87800613dc9a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:29.410375 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e2f98f760cf4df4bc2a10530e2157c6" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:29.394876 27377 raft_consensus.cc:2391] T 7e2f98f760cf4df4bc2a10530e2157c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 89f8f0ee68b940db9ce250c0604f934a in this term.
I20250903 20:05:29.411511 27462 leader_election.cc:304] T 7e2f98f760cf4df4bc2a10530e2157c6 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:29.411917 28817 raft_consensus.cc:3058] T 7e2f98f760cf4df4bc2a10530e2157c6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:29.434240 28522 raft_consensus.cc:513] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.445595 28710 raft_consensus.cc:2747] T 8a223f75045345ae833f52bcc8f239f9 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:29.446735 28770 raft_consensus.cc:513] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.448268 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90baf83357494a0aae325908c118b471" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
W20250903 20:05:29.448820 27320 leader_election.cc:343] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:29.448999 28770 leader_election.cc:290] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:29.450042 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 868) took 2200 ms. Trace:
I20250903 20:05:29.450160 27687 rpcz_store.cc:276] 0903 20:05:27.249187 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.249426 (+ 239us) service_pool.cc:225] Handling call
0903 20:05:29.249239 (+1999813us) ts_tablet_manager.cc:1145] Deleting tablet 53881ad30bc843839008ace4012a3f70
0903 20:05:29.249247 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:29.347208 (+ 97961us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:29.382521 (+ 35313us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:29.450023 (+ 67502us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1999560}
I20250903 20:05:29.450636 27687 tablet_replica.cc:331] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:29.450914 27687 raft_consensus.cc:2241] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:29.451427 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 53881ad30bc843839008ace4012a3f70 (table default.Table_2b6469f261a04e5d9cb69bf836d97734 [id=cedbbfab7d0e41369cafd9ce516f2dba]) successfully deleted
I20250903 20:05:29.451551 27687 raft_consensus.cc:2270] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:29.454241 27687 ts_tablet_manager.cc:1905] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:29.454528 28655 raft_consensus.cc:513] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.455489 28655 leader_election.cc:290] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:29.470566 28655 raft_consensus.cc:491] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:29.470830 28655 raft_consensus.cc:513] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.470954 27693 raft_consensus.cc:357] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.471174 27693 raft_consensus.cc:383] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.471283 27693 raft_consensus.cc:738] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.471596 28655 leader_election.cc:290] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:29.471580 27693 consensus_queue.cc:260] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.472383 27693 ts_tablet_manager.cc:1428] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.128s user 0.001s sys 0.001s
I20250903 20:05:29.472896 27693 tablet_bootstrap.cc:492] T 5f27f525fdab407280aa23df1ae71b43 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:29.474377 28522 leader_election.cc:290] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:29.476401 27693 tablet_bootstrap.cc:654] T 5f27f525fdab407280aa23df1ae71b43 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.482613 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9709dfd18654c5bb071e39da7c8889f" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:29.482823 27240 raft_consensus.cc:3058] T f9709dfd18654c5bb071e39da7c8889f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:29.483261 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90baf83357494a0aae325908c118b471" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:29.483777 27239 raft_consensus.cc:3058] T 90baf83357494a0aae325908c118b471 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:29.484586 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2629f46704e4a0fa7922533928ca2ed" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:29.484790 27386 raft_consensus.cc:3058] T f2629f46704e4a0fa7922533928ca2ed P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:29.487792 27695 tablet_bootstrap.cc:492] T 4af8c03c9d834405b4e25d35441db961 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:29.487942 27695 ts_tablet_manager.cc:1397] T 4af8c03c9d834405b4e25d35441db961 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.118s user 0.006s sys 0.005s
I20250903 20:05:29.488795 27695 raft_consensus.cc:357] T 4af8c03c9d834405b4e25d35441db961 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.488999 27695 raft_consensus.cc:383] T 4af8c03c9d834405b4e25d35441db961 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.489066 27695 raft_consensus.cc:738] T 4af8c03c9d834405b4e25d35441db961 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.489403 27695 consensus_queue.cc:260] T 4af8c03c9d834405b4e25d35441db961 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.489938 27695 ts_tablet_manager.cc:1428] T 4af8c03c9d834405b4e25d35441db961 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:29.490387 27692 tablet_bootstrap.cc:492] T 90baf83357494a0aae325908c118b471 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:29.490586 27692 ts_tablet_manager.cc:1397] T 90baf83357494a0aae325908c118b471 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.100s user 0.011s sys 0.009s
I20250903 20:05:29.491447 27692 raft_consensus.cc:357] T 90baf83357494a0aae325908c118b471 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.491925 27057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_de42081b245243918dff0bc0301b027a" } new_table_name: "default.Table_0a08d5d27dd24365b9e527123d15e771" modify_external_catalogs: true
I20250903 20:05:29.494292 27692 raft_consensus.cc:383] T 90baf83357494a0aae325908c118b471 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.494390 27692 raft_consensus.cc:738] T 90baf83357494a0aae325908c118b471 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.494592 27695 tablet_bootstrap.cc:492] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:29.494565 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e0347eab6c8642beabdc2d7e7bbfdc9b" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:29.494732 27379 raft_consensus.cc:3058] T e0347eab6c8642beabdc2d7e7bbfdc9b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:29.494649 27692 consensus_queue.cc:260] T 90baf83357494a0aae325908c118b471 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.495352 27692 ts_tablet_manager.cc:1428] T 90baf83357494a0aae325908c118b471 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250903 20:05:29.495298 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2629f46704e4a0fa7922533928ca2ed" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:29.495487 27518 raft_consensus.cc:3058] T f2629f46704e4a0fa7922533928ca2ed P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:29.495322 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e0347eab6c8642beabdc2d7e7bbfdc9b" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:29.495795 27527 raft_consensus.cc:3058] T e0347eab6c8642beabdc2d7e7bbfdc9b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:29.495288 27517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "607c9a9cbc5540e5bd0fa436fcba564c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:29.496688 27183 leader_election.cc:343] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:29.497295 27695 tablet_bootstrap.cc:654] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.498739 28596 raft_consensus.cc:491] T a4f3230f50ec4c95b528fead8c191224 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:29.498875 28596 raft_consensus.cc:513] T a4f3230f50ec4c95b528fead8c191224 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:29.499776 28596 leader_election.cc:290] T a4f3230f50ec4c95b528fead8c191224 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:29.499963 28596 raft_consensus.cc:491] T 466e501e424e46bc98e8bbc64f97005e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:29.500087 28596 raft_consensus.cc:513] T 466e501e424e46bc98e8bbc64f97005e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.500898 28596 leader_election.cc:290] T 466e501e424e46bc98e8bbc64f97005e P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:29.501864 28852 raft_consensus.cc:491] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 35bdb0090d8747059b5ee103e786e979)
I20250903 20:05:29.502158 28852 raft_consensus.cc:513] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } }
I20250903 20:05:29.502749 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "466e501e424e46bc98e8bbc64f97005e" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:28.791313 28404 rpcz_store.cc:276] 0903 20:05:23.856288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:23.856569 (+ 281us) service_pool.cc:225] Handling call
0903 20:05:27.735851 (+3879282us) raft_consensus.cc:1469] Updating replica for 445 ops
0903 20:05:27.885741 (+149890us) raft_consensus.cc:1532] Early marking committed up to index 0
0903 20:05:27.885750 (+ 9us) raft_consensus.cc:1537] Triggering prepare for 445 ops
0903 20:05:27.891432 (+ 5682us) write_op.cc:270] Start()
0903 20:05:27.891507 (+ 75us) write_op.cc:276] Timestamp: P: 1756929915920687 usec, L: 0
0903 20:05:27.913705 (+ 22198us) write_op.cc:270] Start()
0903 20:05:27.913750 (+ 45us) write_op.cc:276] Timestamp: P: 1756929915921391 usec, L: 0
0903 20:05:27.914028 (+ 278us) write_op.cc:270] Start()
0903 20:05:27.914087 (+ 59us) write_op.cc:276] Timestamp: P: 1756929915921919 usec, L: 0
0903 20:05:27.914300 (+ 213us) write_op.cc:270] Start()
0903 20:05:27.918339 (+ 4039us) write_op.cc:276] Timestamp: P: 1756929915927215 usec, L: 0
0903 20:05:27.918989 (+ 650us) write_op.cc:270] Start()
0903 20:05:27.919025 (+ 36us) write_op.cc:276] Timestamp: P: 1756929915930350 usec, L: 0
0903 20:05:27.919197 (+ 172us) write_op.cc:270] Start()
0903 20:05:27.919219 (+ 22us) write_op.cc:276] Timestamp: P: 1756929915930988 usec, L: 0
0903 20:05:27.922774 (+ 3555us) write_op.cc:270] Start()
0903 20:05:27.922854 (+ 80us) write_op.cc:276] Timestamp: P: 1756929915933849 usec, L: 0
0903 20:05:27.923138 (+ 284us) write_op.cc:270] Start()
0903 20:05:27.923165 (+ 27us) write_op.cc:276] Timestamp: P: 1756929915934472 usec, L: 0
0903 20:05:27.923353 (+ 188us) write_op.cc:270] Start()
0903 20:05:27.923384 (+ 31us) write_op.cc:276] Timestamp: P: 1756929915935047 usec, L: 0
0903 20:05:27.923531 (+ 147us) write_op.cc:270] Start()
0903 20:05:27.923555 (+ 24us) write_op.cc:276] Timestamp: P: 1756929915964716 usec, L: 0
0903 20:05:27.923785 (+ 230us) write_op.cc:270] Start()
0903 20:05:27.923815 (+ 30us) write_op.cc:276] Timestamp: P: 1756929915992354 usec, L: 0
0903 20:05:27.923988 (+ 173us) write_op.cc:270] Start()
0903 20:05:27.924011 (+ 23us) write_op.cc:276] Timestamp: P: 1756929916005586 usec, L: 0
0903 20:05:27.924193 (+ 182us) write_op.cc:270] Start()
0903 20:05:27.924220 (+ 27us) write_op.cc:276] Timestamp: P: 1756929916016204 usec, L: 0
0903 20:05:27.924418 (+ 198us) write_op.cc:270] Start()
0903 20:05:27.924448 (+ 30us) write_op.cc:276] Timestamp: P: 1756929916020596 usec, L: 0
0903 20:05:27.924632 (+ 184us) write_op.cc:270] Start()
0903 20:05:27.924660 (+ 28us) write_op.cc:276] Timestamp: P: 1756929916022460 usec, L: 0
0903 20:05:27.924815 (+ 155us) write_op.cc:270] Start()
0903 20:05:27.924839 (+ 24us) write_op.cc:276] Timestamp: P: 1756929916024064 usec, L: 0
0903 20:05:27.925029 (+ 190us) write_op.cc:270] Start()
0903 20:05:27.925074 (+ 45us) write_op.cc:276] Timestamp: P: 1756929916062887 usec, L: 0
0903 20:05:27.964652 (+ 39578us) write_op.cc:270] Start()
0903 20:05:27.964711 (+ 59us) write_op.cc:276] Timestamp: P: 1756929916065500 usec, L: 0
0903 20:05:27.966252 (+ 1541us) write_op.cc:270] Start()
0903 20:05:27.966291 (+ 39us) write_op.cc:276] Timestamp: P: 1756929916073259 usec, L: 0
0903 20:05:27.966474 (+ 183us) write_op.cc:270] Start()
0903 20:05:27.966496 (+ 22us) write_op.cc:276] Timestamp: P: 1756929916099046 usec, L: 0
0903 20:05:27.966635 (+ 139us) write_op.cc:270] Start()
0903 20:05:27.966654 (+ 19us) write_op.cc:276] Timestamp: P: 1756929916166330 usec, L: 0
0903 20:05:27.966847 (+ 193us) write_op.cc:270] Start()
0903 20:05:27.966868 (+ 21us) write_op.cc:276] Timestamp: P: 1756929916167006 usec, L: 0
0903 20:05:27.966999 (+ 131us) write_op.cc:270] Start()
0903 20:05:27.967018 (+ 19us) write_op.cc:276] Timestamp: P: 1756929916175405 usec, L: 0
0903 20:05:27.967280 (+ 262us) write_op.cc:270] Start()
0903 20:05:27.967305 (+ 25us) write_op.cc:276] Timestamp: P: 1756929916175937 usec, L: 0
0903 20:05:27.967477 (+ 172us) write_op.cc:270] Start()
0903 20:05:27.987956 (+ 20479us) write_op.cc:276] Timestamp: P: 1756929916176481 usec, L: 0
0903 20:05:27.988243 (+ 287us) write_op.cc:270] Start()
0903 20:05:27.988275 (+ 32us) write_op.cc:276] Timestamp: P: 1756929916179898 usec, L: 0
0903 20:05:27.988447 (+ 172us) write_op.cc:270] Start()
0903 20:05:27.988468 (+ 21us) write_op.cc:276] Timestamp: P: 1756929916196658 usec, L: 0
0903 20:05:27.988590 (+ 122us) write_op.cc:270] Start()
0903 20:05:27.988609 (+ 19us) write_op.cc:276] Timestamp: P: 1756929916207663 usec, L: 0
0903 20:05:27.988752 (+ 143us) write_op.cc:270] Start()
0903 20:05:27.988772 (+ 20us) write_op.cc:276] Timestamp: P: 1756929916208303 usec, L: 0
0903 20:05:27.990180 (+ 1408us) write_op.cc:270] Start()
0903 20:05:27.990228 (+ 48us) write_op.cc:276] Timestamp: P: 1756929916208873 usec, L: 0
0903 20:05:27.990395 (+ 167us) write_op.cc:270] Start()
0903 20:05:27.990419 (+ 24us) write_op.cc:276] Timestamp: P: 1756929916209472 usec, L: 0
0903 20:05:27.990552 (+ 133us) write_op.cc:270] Start()
0903 20:05:27.990571 (+ 19us) write_op.cc:276] Timestamp: P: 1756929916212748 usec, L: 0
0903 20:05:27.990709 (+ 138us) write_op.cc:270] Start()
0903 20:05:27.990736 (+ 27us) write_op.cc:276] Timestamp: P: 1756929916218797 usec, L: 0
0903 20:05:27.990882 (+ 146us) write_op.cc:270] Start()
0903 20:05:27.990901 (+ 19us) write_op.cc:276] Timestamp: P: 1756929916276426 usec, L: 0
0903 20:05:27.991048 (+ 147us) write_op.cc:270] Start()
0903 20:05:27.991081 (+ 33us) write_op.cc:276] Timestamp: P: 1756929916278324 usec, L: 0
0903 20:05:27.991238 (+ 157us) write_op.cc:270] Start()
0903 20:05:27.991263 (+ 25us) write_op.cc:276] Timestamp: P: 1756929916288451 usec, L: 0
0903 20:05:27.991402 (+ 139us) write_op.cc:270] Start()
0903 20:05:27.991438 (+ 36us) write_op.cc:276] Timestamp: P: 1756929916302653 usec, L: 0
0903 20:05:27.991610 (+ 172us) write_op.cc:270] Start()
0903 20:05:27.991633 (+ 23us) write_op.cc:276] Timestamp: P: 1756929916303425 usec, L: 0
0903 20:05:27.991777 (+ 144us) write_op.cc:270] Start()
0903 20:05:27.991795 (+ 18us) write_op.cc:276] Timestamp: P: 1756929916304012 usec, L: 0
0903 20:05:27.991968 (+ 173us) write_op.cc:270] Start()
0903 20:05:27.991990 (+ 22us) write_op.cc:276] Timestamp: P: 1756929916304622 usec, L: 0
0903 20:05:27.992168 (+ 178us) write_op.cc:270] Start()
0903 20:05:27.992192 (+ 24us) write_op.cc:276] Timestamp: P: 1756929916306482 usec, L: 0
0903 20:05:27.992317 (+ 125us) write_op.cc:270] Start()
0903 20:05:27.992337 (+ 20us) write_op.cc:276] Timestamp: P: 1756929916345447 usec, L: 0
0903 20:05:27.992467 (+ 130us) write_op.cc:270] Start()
0903 20:05:27.992485 (+ 18us) write_op.cc:276] Timestamp: P: 1756929916384214 usec, L: 0
0903 20:05:27.992623 (+ 138us) write_op.cc:270] Start()
0903 20:05:27.992649 (+ 26us) write_op.cc:276] Timestamp: P: 1756929916395285 usec, L: 0
0903 20:05:27.992799 (+ 150us) write_op.cc:270] Start()
0903 20:05:27.992818 (+ 19us) write_op.cc:276] Timestamp: P: 1756929916395926 usec, L: 0
0903 20:05:27.992981 (+ 163us) write_op.cc:270] Start()
0903 20:05:27.993005 (+ 24us) write_op.cc:276] Timestamp: P: 1756929916396506 usec, L: 0
0903 20:05:27.993171 (+ 166us) write_op.cc:270] Start()
0903 20:05:27.993189 (+ 18us) write_op.cc:276] Timestamp: P: 1756929916397049 usec, L: 0
0903 20:05:27.993317 (+ 128us) write_op.cc:270] Start()
0903 20:05:27.993335 (+ 18us) write_op.cc:276] Timestamp: P: 1756929916405170 usec, L: 0
0903 20:05:27.993488 (+ 153us) write_op.cc:270] Start()
0903 20:05:27.993506 (+ 18us) write_op.cc:276] Timestamp: P: 1756929916420708 usec, L: 0
0903 20:05:27.993625 (+ 119us) write_op.cc:270] Start()
0903 20:05:27.993644 (+ 19us) write_op.cc:276] Timestamp: P: 1756929916442700 usec, L: 0
0903 20:05:27.993877 (+ 233us) write_op.cc:270] Start()
0903 20:05:27.993897 (+ 20us) write_op.cc:276] Timestamp: P: 1756929916446012 usec, L: 0
0903 20:05:27.994019 (+ 122us) write_op.cc:270] Start()
0903 20:05:27.994038 (+ 19us) write_op.cc:276] Timestamp: P: 1756929916447242 usec, L: 0
0903 20:05:27.994178 (+ 140us) write_op.cc:270] Start()
0903 20:05:27.994197 (+ 19us) write_op.cc:276] Timestamp: P: 1756929916447960 usec, L: 0
0903 20:05:27.994408 (+ 211us) write_op.cc:270] Start()
0903 20:05:27.994429 (+ 21us) write_op.cc:276] Timestamp: P: 1756929916449100 usec, L: 0
0903 20:05:27.994557 (+ 128us) write_op.cc:270] Start()
0903 20:05:27.994575 (+ 18us) write_op.cc:276] Timestamp: P: 1756929916450651 usec, L: 0
0903 20:05:27.994710 (+ 135us) write_op.cc:270] Start()
0903 20:05:27.994727 (+ 17us) write_op.cc:276] Timestamp: P: 1756929916479351 usec, L: 0
0903 20:05:27.994845 (+ 118us) write_op.cc:270] Start()
0903 20:05:27.994863 (+ 18us) write_op.cc:276] Timestamp: P: 1756929916519933 usec, L: 0
0903 20:05:27.995036 (+ 173us) write_op.cc:270] Start()
0903 20:05:27.995066 (+ 30us) write_op.cc:276] Timestamp: P: 1756929916526920 usec, L: 0
0903 20:05:27.995211 (+ 145us) write_op.cc:270] Start()
0903 20:05:27.995236 (+ 25us) write_op.cc:276] Timestamp: P: 1756929916529242 usec, L: 0
0903 20:05:27.995386 (+ 150us) write_op.cc:270] Start()
0903 20:05:27.995424 (+ 38us) write_op.cc:276] Timestamp: P: 1756929916536792 usec, L: 0
0903 20:05:27.995550 (+ 126us) write_op.cc:270] Start()
0903 20:05:27.995570 (+ 20us) write_op.cc:276] Timestamp: P: 1756929916537525 usec, L: 0
0903 20:05:27.995716 (+ 146us) write_op.cc:270] Start()
0903 20:05:27.995736 (+ 20us) write_op.cc:276] Timestamp: P: 1756929916538289 usec, L: 0
0903 20:05:27.995880 (+ 144us) write_op.cc:270] Start()
0903 20:05:27.995897 (+ 17us) write_op.cc:276] Timestamp: P: 1756929916538999 usec, L: 0
0903 20:05:27.996023 (+ 126us) write_op.cc:270] Start()
0903 20:05:27.996046 (+ 23us) write_op.cc:276] Timestamp: P: 1756929916540262 usec, L: 0
0903 20:05:27.996225 (+ 179us) write_op.cc:270] Start()
0903 20:05:27.996241 (+ 16us) write_op.cc:276] Timestamp: P: 1756929916573159 usec, L: 0
0903 20:05:27.996356 (+ 115us) write_op.cc:270] Start()
0903 20:05:27.996379 (+ 23us) write_op.cc:276] Timestamp: P: 1756929916656009 usec, L: 0
0903 20:05:27.996509 (+ 130us) write_op.cc:270] Start()
0903 20:05:27.996527 (+ 18us) write_op.cc:276] Timestamp: P: 1756929916658005 usec, L: 0
0903 20:05:27.996668 (+ 141us) write_op.cc:270] Start()
0903 20:05:27.996686 (+ 18us) write_op.cc:276] Timestamp: P: 1756929916861032 usec, L: 0
0903 20:05:27.996812 (+ 126us) write_op.cc:270] Start()
0903 20:05:27.996834 (+ 22us) write_op.cc:276] Timestamp: P: 1756929916861906 usec, L: 0
0903 20:05:27.996983 (+ 149us) write_op.cc:270] Start()
0903 20:05:27.997002 (+ 19us) write_op.cc:276] Timestamp: P: 1756929916862949 usec, L: 0
0903 20:05:28.005647 (+ 8645us) write_op.cc:270] Start()
0903 20:05:28.005684 (+ 37us) write_op.cc:276] Timestamp: P: 1756929916863500 usec, L: 0
0903 20:05:28.005868 (+ 184us) write_op.cc:270] Start()
0903 20:05:28.005887 (+ 19us) write_op.cc:276] Timestamp: P: 1756929916864050 usec, L: 0
0903 20:05:28.005996 (+ 109us) write_op.cc:270] Start()
0903 20:05:28.006014 (+ 18us) write_op.cc:276] Timestamp: P: 1756929916900019 usec, L: 0
0903 20:05:28.006157 (+ 143us) write_op.cc:270] Start()
0903 20:05:28.006177 (+ 20us) write_op.cc:276] Timestamp: P: 1756929916903873 usec, L: 0
0903 20:05:28.006381 (+ 204us) write_op.cc:270] Start()
0903 20:05:28.006403 (+ 22us) write_op.cc:276] Timestamp: P: 1756929916967481 usec, L: 0
0903 20:05:28.006527 (+ 124us) write_op.cc:270] Start()
0903 20:05:28.006544 (+ 17us) write_op.cc:276] Timestamp: P: 1756929916968536 usec, L: 0
0903 20:05:28.006644 (+ 100us) write_op.cc:270] Start()
0903 20:05:28.006659 (+ 15us) write_op.cc:276] Timestamp: P: 1756929916969477 usec, L: 0
0903 20:05:28.006760 (+ 101us) write_op.cc:270] Start()
0903 20:05:28.006776 (+ 16us) write_op.cc:276] Timestamp: P: 1756929916971030 usec, L: 0
0903 20:05:28.006871 (+ 95us) write_op.cc:270] Start()
0903 20:05:28.006913 (+ 42us) write_op.cc:276] Timestamp: P: 1756929916972775 usec, L: 0
0903 20:05:28.007075 (+ 162us) write_op.cc:270] Start()
0903 20:05:28.007101 (+ 26us) write_op.cc:276] Timestamp: P: 1756929916973904 usec, L: 0
0903 20:05:28.007255 (+ 154us) write_op.cc:270] Start()
0903 20:05:28.007278 (+ 23us) write_op.cc:276] Timestamp: P: 1756929916976131 usec, L: 0
0903 20:05:28.007428 (+ 150us) write_op.cc:270] Start()
0903 20:05:28.007448 (+ 20us) write_op.cc:276] Timestamp: P: 1756929917001834 usec, L: 0
0903 20:05:28.007595 (+ 147us) write_op.cc:270] Start()
0903 20:05:28.007613 (+ 18us) write_op.cc:276] Timestamp: P: 1756929917004240 usec, L: 0
0903 20:05:28.007749 (+ 136us) write_op.cc:270] Start()
0903 20:05:28.007769 (+ 20us) write_op.cc:276] Timestamp: P: 1756929917017516 usec, L: 0
0903 20:05:28.007922 (+ 153us) write_op.cc:270] Start()
0903 20:05:28.007945 (+ 23us) write_op.cc:276] Timestamp: P: 1756929917026927 usec, L: 0
0903 20:05:28.020422 (+ 12477us) write_op.cc:270] Start()
0903 20:05:28.020472 (+ 50us) write_op.cc:276] Timestamp: P: 1756929917033669 usec, L: 0
0903 20:05:28.020694 (+ 222us) write_op.cc:270] Start()
0903 20:05:28.020714 (+ 20us) write_op.cc:276] Timestamp: P: 1756929917101449 usec, L: 0
0903 20:05:28.020846 (+ 132us) write_op.cc:270] Start()
0903 20:05:28.020864 (+ 18us) write_op.cc:276] Timestamp: P: 1756929917136156 usec, L: 0
0903 20:05:28.020979 (+ 115us) write_op.cc:270] Start()
0903 20:05:28.020996 (+ 17us) write_op.cc:276] Timestamp: P: 1756929917202224 usec, L: 0
0903 20:05:28.021138 (+ 142us) write_op.cc:270] Start()
0903 20:05:28.021156 (+ 18us) write_op.cc:276] Timestamp: P: 1756929917253497 usec, L: 0
0903 20:05:28.028853 (+ 7697us) write_op.cc:270] Start()
0903 20:05:28.028895 (+ 42us) write_op.cc:276] Timestamp: P: 1756929917437030 usec, L: 0
0903 20:05:28.029123 (+ 228us) write_op.cc:270] Start()
0903 20:05:28.029145 (+ 22us) write_op.cc:276] Timestamp: P: 1756929917443508 usec, L: 0
0903 20:05:28.029265 (+ 120us) write_op.cc:270] Start()
0903 20:05:28.029282 (+ 17us) write_op.cc:276] Timestamp: P: 1756929917448737 usec, L: 0
0903 20:05:28.029408 (+ 126us) write_op.cc:270] Start()
0903 20:05:28.029447 (+ 39us) write_op.cc:276] Timestamp: P: 1756929917449534 usec, L: 0
0903 20:05:28.029568 (+ 121us) write_op.cc:270] Start()
0903 20:05:28.029584 (+ 16us) write_op.cc:276] Timestamp: P: 1756929917450964 usec, L: 0
0903 20:05:28.029737 (+ 153us) write_op.cc:270] Start()
0903 20:05:28.029753 (+ 16us) write_op.cc:276] Timestamp: P: 1756929917539320 usec, L: 0
0903 20:05:28.029869 (+ 116us) write_op.cc:270] Start()
0903 20:05:28.029885 (+ 16us) write_op.cc:276] Timestamp: P: 1756929917542352 usec, L: 0
0903 20:05:28.029996 (+ 111us) write_op.cc:270] Start()
0903 20:05:28.030013 (+ 17us) write_op.cc:276] Timestamp: P: 1756929917543593 usec, L: 0
0903 20:05:28.030168 (+ 155us) write_op.cc:270] Start()
0903 20:05:28.030190 (+ 22us) write_op.cc:276] Timestamp: P: 1756929917544652 usec, L: 0
0903 20:05:28.030375 (+ 185us) write_op.cc:270] Start()
0903 20:05:28.030396 (+ 21us) write_op.cc:276] Timestamp: P: 1756929917545675 usec, L: 0
0903 20:05:28.030522 (+ 126us) write_op.cc:270] Start()
0903 20:05:28.030540 (+ 18us) write_op.cc:276] Timestamp: P: 1756929917546676 usec, L: 0
0903 20:05:28.030656 (+ 116us) write_op.cc:270] Start()
0903 20:05:28.030672 (+ 16us) write_op.cc:276] Timestamp: P: 1756929917551768 usec, L: 0
0903 20:05:28.030838 (+ 166us) write_op.cc:270] Start()
0903 20:05:28.030860 (+ 22us) write_op.cc:276] Timestamp: P: 1756929917553801 usec, L: 0
0903 20:05:28.031027 (+ 167us) write_op.cc:270] Start()
0903 20:05:28.031051 (+ 24us) write_op.cc:276] Timestamp: P: 1756929917612245 usec, L: 0
0903 20:05:28.031211 (+ 160us) write_op.cc:270] Start()
0903 20:05:28.031232 (+ 21us) write_op.cc:276] Timestamp: P: 1756929917679102 usec, L: 0
0903 20:05:28.031357 (+ 125us) write_op.cc:270] Start()
0903 20:05:28.031375 (+ 18us) write_op.cc:276] Timestamp: P: 1756929917679697 usec, L: 0
0903 20:05:28.031517 (+ 142us) write_op.cc:270] Start()
0903 20:05:28.031540 (+ 23us) write_op.cc:276] Timestamp: P: 1756929917682879 usec, L: 0
0903 20:05:28.031678 (+ 138us) write_op.cc:270] Start()
0903 20:05:28.031698 (+ 20us) write_op.cc:276] Timestamp: P: 1756929917700542 usec, L: 0
0903 20:05:28.031832 (+ 134us) write_op.cc:270] Start()
0903 20:05:28.031855 (+ 23us) write_op.cc:276] Timestamp: P: 1756929917702137 usec, L: 0
0903 20:05:28.031999 (+ 144us) write_op.cc:270] Start()
0903 20:05:28.032018 (+ 19us) write_op.cc:276] Timestamp: P: 1756929917753270 usec, L: 0
0903 20:05:28.032171 (+ 153us) write_op.cc:270] Start()
0903 20:05:28.032192 (+ 21us) write_op.cc:276] Timestamp: P: 1756929917766967 usec, L: 0
0903 20:05:28.032347 (+ 155us) write_op.cc:270] Start()
0903 20:05:28.032396 (+ 49us) write_op.cc:276] Timestamp: P: 1756929917767689 usec, L: 0
0903 20:05:28.032531 (+ 135us) write_op.cc:270] Start()
0903 20:05:28.032550 (+ 19us) write_op.cc:276] Timestamp: P: 1756929917795641 usec, L: 0
0903 20:05:28.032700 (+ 150us) write_op.cc:270] Start()
0903 20:05:28.032720 (+ 20us) write_op.cc:276] Timestamp: P: 1756929917796403 usec, L: 0
0903 20:05:28.032851 (+ 131us) write_op.cc:270] Start()
0903 20:05:28.032871 (+ 20us) write_op.cc:276] Timestamp: P: 1756929917798972 usec, L: 0
0903 20:05:28.032996 (+ 125us) write_op.cc:270] Start()
0903 20:05:28.033014 (+ 18us) write_op.cc:276] Timestamp: P: 1756929917799536 usec, L: 0
0903 20:05:28.033158 (+ 144us) write_op.cc:270] Start()
0903 20:05:28.033177 (+ 19us) write_op.cc:276] Timestamp: P: 1756929917895926 usec, L: 0
0903 20:05:28.033297 (+ 120us) write_op.cc:270] Start()
0903 20:05:28.033314 (+ 17us) write_op.cc:276] Timestamp: P: 1756929917897163 usec, L: 0
0903 20:05:28.033453 (+ 139us) write_op.cc:270] Start()
0903 20:05:28.033473 (+ 20us) write_op.cc:276] Timestamp: P: 1756929917899380 usec, L: 0
0903 20:05:28.033604 (+ 131us) write_op.cc:270] Start()
0903 20:05:28.033623 (+ 19us) write_op.cc:276] Timestamp: P: 1756929917930177 usec, L: 0
0903 20:05:28.033788 (+ 165us) write_op.cc:270] Start()
0903 20:05:28.033806 (+ 18us) write_op.cc:276] Timestamp: P: 1756929917931135 usec, L: 0
0903 20:05:28.033981 (+ 175us) write_op.cc:270] Start()
0903 20:05:28.034007 (+ 26us) write_op.cc:276] Timestamp: P: 1756929917969509 usec, L: 0
0903 20:05:28.034167 (+ 160us) write_op.cc:270] Start()
0903 20:05:28.034187 (+ 20us) write_op.cc:276] Timestamp: P: 1756929918037556 usec, L: 0
0903 20:05:28.079261 (+ 45074us) write_op.cc:270] Start()
0903 20:05:28.079316 (+ 55us) write_op.cc:276] Timestamp: P: 1756929918087518 usec, L: 0
0903 20:05:28.079525 (+ 209us) write_op.cc:270] Start()
0903 20:05:28.079550 (+ 25us) write_op.cc:276] Timestamp: P: 1756929918112488 usec, L: 0
0903 20:05:28.079788 (+ 238us) write_op.cc:270] Start()
0903 20:05:28.079814 (+ 26us) write_op.cc:276] Timestamp: P: 1756929918140557 usec, L: 0
0903 20:05:28.079975 (+ 161us) write_op.cc:270] Start()
0903 20:05:28.079995 (+ 20us) write_op.cc:276] Timestamp: P: 1756929918141362 usec, L: 0
0903 20:05:28.080124 (+ 129us) write_op.cc:270] Start()
0903 20:05:28.080140 (+ 16us) write_op.cc:276] Timestamp: P: 1756929918142322 usec, L: 0
0903 20:05:28.080310 (+ 170us) write_op.cc:270] Start()
0903 20:05:28.080330 (+ 20us) write_op.cc:276] Timestamp: P: 1756929918192993 usec, L: 0
0903 20:05:28.080450 (+ 120us) write_op.cc:270] Start()
0903 20:05:28.080467 (+ 17us) write_op.cc:276] Timestamp: P: 1756929918204164 usec, L: 0
0903 20:05:28.080607 (+ 140us) write_op.cc:270] Start()
0903 20:05:28.080648 (+ 41us) write_op.cc:276] Timestamp: P: 1756929918205148 usec, L: 0
0903 20:05:28.080780 (+ 132us) write_op.cc:270] Start()
0903 20:05:28.080798 (+ 18us) write_op.cc:276] Timestamp: P: 1756929918205698 usec, L: 0
0903 20:05:28.080939 (+ 141us) write_op.cc:270] Start()
0903 20:05:28.080955 (+ 16us) write_op.cc:276] Timestamp: P: 1756929918206808 usec, L: 0
0903 20:05:28.081076 (+ 121us) write_op.cc:270] Start()
0903 20:05:28.081093 (+ 17us) write_op.cc:276] Timestamp: P: 1756929918209941 usec, L: 0
0903 20:05:28.081230 (+ 137us) write_op.cc:270] Start()
0903 20:05:28.081250 (+ 20us) write_op.cc:276] Timestamp: P: 1756929918210791 usec, L: 0
0903 20:05:28.081391 (+ 141us) write_op.cc:270] Start()
0903 20:05:28.081409 (+ 18us) write_op.cc:276] Timestamp: P: 1756929918211384 usec, L: 0
0903 20:05:28.081583 (+ 174us) write_op.cc:270] Start()
0903 20:05:28.081604 (+ 21us) write_op.cc:276] Timestamp: P: 1756929918214004 usec, L: 0
0903 20:05:28.081722 (+ 118us) write_op.cc:270] Start()
0903 20:05:28.081740 (+ 18us) write_op.cc:276] Timestamp: P: 1756929918216859 usec, L: 0
0903 20:05:28.081893 (+ 153us) write_op.cc:270] Start()
0903 20:05:28.081921 (+ 28us) write_op.cc:276] Timestamp: P: 1756929918218984 usec, L: 0
0903 20:05:28.082101 (+ 180us) write_op.cc:270] Start()
0903 20:05:28.082126 (+ 25us) write_op.cc:276] Timestamp: P: 1756929918247266 usec, L: 0
0903 20:05:28.111469 (+ 29343us) write_op.cc:270] Start()
0903 20:05:28.111531 (+ 62us) write_op.cc:276] Timestamp: P: 1756929918259982 usec, L: 0
0903 20:05:28.111805 (+ 274us) write_op.cc:270] Start()
0903 20:05:28.111837 (+ 32us) write_op.cc:276] Timestamp: P: 1756929918269341 usec, L: 0
0903 20:05:28.111990 (+ 153us) write_op.cc:270] Start()
0903 20:05:28.112015 (+ 25us) write_op.cc:276] Timestamp: P: 1756929918297096 usec, L: 0
0903 20:05:28.112259 (+ 244us) write_op.cc:270] Start()
0903 20:05:28.112285 (+ 26us) write_op.cc:276] Timestamp: P: 1756929918336997 usec, L: 0
0903 20:05:28.112456 (+ 171us) write_op.cc:270] Start()
0903 20:05:28.112481 (+ 25us) write_op.cc:276] Timestamp: P: 1756929918349814 usec, L: 0
0903 20:05:28.112632 (+ 151us) write_op.cc:270] Start()
0903 20:05:28.112658 (+ 26us) write_op.cc:276] Timestamp: P: 1756929918379471 usec, L: 0
0903 20:05:28.112772 (+ 114us) write_op.cc:270] Start()
0903 20:05:28.112787 (+ 15us) write_op.cc:276] Timestamp: P: 1756929918390495 usec, L: 0
0903 20:05:28.130615 (+ 17828us) write_op.cc:270] Start()
0903 20:05:28.130656 (+ 41us) write_op.cc:276] Timestamp: P: 1756929918410034 usec, L: 0
0903 20:05:28.130859 (+ 203us) write_op.cc:270] Start()
0903 20:05:28.130894 (+ 35us) write_op.cc:276] Timestamp: P: 1756929918448436 usec, L: 0
0903 20:05:28.131091 (+ 197us) write_op.cc:270] Start()
0903 20:05:28.131111 (+ 20us) write_op.cc:276] Timestamp: P: 1756929918477278 usec, L: 0
0903 20:05:28.131290 (+ 179us) write_op.cc:270] Start()
0903 20:05:28.133432 (+ 2142us) write_op.cc:276] Timestamp: P: 1756929918498452 usec, L: 0
0903 20:05:28.133624 (+ 192us) write_op.cc:270] Start()
0903 20:05:28.133653 (+ 29us) write_op.cc:276] Timestamp: P: 1756929918509795 usec, L: 0
0903 20:05:28.133814 (+ 161us) write_op.cc:270] Start()
0903 20:05:28.133837 (+ 23us) write_op.cc:276] Timestamp: P: 1756929918522546 usec, L: 0
0903 20:05:28.134007 (+ 170us) write_op.cc:270] Start()
0903 20:05:28.134035 (+ 28us) write_op.cc:276] Timestamp: P: 1756929918523977 usec, L: 0
0903 20:05:28.134239 (+ 204us) write_op.cc:270] Start()
0903 20:05:28.134266 (+ 27us) write_op.cc:276] Timestamp: P: 1756929918524680 usec, L: 0
0903 20:05:28.134409 (+ 143us) write_op.cc:270] Start()
0903 20:05:28.134428 (+ 19us) write_op.cc:276] Timestamp: P: 1756929918525823 usec, L: 0
0903 20:05:28.134551 (+ 123us) write_op.cc:270] Start()
0903 20:05:28.134568 (+ 17us) write_op.cc:276] Timestamp: P: 1756929918548543 usec, L: 0
0903 20:05:28.134715 (+ 147us) write_op.cc:270] Start()
0903 20:05:28.134736 (+ 21us) write_op.cc:276] Timestamp: P: 1756929918549196 usec, L: 0
0903 20:05:28.134878 (+ 142us) write_op.cc:270] Start()
0903 20:05:28.134900 (+ 22us) write_op.cc:276] Timestamp: P: 1756929918549726 usec, L: 0
0903 20:05:28.135035 (+ 135us) write_op.cc:270] Start()
0903 20:05:28.135068 (+ 33us) write_op.cc:276] Timestamp: P: 1756929918550206 usec, L: 0
0903 20:05:28.135239 (+ 171us) write_op.cc:270] Start()
0903 20:05:28.135260 (+ 21us) write_op.cc:276] Timestamp: P: 1756929918562825 usec, L: 0
0903 20:05:28.135397 (+ 137us) write_op.cc:270] Start()
0903 20:05:28.135420 (+ 23us) write_op.cc:276] Timestamp: P: 1756929918563441 usec, L: 0
0903 20:05:28.135551 (+ 131us) write_op.cc:270] Start()
0903 20:05:28.135570 (+ 19us) write_op.cc:276] Timestamp: P: 1756929918564107 usec, L: 0
0903 20:05:28.135693 (+ 123us) write_op.cc:270] Start()
0903 20:05:28.135711 (+ 18us) write_op.cc:276] Timestamp: P: 1756929918564844 usec, L: 0
0903 20:05:28.135835 (+ 124us) write_op.cc:270] Start()
0903 20:05:28.135854 (+ 19us) write_op.cc:276] Timestamp: P: 1756929918565521 usec, L: 0
0903 20:05:28.135963 (+ 109us) write_op.cc:270] Start()
0903 20:05:28.135979 (+ 16us) write_op.cc:276] Timestamp: P: 1756929918599317 usec, L: 0
0903 20:05:28.136186 (+ 207us) write_op.cc:270] Start()
0903 20:05:28.136250 (+ 64us) write_op.cc:276] Timestamp: P: 1756929918600261 usec, L: 0
0903 20:05:28.136429 (+ 179us) write_op.cc:270] Start()
0903 20:05:28.136450 (+ 21us) write_op.cc:276] Timestamp: P: 1756929918669559 usec, L: 0
0903 20:05:28.136593 (+ 143us) write_op.cc:270] Start()
0903 20:05:28.136611 (+ 18us) write_op.cc:276] Timestamp: P: 1756929918686254 usec, L: 0
0903 20:05:28.136730 (+ 119us) write_op.cc:270] Start()
0903 20:05:28.136747 (+ 17us) write_op.cc:276] Timestamp: P: 1756929918696759 usec, L: 0
0903 20:05:28.136862 (+ 115us) write_op.cc:270] Start()
0903 20:05:28.136881 (+ 19us) write_op.cc:276] Timestamp: P: 1756929918698820 usec, L: 0
0903 20:05:28.137004 (+ 123us) write_op.cc:270] Start()
0903 20:05:28.137023 (+ 19us) write_op.cc:276] Timestamp: P: 1756929918733586 usec, L: 0
0903 20:05:28.137183 (+ 160us) write_op.cc:270] Start()
0903 20:05:28.137214 (+ 31us) write_op.cc:276] Timestamp: P: 1756929918734924 usec, L: 0
0903 20:05:28.137332 (+ 118us) write_op.cc:270] Start()
0903 20:05:28.137351 (+ 19us) write_op.cc:276] Timestamp: P: 1756929918735925 usec, L: 0
0903 20:05:28.137499 (+ 148us) write_op.cc:270] Start()
0903 20:05:28.137521 (+ 22us) write_op.cc:276] Timestamp: P: 1756929918737562 usec, L: 0
0903 20:05:28.137666 (+ 145us) write_op.cc:270] Start()
0903 20:05:28.137687 (+ 21us) write_op.cc:276] Timestamp: P: 1756929918759173 usec, L: 0
0903 20:05:28.137855 (+ 168us) write_op.cc:270] Start()
0903 20:05:28.137876 (+ 21us) write_op.cc:276] Timestamp: P: 1756929918760763 usec, L: 0
0903 20:05:28.138039 (+ 163us) write_op.cc:270] Start()
0903 20:05:28.138073 (+ 34us) write_op.cc:276] Timestamp: P: 1756929918761950 usec, L: 0
0903 20:05:28.167946 (+ 29873us) write_op.cc:270] Start()
0903 20:05:28.168035 (+ 89us) write_op.cc:276] Timestamp: P: 1756929918763089 usec, L: 0
0903 20:05:28.169679 (+ 1644us) write_op.cc:270] Start()
0903 20:05:28.169734 (+ 55us) write_op.cc:276] Timestamp: P: 1756929918780639 usec, L: 0
0903 20:05:28.169971 (+ 237us) write_op.cc:270] Start()
0903 20:05:28.169993 (+ 22us) write_op.cc:276] Timestamp: P: 1756929918794276 usec, L: 0
0903 20:05:28.170167 (+ 174us) write_op.cc:270] Start()
0903 20:05:28.170190 (+ 23us) write_op.cc:276] Timestamp: P: 1756929918802050 usec, L: 0
0903 20:05:28.170393 (+ 203us) write_op.cc:270] Start()
0903 20:05:28.170419 (+ 26us) write_op.cc:276] Timestamp: P: 1756929918807546 usec, L: 0
0903 20:05:28.170590 (+ 171us) write_op.cc:270] Start()
0903 20:05:28.170618 (+ 28us) write_op.cc:276] Timestamp: P: 1756929918823668 usec, L: 0
0903 20:05:28.170802 (+ 184us) write_op.cc:270] Start()
0903 20:05:28.170838 (+ 36us) write_op.cc:276] Timestamp: P: 1756929918837550 usec, L: 0
0903 20:05:28.171015 (+ 177us) write_op.cc:270] Start()
0903 20:05:28.171039 (+ 24us) write_op.cc:276] Timestamp: P: 1756929918874096 usec, L: 0
0903 20:05:28.171171 (+ 132us) write_op.cc:270] Start()
0903 20:05:28.171190 (+ 19us) write_op.cc:276] Timestamp: P: 1756929918893745 usec, L: 0
0903 20:05:28.171348 (+ 158us) write_op.cc:270] Start()
0903 20:05:28.171379 (+ 31us) write_op.cc:276] Timestamp: P: 1756929918894499 usec, L: 0
0903 20:05:28.171555 (+ 176us) write_op.cc:270] Start()
0903 20:05:28.171579 (+ 24us) write_op.cc:276] Timestamp: P: 1756929918917018 usec, L: 0
0903 20:05:28.171746 (+ 167us) write_op.cc:270] Start()
0903 20:05:28.171770 (+ 24us) write_op.cc:276] Timestamp: P: 1756929918936298 usec, L: 0
0903 20:05:28.171925 (+ 155us) write_op.cc:270] Start()
0903 20:05:28.171948 (+ 23us) write_op.cc:276] Timestamp: P: 1756929918945099 usec, L: 0
0903 20:05:28.172116 (+ 168us) write_op.cc:270] Start()
0903 20:05:28.172141 (+ 25us) write_op.cc:276] Timestamp: P: 1756929918955090 usec, L: 0
0903 20:05:28.172302 (+ 161us) write_op.cc:270] Start()
0903 20:05:28.172322 (+ 20us) write_op.cc:276] Timestamp: P: 1756929918957530 usec, L: 0
0903 20:05:28.172475 (+ 153us) write_op.cc:270] Start()
0903 20:05:28.172498 (+ 23us) write_op.cc:276] Timestamp: P: 1756929918958926 usec, L: 0
0903 20:05:28.172635 (+ 137us) write_op.cc:270] Start()
0903 20:05:28.172652 (+ 17us) write_op.cc:276] Timestamp: P: 1756929918959543 usec, L: 0
0903 20:05:28.172783 (+ 131us) write_op.cc:270] Start()
0903 20:05:28.172802 (+ 19us) write_op.cc:276] Timestamp: P: 1756929918962313 usec, L: 0
0903 20:05:28.172946 (+ 144us) write_op.cc:270] Start()
0903 20:05:28.172973 (+ 27us) write_op.cc:276] Timestamp: P: 1756929
I20250903 20:05:29.503614 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a4f3230f50ec4c95b528fead8c191224" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:29.503680 27873 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" candidate_term: 4 candidate_status { last_received { term: 3 index: 933 } } ignore_live_leader: false dest_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" is_pre_election: true
I20250903 20:05:29.504997 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a4f3230f50ec4c95b528fead8c191224" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:29.505244 27385 raft_consensus.cc:2391] T a4f3230f50ec4c95b528fead8c191224 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 89f8f0ee68b940db9ce250c0604f934a in this term.
I20250903 20:05:29.505491 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "466e501e424e46bc98e8bbc64f97005e" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:29.505452 27076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" candidate_term: 4 candidate_status { last_received { term: 3 index: 933 } } ignore_live_leader: false dest_uuid: "35bdb0090d8747059b5ee103e786e979" is_pre_election: true
I20250903 20:05:29.505663 27381 raft_consensus.cc:2391] T 466e501e424e46bc98e8bbc64f97005e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 89f8f0ee68b940db9ce250c0604f934a in this term.
I20250903 20:05:29.506332 27462 leader_election.cc:304] T a4f3230f50ec4c95b528fead8c191224 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:29.506779 27462 leader_election.cc:304] T 466e501e424e46bc98e8bbc64f97005e P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:29.506067 28317 leader_election.cc:304] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35dd69f3fa7a4fd8843b97f8a5d68296; no voters: 35bdb0090d8747059b5ee103e786e979, 8857b5d4ea1e4f45aafb8e029837cca2
I20250903 20:05:29.506428 28852 leader_election.cc:290] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8857b5d4ea1e4f45aafb8e029837cca2 (127.26.65.126:39015), 35bdb0090d8747059b5ee103e786e979 (127.26.65.124:40911)
I20250903 20:05:29.507375 28596 raft_consensus.cc:3058] T 466e501e424e46bc98e8bbc64f97005e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:29.507535 28852 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250903 20:05:29.508186 28761 raft_consensus.cc:3058] T a4f3230f50ec4c95b528fead8c191224 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:29.521196 27377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "607c9a9cbc5540e5bd0fa436fcba564c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:29.521466 27377 raft_consensus.cc:2466] T 607c9a9cbc5540e5bd0fa436fcba564c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:29.522537 27692 tablet_bootstrap.cc:492] T a72b85dfb34047528b2b28ff0d40f828 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:29.528919 27695 tablet_bootstrap.cc:492] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:29.529080 27695 ts_tablet_manager.cc:1397] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.002s
I20250903 20:05:29.529117 27692 tablet_bootstrap.cc:654] T a72b85dfb34047528b2b28ff0d40f828 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:29.536995 27184 leader_election.cc:343] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 7cc6cf579e2642e99fefed723b0ee900
I20250903 20:05:29.537297 27695 raft_consensus.cc:357] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.541574 27695 raft_consensus.cc:383] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.541690 27695 raft_consensus.cc:738] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.542969 27692 tablet_bootstrap.cc:492] T a72b85dfb34047528b2b28ff0d40f828 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:29.543228 27692 ts_tablet_manager.cc:1397] T a72b85dfb34047528b2b28ff0d40f828 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.005s
I20250903 20:05:29.544097 28759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "35bdb0090d8747059b5ee103e786e979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:29.544348 28759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [sys.catalog]: This master's current role is: LEADER
I20250903 20:05:29.544523 27692 raft_consensus.cc:357] T a72b85dfb34047528b2b28ff0d40f828 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.544705 27692 raft_consensus.cc:383] T a72b85dfb34047528b2b28ff0d40f828 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.544765 27692 raft_consensus.cc:738] T a72b85dfb34047528b2b28ff0d40f828 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.545756 27692 consensus_queue.cc:260] T a72b85dfb34047528b2b28ff0d40f828 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.545881 28817 raft_consensus.cc:2747] T 7e2f98f760cf4df4bc2a10530e2157c6 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:29.548372 27518 raft_consensus.cc:2466] T f2629f46704e4a0fa7922533928ca2ed P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:29.550132 27692 ts_tablet_manager.cc:1428] T a72b85dfb34047528b2b28ff0d40f828 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250903 20:05:29.550510 27692 tablet_bootstrap.cc:492] T 0a9b7770432942da9373d5100c71bd6a P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:29.551424 27183 leader_election.cc:304] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:29.552270 27695 consensus_queue.cc:260] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.552800 27695 ts_tablet_manager.cc:1428] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20250903 20:05:29.553097 27695 tablet_bootstrap.cc:492] T 2fed2821d2644f75aff6e0ceb5944a37 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:29.553125 28780 raft_consensus.cc:513] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.554026 28780 leader_election.cc:290] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:29.554189 28780 raft_consensus.cc:2802] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:29.554472 28780 raft_consensus.cc:695] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:29.554744 27695 tablet_bootstrap.cc:654] T 2fed2821d2644f75aff6e0ceb5944a37 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.554770 28780 consensus_queue.cc:237] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.555831 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cc6cf579e2642e99fefed723b0ee900" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:29.556017 27518 raft_consensus.cc:3058] T 7cc6cf579e2642e99fefed723b0ee900 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Advancing to term 2
I20250903 20:05:29.557965 27386 raft_consensus.cc:2466] T f2629f46704e4a0fa7922533928ca2ed P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:29.563041 27692 tablet_bootstrap.cc:654] T 0a9b7770432942da9373d5100c71bd6a P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.565186 27693 tablet_bootstrap.cc:492] T 5f27f525fdab407280aa23df1ae71b43 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:29.565420 27693 ts_tablet_manager.cc:1397] T 5f27f525fdab407280aa23df1ae71b43 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.093s user 0.007s sys 0.008s
I20250903 20:05:29.566730 27685 ts_tablet_manager.cc:1918] T 45d9b6ff520348628d19671127b9426c P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:29.566919 27685 log.cc:1199] T 45d9b6ff520348628d19671127b9426c P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/45d9b6ff520348628d19671127b9426c
I20250903 20:05:29.567420 27685 ts_tablet_manager.cc:1939] T 45d9b6ff520348628d19671127b9426c P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:29.564885 28849 tablet.cc:1722] T d859c0222c3647dab3dddd105e4cf0fc P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250903 20:05:29.568657 27693 raft_consensus.cc:357] T 5f27f525fdab407280aa23df1ae71b43 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.568877 27693 raft_consensus.cc:383] T 5f27f525fdab407280aa23df1ae71b43 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.568949 27693 raft_consensus.cc:738] T 5f27f525fdab407280aa23df1ae71b43 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.569216 27693 consensus_queue.cc:260] T 5f27f525fdab407280aa23df1ae71b43 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.569770 27693 ts_tablet_manager.cc:1428] T 5f27f525fdab407280aa23df1ae71b43 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:29.570078 27693 tablet_bootstrap.cc:492] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:29.573138 27684 ts_tablet_manager.cc:1918] T 283a3afed7414a958b2dc4176f00bb01 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:29.577081 27684 log.cc:1199] T 283a3afed7414a958b2dc4176f00bb01 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/283a3afed7414a958b2dc4176f00bb01
I20250903 20:05:29.578121 27684 ts_tablet_manager.cc:1939] T 283a3afed7414a958b2dc4176f00bb01 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:29.578577 27695 tablet_bootstrap.cc:492] T 2fed2821d2644f75aff6e0ceb5944a37 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:29.578723 27695 ts_tablet_manager.cc:1397] T 2fed2821d2644f75aff6e0ceb5944a37 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20250903 20:05:29.578742 27693 tablet_bootstrap.cc:654] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.579489 27695 raft_consensus.cc:357] T 2fed2821d2644f75aff6e0ceb5944a37 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:29.579655 27695 raft_consensus.cc:383] T 2fed2821d2644f75aff6e0ceb5944a37 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.579715 27695 raft_consensus.cc:738] T 2fed2821d2644f75aff6e0ceb5944a37 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.580031 27695 consensus_queue.cc:260] T 2fed2821d2644f75aff6e0ceb5944a37 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:29.580693 27695 ts_tablet_manager.cc:1428] T 2fed2821d2644f75aff6e0ceb5944a37 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:29.566993 28848 tablet.cc:1722] T d859c0222c3647dab3dddd105e4cf0fc P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250903 20:05:29.581015 27695 tablet_bootstrap.cc:492] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:29.583114 28746 consensus_queue.cc:1035] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:29.584930 27695 tablet_bootstrap.cc:654] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.594390 28745 raft_consensus.cc:491] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:29.594856 27693 tablet_bootstrap.cc:492] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:29.594610 28745 raft_consensus.cc:513] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.595155 27693 ts_tablet_manager.cc:1397] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.000s
I20250903 20:05:29.595806 28745 leader_election.cc:290] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:29.595971 27693 raft_consensus.cc:357] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:29.596011 28745 consensus_queue.cc:1035] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:29.596870 27693 raft_consensus.cc:383] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.597072 27693 raft_consensus.cc:738] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.597386 27693 consensus_queue.cc:260] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:29.595806 28808 raft_consensus.cc:491] T 9e98cd1d9bd246e182e2969d147e68bf P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:29.597805 28808 raft_consensus.cc:513] T 9e98cd1d9bd246e182e2969d147e68bf P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.598585 27693 ts_tablet_manager.cc:1428] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250903 20:05:29.598872 27693 tablet_bootstrap.cc:492] T b7a6d5d8a97a4769a81ec7224d591f8c P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:29.598860 28808 leader_election.cc:290] T 9e98cd1d9bd246e182e2969d147e68bf P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:29.600136 28808 consensus_queue.cc:1035] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:29.600579 27693 tablet_bootstrap.cc:654] T b7a6d5d8a97a4769a81ec7224d591f8c P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:29.600896 27184 consensus_peers.cc:489] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53881ad30bc843839008ace4012a3f70. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:29.601258 27184 leader_election.cc:304] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:29.603140 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2cc35f4244a42eeb4a65de749824c11" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:29.603435 27526 raft_consensus.cc:2466] T f2cc35f4244a42eeb4a65de749824c11 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:29.605089 27183 leader_election.cc:304] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:29.601933 28808 raft_consensus.cc:2802] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:29.606938 28808 raft_consensus.cc:491] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:29.607072 28808 raft_consensus.cc:3058] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:29.607385 28655 raft_consensus.cc:2802] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250903 20:05:29.607441 28655 raft_consensus.cc:491] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:29.607491 28655 raft_consensus.cc:3058] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Advancing to term 2
I20250903 20:05:29.609380 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e98cd1d9bd246e182e2969d147e68bf" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:29.610617 27183 leader_election.cc:343] T 9e98cd1d9bd246e182e2969d147e68bf P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:29.612663 28770 raft_consensus.cc:491] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:29.612818 28770 raft_consensus.cc:513] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.613533 28770 leader_election.cc:290] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:29.614426 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4af8c03c9d834405b4e25d35441db961" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:29.614713 27249 raft_consensus.cc:2466] T 4af8c03c9d834405b4e25d35441db961 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:29.614789 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4af8c03c9d834405b4e25d35441db961" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:29.615332 27320 leader_election.cc:343] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:29.615499 27321 leader_election.cc:304] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:29.615919 28770 raft_consensus.cc:2802] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:29.616006 28770 raft_consensus.cc:491] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:29.616067 28770 raft_consensus.cc:3058] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:29.622623 27692 tablet_bootstrap.cc:492] T 0a9b7770432942da9373d5100c71bd6a P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:29.623063 27692 ts_tablet_manager.cc:1397] T 0a9b7770432942da9373d5100c71bd6a P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.073s user 0.004s sys 0.003s
I20250903 20:05:29.624009 27692 raft_consensus.cc:357] T 0a9b7770432942da9373d5100c71bd6a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.624276 27692 raft_consensus.cc:383] T 0a9b7770432942da9373d5100c71bd6a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.624368 27692 raft_consensus.cc:738] T 0a9b7770432942da9373d5100c71bd6a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.624622 27692 consensus_queue.cc:260] T 0a9b7770432942da9373d5100c71bd6a P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.625239 27692 ts_tablet_manager.cc:1428] T 0a9b7770432942da9373d5100c71bd6a P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:29.625774 27692 tablet_bootstrap.cc:492] T 607c9a9cbc5540e5bd0fa436fcba564c P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:29.628413 27692 tablet_bootstrap.cc:654] T 607c9a9cbc5540e5bd0fa436fcba564c P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.632685 27692 tablet_bootstrap.cc:492] T 607c9a9cbc5540e5bd0fa436fcba564c P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:29.632855 27692 ts_tablet_manager.cc:1397] T 607c9a9cbc5540e5bd0fa436fcba564c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20250903 20:05:29.634505 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_8ca1972ab99a4f4db9225a77b8d5ebc4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:29.634794 27692 raft_consensus.cc:357] T 607c9a9cbc5540e5bd0fa436fcba564c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.634959 27692 raft_consensus.cc:383] T 607c9a9cbc5540e5bd0fa436fcba564c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.635008 27692 raft_consensus.cc:738] T 607c9a9cbc5540e5bd0fa436fcba564c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
W20250903 20:05:29.635202 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ca1972ab99a4f4db9225a77b8d5ebc4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:29.635216 27692 consensus_queue.cc:260] T 607c9a9cbc5540e5bd0fa436fcba564c P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.635763 27692 ts_tablet_manager.cc:1428] T 607c9a9cbc5540e5bd0fa436fcba564c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250903 20:05:29.636062 27692 tablet_bootstrap.cc:492] T 9e98cd1d9bd246e182e2969d147e68bf P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:29.636821 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_647a8fb3a7d74bd48d49b76c3266b37f" } new_table_name: "default.Table_99db6c94d5964e95921925a5bcbc7f0e" modify_external_catalogs: true
I20250903 20:05:29.638331 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_c9acca1de7954498ab2dcc3ee1ff2f45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:29.638947 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9acca1de7954498ab2dcc3ee1ff2f45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:29.641673 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet cc1bc205bf39447bb9c3c7863a0959cf
I20250903 20:05:29.643836 27527 raft_consensus.cc:2466] T e0347eab6c8642beabdc2d7e7bbfdc9b P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:29.646138 27692 tablet_bootstrap.cc:654] T 9e98cd1d9bd246e182e2969d147e68bf P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.647461 28596 raft_consensus.cc:2747] T 466e501e424e46bc98e8bbc64f97005e P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:29.650738 27183 leader_election.cc:304] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:29.652079 28746 raft_consensus.cc:2802] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:29.652282 28746 raft_consensus.cc:695] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:29.652573 28746 consensus_queue.cc:237] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.653112 27379 raft_consensus.cc:2466] T e0347eab6c8642beabdc2d7e7bbfdc9b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:29.653498 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet cc1bc205bf39447bb9c3c7863a0959cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:29 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:29.654050 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d4af3ffb74834cccbbbee39223164dab" } modify_external_catalogs: true
I20250903 20:05:29.654304 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d4af3ffb74834cccbbbee39223164dab" } modify_external_catalogs: true
I20250903 20:05:29.654817 28751 raft_consensus.cc:491] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:29.654953 28751 raft_consensus.cc:513] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:29.655648 28751 leader_election.cc:290] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:29.656749 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "726f4d920f3e47fea41b4e12c021c988" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:29.657262 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "726f4d920f3e47fea41b4e12c021c988" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:29.657826 27321 leader_election.cc:343] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:29.658237 27320 leader_election.cc:343] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:29.658336 27320 leader_election.cc:304] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:29.658605 27240 raft_consensus.cc:2466] T f9709dfd18654c5bb071e39da7c8889f P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:29.662439 27321 leader_election.cc:304] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:29.662896 28567 raft_consensus.cc:2747] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:29.663103 28567 raft_consensus.cc:2802] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:29.663203 28567 raft_consensus.cc:695] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:29.663447 28567 consensus_queue.cc:237] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.664856 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet cc1bc205bf39447bb9c3c7863a0959cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:29 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:29.667515 27687 ts_tablet_manager.cc:1918] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:29.667675 27687 log.cc:1199] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/1bad3c509460422396a17a3c4ae3d1e8
I20250903 20:05:29.668035 27687 ts_tablet_manager.cc:1939] T 1bad3c509460422396a17a3c4ae3d1e8 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:29.674300 27377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e98cd1d9bd246e182e2969d147e68bf" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:29.687809 27692 tablet_bootstrap.cc:492] T 9e98cd1d9bd246e182e2969d147e68bf P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:29.688019 27692 ts_tablet_manager.cc:1397] T 9e98cd1d9bd246e182e2969d147e68bf P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.005s
I20250903 20:05:29.691147 27692 raft_consensus.cc:357] T 9e98cd1d9bd246e182e2969d147e68bf P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.691350 27692 raft_consensus.cc:383] T 9e98cd1d9bd246e182e2969d147e68bf P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.691430 27692 raft_consensus.cc:738] T 9e98cd1d9bd246e182e2969d147e68bf P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.691642 27692 consensus_queue.cc:260] T 9e98cd1d9bd246e182e2969d147e68bf P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.695756 27692 ts_tablet_manager.cc:1428] T 9e98cd1d9bd246e182e2969d147e68bf P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250903 20:05:29.696111 27692 tablet_bootstrap.cc:492] T 20c25c84c6fb4609a1964a4ff3c526da P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:29.697803 27692 tablet_bootstrap.cc:654] T 20c25c84c6fb4609a1964a4ff3c526da P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.708539 27693 tablet_bootstrap.cc:492] T b7a6d5d8a97a4769a81ec7224d591f8c P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:29.708761 27693 ts_tablet_manager.cc:1397] T b7a6d5d8a97a4769a81ec7224d591f8c P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.110s user 0.008s sys 0.018s
I20250903 20:05:29.709811 27693 raft_consensus.cc:357] T b7a6d5d8a97a4769a81ec7224d591f8c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:29.714278 28596 raft_consensus.cc:491] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:29.714537 28596 raft_consensus.cc:513] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.715235 28596 leader_election.cc:290] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:29.717132 27462 leader_election.cc:343] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: dbfdab4332234b2a93fd3c901a9cb645
W20250903 20:05:29.719440 27462 leader_election.cc:343] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: dbfdab4332234b2a93fd3c901a9cb645
I20250903 20:05:29.719626 27462 leader_election.cc:304] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:29.720045 28596 raft_consensus.cc:2747] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:29.710008 27693 raft_consensus.cc:383] T b7a6d5d8a97a4769a81ec7224d591f8c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.723783 27693 raft_consensus.cc:738] T b7a6d5d8a97a4769a81ec7224d591f8c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.724838 27692 tablet_bootstrap.cc:492] T 20c25c84c6fb4609a1964a4ff3c526da P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:29.725028 27692 ts_tablet_manager.cc:1397] T 20c25c84c6fb4609a1964a4ff3c526da P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.002s
I20250903 20:05:29.726097 27693 consensus_queue.cc:260] T b7a6d5d8a97a4769a81ec7224d591f8c P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:29.726749 27693 ts_tablet_manager.cc:1428] T b7a6d5d8a97a4769a81ec7224d591f8c P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20250903 20:05:29.727217 27693 tablet_bootstrap.cc:492] T 512908bc4d6941ec81b145ee5795405f P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:29.728821 27693 tablet_bootstrap.cc:654] T 512908bc4d6941ec81b145ee5795405f P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.725778 27692 raft_consensus.cc:357] T 20c25c84c6fb4609a1964a4ff3c526da P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.729738 27692 raft_consensus.cc:383] T 20c25c84c6fb4609a1964a4ff3c526da P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.729815 27692 raft_consensus.cc:738] T 20c25c84c6fb4609a1964a4ff3c526da P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.730542 27692 consensus_queue.cc:260] T 20c25c84c6fb4609a1964a4ff3c526da P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.731315 27692 ts_tablet_manager.cc:1428] T 20c25c84c6fb4609a1964a4ff3c526da P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250903 20:05:29.740761 28596 raft_consensus.cc:491] T 0a9b7770432942da9373d5100c71bd6a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:29.740921 28596 raft_consensus.cc:513] T 0a9b7770432942da9373d5100c71bd6a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.741464 28596 leader_election.cc:290] T 0a9b7770432942da9373d5100c71bd6a P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:29.742272 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0a9b7770432942da9373d5100c71bd6a" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:29.742667 27379 raft_consensus.cc:2391] T 0a9b7770432942da9373d5100c71bd6a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 89f8f0ee68b940db9ce250c0604f934a in this term.
I20250903 20:05:29.743347 28770 raft_consensus.cc:513] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.743908 28770 leader_election.cc:290] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:29.744251 27239 raft_consensus.cc:2466] T 90baf83357494a0aae325908c118b471 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:29.744626 28761 raft_consensus.cc:2747] T a4f3230f50ec4c95b528fead8c191224 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:29.745496 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 869) took 2496 ms. Trace:
I20250903 20:05:29.745592 27687 rpcz_store.cc:276] 0903 20:05:27.249275 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.249714 (+ 439us) service_pool.cc:225] Handling call
0903 20:05:29.450573 (+2200859us) ts_tablet_manager.cc:1145] Deleting tablet 1bad3c509460422396a17a3c4ae3d1e8
0903 20:05:29.450586 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:29.567824 (+117238us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:29.666417 (+ 98593us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:29.745474 (+ 79057us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2200682}
I20250903 20:05:29.745936 27687 tablet_replica.cc:331] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:29.746122 27687 raft_consensus.cc:2241] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:29.749564 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 809) took 3605 ms. Trace:
I20250903 20:05:29.749680 27684 rpcz_store.cc:276] 0903 20:05:26.143983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.145777 (+ 1794us) service_pool.cc:225] Handling call
0903 20:05:29.365747 (+3219970us) ts_tablet_manager.cc:1145] Deleting tablet 283a3afed7414a958b2dc4176f00bb01
0903 20:05:29.365760 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:29.467435 (+101675us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:29.573015 (+105580us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:29.749536 (+176521us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3219805}
I20250903 20:05:29.749992 27684 tablet_replica.cc:331] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:29.750144 27684 raft_consensus.cc:2241] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:29.750316 27684 raft_consensus.cc:2270] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:29.750798 27684 ts_tablet_manager.cc:1905] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:29.754706 28770 raft_consensus.cc:491] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:29.754920 28770 raft_consensus.cc:513] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.755728 28770 leader_election.cc:290] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:29.757575 27687 raft_consensus.cc:2270] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:29.757576 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 283a3afed7414a958b2dc4176f00bb01 (table default.Table_37d794e122f2457294c1eb6115be9472 [id=2ed5c84cc5a3466ba8a8cfeda62b643e]) successfully deleted
I20250903 20:05:29.759393 27687 ts_tablet_manager.cc:1905] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:29.759600 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 716) took 5980 ms. Trace:
I20250903 20:05:29.760535 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9f43781299e47a8aecc37a0b73d7a64" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:29.761157 27320 leader_election.cc:343] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:29.764362 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0a9b7770432942da9373d5100c71bd6a" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:29.765013 27462 leader_election.cc:304] T 0a9b7770432942da9373d5100c71bd6a P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:29.766301 28761 raft_consensus.cc:3058] T 0a9b7770432942da9373d5100c71bd6a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:29.767385 27522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4af8c03c9d834405b4e25d35441db961" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
W20250903 20:05:29.767815 27320 leader_election.cc:343] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:29.769495 27692 tablet_bootstrap.cc:492] T a9f43781299e47a8aecc37a0b73d7a64 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:29.771097 27692 tablet_bootstrap.cc:654] T a9f43781299e47a8aecc37a0b73d7a64 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.771111 27685 rpcz_store.cc:276] 0903 20:05:23.778648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:23.779640 (+ 992us) service_pool.cc:225] Handling call
0903 20:05:29.388775 (+5609135us) ts_tablet_manager.cc:1145] Deleting tablet 45d9b6ff520348628d19671127b9426c
0903 20:05:29.388785 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:29.456559 (+ 67774us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:29.566625 (+110066us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:29.759577 (+192952us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5609023}
I20250903 20:05:29.771613 27685 tablet_replica.cc:331] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:29.772706 28869 consensus_queue.cc:1035] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250903 20:05:29.773566 28806 raft_consensus.cc:491] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:29.775435 27693 tablet_bootstrap.cc:492] T 512908bc4d6941ec81b145ee5795405f P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:29.775584 27693 ts_tablet_manager.cc:1397] T 512908bc4d6941ec81b145ee5795405f P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.002s
I20250903 20:05:29.776352 27693 raft_consensus.cc:357] T 512908bc4d6941ec81b145ee5795405f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:29.776540 27693 raft_consensus.cc:383] T 512908bc4d6941ec81b145ee5795405f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.776593 27693 raft_consensus.cc:738] T 512908bc4d6941ec81b145ee5795405f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.776790 27693 consensus_queue.cc:260] T 512908bc4d6941ec81b145ee5795405f P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:29.777261 27693 ts_tablet_manager.cc:1428] T 512908bc4d6941ec81b145ee5795405f P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:29.777540 27693 tablet_bootstrap.cc:492] T 1bf69315f07a470bb2b897ad7c590729 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:29.777725 28806 raft_consensus.cc:513] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.778707 28806 leader_election.cc:290] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:29.780616 28806 consensus_queue.cc:1035] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:29.785596 27517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f6cd4d475f147fe8dfcc7e3a73dedf8" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:29.787225 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet d1c96296a579440c887ba6c2a93f7204
I20250903 20:05:29.796808 28849 tablet.cc:1722] T 0ad64c699b294679b6fec37b5ad23528 P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250903 20:05:29.797354 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet d1c96296a579440c887ba6c2a93f7204 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:29 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:29.798187 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet d1c96296a579440c887ba6c2a93f7204 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:29 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:29.803392 28882 tablet.cc:1722] T 0ad64c699b294679b6fec37b5ad23528 P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250903 20:05:29.803812 27052 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 908e748d08f74ed2a3071312c95ec07c (table default.Table_9c69923fa15e405690f9dd166a271493 [id=0c3f0f04e2064efea6f5e9efb1bfc0aa], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 908e748d08f74ed2a3071312c95ec07c consensus state has no leader
I20250903 20:05:29.805449 27692 tablet_bootstrap.cc:492] T a9f43781299e47a8aecc37a0b73d7a64 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
W20250903 20:05:29.806798 27059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c50e887828514d15935312d2e573f8ff consensus state has no leader
I20250903 20:05:29.807528 27692 ts_tablet_manager.cc:1397] T a9f43781299e47a8aecc37a0b73d7a64 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.004s
I20250903 20:05:29.812119 27692 raft_consensus.cc:357] T a9f43781299e47a8aecc37a0b73d7a64 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.812363 27692 raft_consensus.cc:383] T a9f43781299e47a8aecc37a0b73d7a64 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.812420 27692 raft_consensus.cc:738] T a9f43781299e47a8aecc37a0b73d7a64 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.812702 27693 tablet_bootstrap.cc:654] T 1bf69315f07a470bb2b897ad7c590729 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.813135 27692 consensus_queue.cc:260] T a9f43781299e47a8aecc37a0b73d7a64 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.824271 27692 ts_tablet_manager.cc:1428] T a9f43781299e47a8aecc37a0b73d7a64 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20250903 20:05:29.824596 27692 tablet_bootstrap.cc:492] T f9709dfd18654c5bb071e39da7c8889f P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
W20250903 20:05:29.827554 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1598824a733a4a93bed043e81b86abf3 consensus state has no leader
I20250903 20:05:29.829283 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_17ef4b4f7c1845088dac85dd6d8f0c0c" } modify_external_catalogs: true
I20250903 20:05:29.829538 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_17ef4b4f7c1845088dac85dd6d8f0c0c" } modify_external_catalogs: true
I20250903 20:05:29.831027 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 45d9b6ff520348628d19671127b9426c (table default.Table_509afdab983d4eed97a5d124ff19ab3b [id=7a023e4098354d038d92f90d46fc5125]) successfully deleted
I20250903 20:05:29.831415 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f6cd4d475f147fe8dfcc7e3a73dedf8" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:29.831732 27383 raft_consensus.cc:2466] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:29.831933 27695 tablet_bootstrap.cc:492] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
W20250903 20:05:29.833258 27057 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 32164b4e88b94e1fbca2e714e6bddfc4 consensus state has no leader
I20250903 20:05:29.835448 27057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1afb82628b3049949cc9dfdd46071a9e" } new_table_name: "default.Table_e75420178eca41cf8ce8565c4b51d181" modify_external_catalogs: true
I20250903 20:05:29.836696 28885 tablet.cc:1722] T d859c0222c3647dab3dddd105e4cf0fc P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20250903 20:05:29.838675 27183 leader_election.cc:343] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:29.839141 27692 tablet_bootstrap.cc:654] T f9709dfd18654c5bb071e39da7c8889f P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.840811 27693 tablet_bootstrap.cc:492] T 1bf69315f07a470bb2b897ad7c590729 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:29.840976 27693 ts_tablet_manager.cc:1397] T 1bf69315f07a470bb2b897ad7c590729 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.000s
I20250903 20:05:29.841895 27693 raft_consensus.cc:357] T 1bf69315f07a470bb2b897ad7c590729 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:29.842173 27693 raft_consensus.cc:383] T 1bf69315f07a470bb2b897ad7c590729 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.842305 27693 raft_consensus.cc:738] T 1bf69315f07a470bb2b897ad7c590729 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.839653 27685 raft_consensus.cc:2241] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:29.842700 27685 raft_consensus.cc:2270] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:29.843308 27685 ts_tablet_manager.cc:1905] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:29.847754 27518 raft_consensus.cc:2466] T 7cc6cf579e2642e99fefed723b0ee900 P e72ef1dbcc5143cf9f04a808f101c80f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 2.
I20250903 20:05:29.847713 28655 raft_consensus.cc:513] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.847482 27695 ts_tablet_manager.cc:1397] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.251s user 0.013s sys 0.002s
I20250903 20:05:29.852011 27321 leader_election.cc:304] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:29.852640 28770 raft_consensus.cc:2802] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:29.852797 28770 raft_consensus.cc:695] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:29.853140 28770 consensus_queue.cc:237] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.853926 28761 raft_consensus.cc:2747] T 0a9b7770432942da9373d5100c71bd6a P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:29.858280 27184 leader_election.cc:343] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 7cc6cf579e2642e99fefed723b0ee900
W20250903 20:05:29.858562 27184 leader_election.cc:343] T 9e98cd1d9bd246e182e2969d147e68bf P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:29.858651 27184 leader_election.cc:304] T 9e98cd1d9bd246e182e2969d147e68bf P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:29.859036 27184 leader_election.cc:343] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: f2cc35f4244a42eeb4a65de749824c11
I20250903 20:05:29.859515 28745 consensus_queue.cc:1035] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:29.864249 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b418b651df1e4007a0cb50170fc85a46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:29.865141 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b418b651df1e4007a0cb50170fc85a46 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:29.842563 27693 consensus_queue.cc:260] T 1bf69315f07a470bb2b897ad7c590729 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:29.869916 27693 ts_tablet_manager.cc:1428] T 1bf69315f07a470bb2b897ad7c590729 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20250903 20:05:29.870303 27693 tablet_bootstrap.cc:492] T 2059ccbd6d884085886d7d564afe03aa P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:29.871604 28850 consensus_queue.cc:1035] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.037s
W20250903 20:05:29.872740 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eb878263eda9447e870a782907fe6a8d consensus state has no leader
I20250903 20:05:29.874697 28655 leader_election.cc:290] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:29.875669 28808 raft_consensus.cc:513] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.876386 28808 leader_election.cc:290] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:29.877369 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4af8c03c9d834405b4e25d35441db961" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:29.877552 27239 raft_consensus.cc:3058] T 4af8c03c9d834405b4e25d35441db961 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:29.877846 28746 raft_consensus.cc:2747] T 9e98cd1d9bd246e182e2969d147e68bf P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:29.878754 27692 tablet_bootstrap.cc:492] T f9709dfd18654c5bb071e39da7c8889f P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:29.878887 27692 ts_tablet_manager.cc:1397] T f9709dfd18654c5bb071e39da7c8889f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.054s user 0.001s sys 0.003s
I20250903 20:05:29.879595 27692 raft_consensus.cc:357] T f9709dfd18654c5bb071e39da7c8889f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.879755 27692 raft_consensus.cc:383] T f9709dfd18654c5bb071e39da7c8889f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.879796 27692 raft_consensus.cc:738] T f9709dfd18654c5bb071e39da7c8889f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.880028 27692 consensus_queue.cc:260] T f9709dfd18654c5bb071e39da7c8889f P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.880663 27692 ts_tablet_manager.cc:1428] T f9709dfd18654c5bb071e39da7c8889f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:29.880921 27692 tablet_bootstrap.cc:492] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:29.888157 27692 tablet_bootstrap.cc:654] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.893326 28567 raft_consensus.cc:491] T 1bf69315f07a470bb2b897ad7c590729 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:29.893556 28567 raft_consensus.cc:513] T 1bf69315f07a470bb2b897ad7c590729 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:29.894510 28567 leader_election.cc:290] T 1bf69315f07a470bb2b897ad7c590729 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:29.897068 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bf69315f07a470bb2b897ad7c590729" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:29.897756 27320 leader_election.cc:343] T 1bf69315f07a470bb2b897ad7c590729 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:29.904839 27693 tablet_bootstrap.cc:654] T 2059ccbd6d884085886d7d564afe03aa P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.909785 27692 tablet_bootstrap.cc:492] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:29.909956 27692 ts_tablet_manager.cc:1397] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.001s
I20250903 20:05:29.910768 27692 raft_consensus.cc:357] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.910951 27692 raft_consensus.cc:383] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.911006 27692 raft_consensus.cc:738] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.911229 27692 consensus_queue.cc:260] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.911746 27692 ts_tablet_manager.cc:1428] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:29.912344 27692 tablet_bootstrap.cc:492] T 4af8c03c9d834405b4e25d35441db961 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:29.912388 27695 raft_consensus.cc:357] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:29.912819 27695 raft_consensus.cc:383] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.912884 27695 raft_consensus.cc:738] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.914088 27692 tablet_bootstrap.cc:654] T 4af8c03c9d834405b4e25d35441db961 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.916296 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9f43781299e47a8aecc37a0b73d7a64" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:29.916584 27240 raft_consensus.cc:2466] T a9f43781299e47a8aecc37a0b73d7a64 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:29.921115 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bf69315f07a470bb2b897ad7c590729" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:29.922097 27321 leader_election.cc:304] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:29.922914 28567 raft_consensus.cc:2802] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:29.923065 28567 raft_consensus.cc:491] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:29.923173 28567 raft_consensus.cc:3058] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:29.925635 28761 raft_consensus.cc:491] T 90baf83357494a0aae325908c118b471 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:29.925791 28761 raft_consensus.cc:513] T 90baf83357494a0aae325908c118b471 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.926441 28761 leader_election.cc:290] T 90baf83357494a0aae325908c118b471 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:29.927246 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90baf83357494a0aae325908c118b471" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:29.927596 27249 raft_consensus.cc:2391] T 90baf83357494a0aae325908c118b471 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
I20250903 20:05:29.928436 28761 raft_consensus.cc:491] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:29.928519 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90baf83357494a0aae325908c118b471" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:29.928603 28761 raft_consensus.cc:513] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.929358 28761 leader_election.cc:290] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:29.929455 27462 leader_election.cc:304] T 90baf83357494a0aae325908c118b471 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:29.929812 28761 raft_consensus.cc:3058] T 90baf83357494a0aae325908c118b471 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:29.930866 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6788b1683f984949816edfc5b68de8db" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:29.931123 27249 raft_consensus.cc:2466] T 6788b1683f984949816edfc5b68de8db P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
W20250903 20:05:29.933130 27462 leader_election.cc:343] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 6788b1683f984949816edfc5b68de8db
I20250903 20:05:29.933455 27462 leader_election.cc:304] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:29.933789 28596 raft_consensus.cc:2802] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250903 20:05:29.933871 28596 raft_consensus.cc:491] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:29.933950 28596 raft_consensus.cc:3058] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Advancing to term 2
I20250903 20:05:29.940761 27693 tablet_bootstrap.cc:492] T 2059ccbd6d884085886d7d564afe03aa P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:29.940973 27693 ts_tablet_manager.cc:1397] T 2059ccbd6d884085886d7d564afe03aa P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.071s user 0.006s sys 0.008s
I20250903 20:05:29.941975 27693 raft_consensus.cc:357] T 2059ccbd6d884085886d7d564afe03aa P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:29.942315 27693 raft_consensus.cc:383] T 2059ccbd6d884085886d7d564afe03aa P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:29.942406 27693 raft_consensus.cc:738] T 2059ccbd6d884085886d7d564afe03aa P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:29.942708 27693 consensus_queue.cc:260] T 2059ccbd6d884085886d7d564afe03aa P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:29.942610 27695 consensus_queue.cc:260] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:29.943269 27693 ts_tablet_manager.cc:1428] T 2059ccbd6d884085886d7d564afe03aa P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:29.943616 27693 tablet_bootstrap.cc:492] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:29.947621 27693 tablet_bootstrap.cc:654] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:29.951978 28761 raft_consensus.cc:2747] T 90baf83357494a0aae325908c118b471 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:29.951957 27239 raft_consensus.cc:2466] T 4af8c03c9d834405b4e25d35441db961 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:29.952492 28567 raft_consensus.cc:513] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.952831 27687 ts_tablet_manager.cc:1918] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:29.952966 27687 log.cc:1199] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/6f8cfe53b91b41d7ba7f9074e1005af3
I20250903 20:05:29.953367 27687 ts_tablet_manager.cc:1939] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:29.953866 28567 leader_election.cc:290] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:29.954861 28596 raft_consensus.cc:513] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.955524 28596 leader_election.cc:290] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 2 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:29.958583 27685 ts_tablet_manager.cc:1918] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:29.958734 27685 log.cc:1199] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/7cc6cf579e2642e99fefed723b0ee900
I20250903 20:05:29.959125 27685 ts_tablet_manager.cc:1939] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
W20250903 20:05:29.960454 27462 leader_election.cc:343] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 6788b1683f984949816edfc5b68de8db
I20250903 20:05:29.960841 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6788b1683f984949816edfc5b68de8db" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:29.961103 27239 raft_consensus.cc:3058] T 6788b1683f984949816edfc5b68de8db P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Advancing to term 2
I20250903 20:05:29.962695 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9f43781299e47a8aecc37a0b73d7a64" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:29.962873 27518 raft_consensus.cc:3058] T a9f43781299e47a8aecc37a0b73d7a64 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:29.972128 27321 leader_election.cc:343] T 1bf69315f07a470bb2b897ad7c590729 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:29.972359 27321 leader_election.cc:304] T 1bf69315f07a470bb2b897ad7c590729 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:29.973717 28874 raft_consensus.cc:491] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:29.973958 28874 raft_consensus.cc:513] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:29.975659 28874 leader_election.cc:290] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:29.976058 28567 raft_consensus.cc:2747] T 1bf69315f07a470bb2b897ad7c590729 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:29.976858 27695 ts_tablet_manager.cc:1428] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.128s user 0.002s sys 0.000s
I20250903 20:05:29.987777 28874 raft_consensus.cc:491] T 2fed2821d2644f75aff6e0ceb5944a37 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:29.988049 28874 raft_consensus.cc:513] T 2fed2821d2644f75aff6e0ceb5944a37 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:29.988974 28874 leader_election.cc:290] T 2fed2821d2644f75aff6e0ceb5944a37 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:30.007721 27321 leader_election.cc:304] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:30.010476 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9f43781299e47a8aecc37a0b73d7a64" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:30.010752 27249 raft_consensus.cc:3058] T a9f43781299e47a8aecc37a0b73d7a64 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:30.012136 28770 raft_consensus.cc:2802] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:30.012418 28770 raft_consensus.cc:695] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:30.012786 28770 consensus_queue.cc:237] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:30.013067 28848 tablet.cc:1722] T 0ad64c699b294679b6fec37b5ad23528 P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250903 20:05:30.022647 27695 tablet_bootstrap.cc:492] T 120ca61b3da742dc9a2efa483ead7e97 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:30.025029 27695 tablet_bootstrap.cc:654] T 120ca61b3da742dc9a2efa483ead7e97 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:30.047365 27692 tablet_bootstrap.cc:492] T 4af8c03c9d834405b4e25d35441db961 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:30.047535 27692 ts_tablet_manager.cc:1397] T 4af8c03c9d834405b4e25d35441db961 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.135s user 0.013s sys 0.027s
I20250903 20:05:30.048399 27692 raft_consensus.cc:357] T 4af8c03c9d834405b4e25d35441db961 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:30.048635 27692 raft_consensus.cc:383] T 4af8c03c9d834405b4e25d35441db961 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:30.048684 27692 raft_consensus.cc:738] T 4af8c03c9d834405b4e25d35441db961 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:30.048952 27692 consensus_queue.cc:260] T 4af8c03c9d834405b4e25d35441db961 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:30.048988 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 717) took 6230 ms. Trace:
I20250903 20:05:30.049149 27685 rpcz_store.cc:276] 0903 20:05:23.819678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:23.819832 (+ 154us) service_pool.cc:225] Handling call
0903 20:05:29.771510 (+5951678us) ts_tablet_manager.cc:1145] Deleting tablet 7cc6cf579e2642e99fefed723b0ee900
0903 20:05:29.771528 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:29.872468 (+100940us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:29.958445 (+ 85977us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.048962 (+ 90517us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":52197,"tablet-delete.queue_time_us":5951487}
I20250903 20:05:30.049503 27685 tablet_replica.cc:331] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:30.049546 27692 ts_tablet_manager.cc:1428] T 4af8c03c9d834405b4e25d35441db961 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:30.049688 27685 raft_consensus.cc:2241] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:30.049829 27685 raft_consensus.cc:2270] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:30.050143 27692 tablet_bootstrap.cc:492] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:30.050422 27685 ts_tablet_manager.cc:1905] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:30.050827 27684 ts_tablet_manager.cc:1918] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:30.050918 27684 log.cc:1199] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/dbfdab4332234b2a93fd3c901a9cb645
I20250903 20:05:30.051239 27684 ts_tablet_manager.cc:1939] T dbfdab4332234b2a93fd3c901a9cb645 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:30.052282 27518 raft_consensus.cc:2466] T a9f43781299e47a8aecc37a0b73d7a64 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:30.052407 27692 tablet_bootstrap.cc:654] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:30.052767 27320 leader_election.cc:304] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:30.053838 28567 raft_consensus.cc:2802] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:30.054082 28567 raft_consensus.cc:695] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:30.054447 28567 consensus_queue.cc:237] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:30.059231 27239 raft_consensus.cc:2466] T 6788b1683f984949816edfc5b68de8db P 89f8f0ee68b940db9ce250c0604f934a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 2.
I20250903 20:05:30.059839 27462 leader_election.cc:304] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:30.060317 28596 raft_consensus.cc:2802] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f [term 2 FOLLOWER]: Leader election won for term 2
I20250903 20:05:30.060438 28596 raft_consensus.cc:695] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f [term 2 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:30.060706 28596 consensus_queue.cc:237] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:30.064525 27057 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e6d019ca2fcd41309e74cdd3a1e20daf consensus state has no leader
I20250903 20:05:30.066301 28761 raft_consensus.cc:491] T 9e98cd1d9bd246e182e2969d147e68bf P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:30.066471 28761 raft_consensus.cc:513] T 9e98cd1d9bd246e182e2969d147e68bf P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:30.067167 28761 leader_election.cc:290] T 9e98cd1d9bd246e182e2969d147e68bf P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:30.067831 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e98cd1d9bd246e182e2969d147e68bf" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:30.068418 28761 raft_consensus.cc:491] T a72b85dfb34047528b2b28ff0d40f828 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:30.068531 28761 raft_consensus.cc:513] T a72b85dfb34047528b2b28ff0d40f828 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:30.068656 27462 leader_election.cc:343] T 9e98cd1d9bd246e182e2969d147e68bf P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:30.069092 28761 leader_election.cc:290] T a72b85dfb34047528b2b28ff0d40f828 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:30.069509 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a72b85dfb34047528b2b28ff0d40f828" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:30.069761 27379 raft_consensus.cc:2391] T a72b85dfb34047528b2b28ff0d40f828 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 89f8f0ee68b940db9ce250c0604f934a in this term.
I20250903 20:05:30.070647 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a72b85dfb34047528b2b28ff0d40f828" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:30.071177 27462 leader_election.cc:304] T a72b85dfb34047528b2b28ff0d40f828 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:30.071852 27063 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 0f5b87c1c4a24f8abdbfd6194f3b2c4a (table default.Table_6e9cf5ead5a24831a6a14d131dd8397e [id=524a27cbe1ad4c10aa2fd4f4074105c6]). Expected version 3 got 0
I20250903 20:05:30.072104 27063 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet a9f43781299e47a8aecc37a0b73d7a64 (table default.Table_73a4ac2afbad4654ac623cd581a45eb2 [id=f70d7ea93e154e6199f6766316353862]). Expected version 3 got 0
I20250903 20:05:30.072777 28761 raft_consensus.cc:3058] T a72b85dfb34047528b2b28ff0d40f828 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:30.071566 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e98cd1d9bd246e182e2969d147e68bf" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:30.077605 27239 raft_consensus.cc:2466] T 9e98cd1d9bd246e182e2969d147e68bf P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:30.078047 27462 leader_election.cc:304] T 9e98cd1d9bd246e182e2969d147e68bf P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:30.079015 28596 raft_consensus.cc:2802] T 9e98cd1d9bd246e182e2969d147e68bf P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:30.079088 28596 raft_consensus.cc:491] T 9e98cd1d9bd246e182e2969d147e68bf P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:30.079141 28596 raft_consensus.cc:3058] T 9e98cd1d9bd246e182e2969d147e68bf P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:30.081642 27064 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet b0d79c78e0024b7ba3bb3356be7c8ceb (table default.Table_94e9e32ef1134f07ade1cdf8da69ead6 [id=e248f9c4b9c9406395aaea7ba3feb9ba]). Expected version 1 got 0
I20250903 20:05:30.082093 27064 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet a9f43781299e47a8aecc37a0b73d7a64 (table default.Table_73a4ac2afbad4654ac623cd581a45eb2 [id=f70d7ea93e154e6199f6766316353862]). Expected version 3 got 0
I20250903 20:05:30.085984 27353 tablet_service.cc:1468] Processing CreateTablet for tablet 8eeada542d3744728095907ea224e282 (DEFAULT_TABLE table=default.Table_c51dfa63afbc4a9aabd9a43355859503 [id=ea7f6cdb93424c91bbb43b5bd98eda06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.086200 27350 tablet_service.cc:1468] Processing CreateTablet for tablet 3393da8d8d1148d2ba82c0b32728595d (DEFAULT_TABLE table=default.Table_158bc2760d3c4441840cb7f7eae71ff0 [id=32919fac502a44dc9227d86251187703]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.086627 27352 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bad3c509460422396a17a3c4ae3d1e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 13496efe34944b7280e7a684f180e6e0) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:30.086879 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet 607c9a9cbc5540e5bd0fa436fcba564c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:30.087545 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 810) took 3943 ms. Trace:
I20250903 20:05:30.087643 27684 rpcz_store.cc:276] 0903 20:05:26.144116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.145965 (+ 1849us) service_pool.cc:225] Handling call
0903 20:05:29.749935 (+3603970us) ts_tablet_manager.cc:1145] Deleting tablet dbfdab4332234b2a93fd3c901a9cb645
0903 20:05:29.749945 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:29.848175 (+ 98230us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.050698 (+202523us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.087516 (+ 36818us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3603819}
I20250903 20:05:30.088076 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 811) took 3925 ms. Trace:
I20250903 20:05:30.088142 27684 rpcz_store.cc:276] 0903 20:05:26.162639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.162715 (+ 76us) service_pool.cc:225] Handling call
0903 20:05:30.087948 (+3925233us) ts_tablet_manager.cc:1145] Deleting tablet 121a8ba102454cb4807483fac16e3781
0903 20:05:30.087957 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.088060 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3924994}
I20250903 20:05:30.088398 27684 tablet_replica.cc:331] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:30.088582 27684 raft_consensus.cc:2241] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:30.088550 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 8eeada542d3744728095907ea224e282 (DEFAULT_TABLE table=default.Table_c51dfa63afbc4a9aabd9a43355859503 [id=ea7f6cdb93424c91bbb43b5bd98eda06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.088934 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eeada542d3744728095907ea224e282. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.089609 27684 pending_rounds.cc:70] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f: Trying to abort 1 pending ops.
I20250903 20:05:30.089659 27684 pending_rounds.cc:77] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384990381248512 op_type: NO_OP noop_request { }
I20250903 20:05:30.089751 27684 raft_consensus.cc:2887] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:30.089828 27684 raft_consensus.cc:2270] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:30.090792 27684 ts_tablet_manager.cc:1905] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:30.092607 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 3393da8d8d1148d2ba82c0b32728595d (DEFAULT_TABLE table=default.Table_158bc2760d3c4441840cb7f7eae71ff0 [id=32919fac502a44dc9227d86251187703]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.092969 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3393da8d8d1148d2ba82c0b32728595d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.090822 27692 tablet_bootstrap.cc:492] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:30.093727 27692 ts_tablet_manager.cc:1397] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.002s
I20250903 20:05:30.094646 27692 raft_consensus.cc:357] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.094800 27692 raft_consensus.cc:383] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:30.094844 27692 raft_consensus.cc:738] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:30.095093 27692 consensus_queue.cc:260] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.095572 27692 ts_tablet_manager.cc:1428] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:30.095856 27692 tablet_bootstrap.cc:492] T 2fed2821d2644f75aff6e0ceb5944a37 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:30.097626 27692 tablet_bootstrap.cc:654] T 2fed2821d2644f75aff6e0ceb5944a37 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:30.106173 27695 tablet_bootstrap.cc:492] T 120ca61b3da742dc9a2efa483ead7e97 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:30.106382 27695 ts_tablet_manager.cc:1397] T 120ca61b3da742dc9a2efa483ead7e97 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.084s user 0.003s sys 0.002s
I20250903 20:05:30.107084 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet 90baf83357494a0aae325908c118b471 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:30.107225 27695 raft_consensus.cc:357] T 120ca61b3da742dc9a2efa483ead7e97 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.107405 27695 raft_consensus.cc:383] T 120ca61b3da742dc9a2efa483ead7e97 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:30.107455 27695 raft_consensus.cc:738] T 120ca61b3da742dc9a2efa483ead7e97 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:30.107594 27352 tablet_service.cc:1468] Processing CreateTablet for tablet ca722dfe38ac4e63b024508a3997cedb (DEFAULT_TABLE table=default.Table_7197fb56c22c40948ede1c2959901507 [id=7a408d6b802d4dfda68a647d3a38f805]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.107635 27695 consensus_queue.cc:260] T 120ca61b3da742dc9a2efa483ead7e97 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.108327 27695 ts_tablet_manager.cc:1428] T 120ca61b3da742dc9a2efa483ead7e97 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:30.108448 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet dbfdab4332234b2a93fd3c901a9cb645 (table default.Table_7e582ad4ca9446c68da4850af02126ea [id=67cd1b33fab14606babd6ba4524a6966]) successfully deleted
I20250903 20:05:30.108757 27695 tablet_bootstrap.cc:492] T 2059ccbd6d884085886d7d564afe03aa P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:30.109272 27506 tablet_service.cc:1468] Processing CreateTablet for tablet ca722dfe38ac4e63b024508a3997cedb (DEFAULT_TABLE table=default.Table_7197fb56c22c40948ede1c2959901507 [id=7a408d6b802d4dfda68a647d3a38f805]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:30.110303 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 121a8ba102454cb4807483fac16e3781 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 121a8ba102454cb4807483fac16e3781 already in progress: creating tablet
I20250903 20:05:30.110885 27692 tablet_bootstrap.cc:492] T 2fed2821d2644f75aff6e0ceb5944a37 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:30.111068 27692 ts_tablet_manager.cc:1397] T 2fed2821d2644f75aff6e0ceb5944a37 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.001s
I20250903 20:05:30.111956 27059 master_service.cc:948] ReplaceTablet: received request to replace tablet e8bc99de4ffe43d58351d170b7c61d8a from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:30.112071 27692 raft_consensus.cc:357] T 2fed2821d2644f75aff6e0ceb5944a37 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.112282 27692 raft_consensus.cc:383] T 2fed2821d2644f75aff6e0ceb5944a37 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:30.112339 27692 raft_consensus.cc:738] T 2fed2821d2644f75aff6e0ceb5944a37 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:30.112625 27692 consensus_queue.cc:260] T 2fed2821d2644f75aff6e0ceb5944a37 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.113312 27692 ts_tablet_manager.cc:1428] T 2fed2821d2644f75aff6e0ceb5944a37 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:30.113610 27692 tablet_bootstrap.cc:492] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:30.114368 27489 tablet_service.cc:1515] Processing DeleteTablet for tablet f2629f46704e4a0fa7922533928ca2ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:30.114646 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet 90baf83357494a0aae325908c118b471 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:30.114792 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet 673c45f3f2a14da9bd306011d396eaca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:30.114877 27488 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f27f525fdab407280aa23df1ae71b43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:30.114957 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet a72b85dfb34047528b2b28ff0d40f828 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:30.115116 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet 466e501e424e46bc98e8bbc64f97005e with delete_type TABLET_DATA_DELETED (replaced by tablet 06a79706f1184fa49d10182ba669b0ab) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:30.117101 27359 tablet_service.cc:1468] Processing CreateTablet for tablet 6f90a544e3b1472491237845706cf803 (DEFAULT_TABLE table=default.Table_5b06e203567741abb0b8790b44715a44 [id=f8d6c4936659476c989fcdc5fd090594]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.117766 28710 raft_consensus.cc:491] T 673c45f3f2a14da9bd306011d396eaca P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:30.117908 28710 raft_consensus.cc:513] T 673c45f3f2a14da9bd306011d396eaca P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:30.118615 28710 leader_election.cc:290] T 673c45f3f2a14da9bd306011d396eaca P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:30.119341 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "673c45f3f2a14da9bd306011d396eaca" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:30.119624 27379 raft_consensus.cc:2391] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
I20250903 20:05:30.119911 27692 tablet_bootstrap.cc:654] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:30.120668 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "673c45f3f2a14da9bd306011d396eaca" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:30.120895 27239 raft_consensus.cc:2391] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 89f8f0ee68b940db9ce250c0604f934a in this term.
I20250903 20:05:30.121345 27462 leader_election.cc:304] T 673c45f3f2a14da9bd306011d396eaca P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:30.121888 28710 raft_consensus.cc:3058] T 673c45f3f2a14da9bd306011d396eaca P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:30.124017 27493 tablet_service.cc:1468] Processing CreateTablet for tablet 6f90a544e3b1472491237845706cf803 (DEFAULT_TABLE table=default.Table_5b06e203567741abb0b8790b44715a44 [id=f8d6c4936659476c989fcdc5fd090594]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.126966 28817 raft_consensus.cc:491] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:30.127005 27348 tablet_service.cc:1468] Processing CreateTablet for tablet c6a6b64e823243eea2bab26a9fc40190 (DEFAULT_TABLE table=default.Table_7eab6c911ff34aa7a4e2e94cd4087de4 [id=0f86928f520643b0a351c9d8f3cf0afd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.127115 28817 raft_consensus.cc:513] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:30.127698 28817 leader_election.cc:290] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:30.129052 27462 leader_election.cc:343] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 4157633525034a019fd6a80185892cec
I20250903 20:05:30.129179 27692 tablet_bootstrap.cc:492] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:30.129315 27692 ts_tablet_manager.cc:1397] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
W20250903 20:05:30.129870 27462 leader_election.cc:343] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 4157633525034a019fd6a80185892cec
I20250903 20:05:30.129980 27462 leader_election.cc:304] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:30.130391 28817 raft_consensus.cc:2747] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:30.131100 27693 tablet_bootstrap.cc:492] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:30.131283 27693 ts_tablet_manager.cc:1397] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.188s user 0.017s sys 0.029s
I20250903 20:05:30.131145 27692 raft_consensus.cc:357] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.132256 27693 raft_consensus.cc:357] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.132455 27693 raft_consensus.cc:383] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:30.132548 27693 raft_consensus.cc:738] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:30.132798 27693 consensus_queue.cc:260] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.132407 27499 tablet_service.cc:1468] Processing CreateTablet for tablet c6a6b64e823243eea2bab26a9fc40190 (DEFAULT_TABLE table=default.Table_7eab6c911ff34aa7a4e2e94cd4087de4 [id=0f86928f520643b0a351c9d8f3cf0afd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.132684 28817 raft_consensus.cc:491] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:30.133262 28817 raft_consensus.cc:513] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:30.133471 27693 ts_tablet_manager.cc:1428] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20250903 20:05:30.133833 28817 leader_election.cc:290] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:30.134649 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f27f525fdab407280aa23df1ae71b43" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:30.134812 27379 raft_consensus.cc:2466] T 5f27f525fdab407280aa23df1ae71b43 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:30.136152 27462 leader_election.cc:304] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:30.136651 28817 raft_consensus.cc:2802] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:30.136729 28817 raft_consensus.cc:491] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:30.136793 28817 raft_consensus.cc:3058] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:30.137473 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f27f525fdab407280aa23df1ae71b43" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:30.137706 27239 raft_consensus.cc:2466] T 5f27f525fdab407280aa23df1ae71b43 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:30.138598 27692 raft_consensus.cc:383] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:30.139303 27692 raft_consensus.cc:738] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:30.139613 27692 consensus_queue.cc:260] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.141032 27349 tablet_service.cc:1468] Processing CreateTablet for tablet 271131faf78e4008bb0060ebd15fa415 (DEFAULT_TABLE table=default.Table_ffe9c64658a24e47b91b43fe8ae41eba [id=cd3ebf08b0804a63a1077aa335160dc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.145978 27488 tablet_service.cc:1468] Processing CreateTablet for tablet 271131faf78e4008bb0060ebd15fa415 (DEFAULT_TABLE table=default.Table_ffe9c64658a24e47b91b43fe8ae41eba [id=cd3ebf08b0804a63a1077aa335160dc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.146642 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 3c47c5fca0ee4349b2667494f45af5d0 (DEFAULT_TABLE table=default.Table_8851a4dad8da4bb8907db1ff956dd28f [id=27dbd6d3bb504a82a84b666698c93299]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.146898 27494 tablet_service.cc:1468] Processing CreateTablet for tablet 3c47c5fca0ee4349b2667494f45af5d0 (DEFAULT_TABLE table=default.Table_8851a4dad8da4bb8907db1ff956dd28f [id=27dbd6d3bb504a82a84b666698c93299]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.148236 27347 tablet_service.cc:1468] Processing CreateTablet for tablet 9425f568adf944c2a1910f20ffc05352 (DEFAULT_TABLE table=default.Table_2b9db406a0804f1c8f25adb358424581 [id=6de596de893e474fa9dbf3627f6be37e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.149348 27495 tablet_service.cc:1468] Processing CreateTablet for tablet 9425f568adf944c2a1910f20ffc05352 (DEFAULT_TABLE table=default.Table_2b9db406a0804f1c8f25adb358424581 [id=6de596de893e474fa9dbf3627f6be37e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.150684 27365 tablet_service.cc:1468] Processing CreateTablet for tablet d9daec5ba727475bb4ce7eb531424421 (DEFAULT_TABLE table=default.Table_67b0c9f0d9164596a34c1f4a5a8cfe2f [id=ab99ffeb46d94924b97d7fdde265d09f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.151729 27489 tablet_service.cc:1468] Processing CreateTablet for tablet d9daec5ba727475bb4ce7eb531424421 (DEFAULT_TABLE table=default.Table_67b0c9f0d9164596a34c1f4a5a8cfe2f [id=ab99ffeb46d94924b97d7fdde265d09f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.152609 27249 raft_consensus.cc:2466] T a9f43781299e47a8aecc37a0b73d7a64 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:30.153036 28761 raft_consensus.cc:2747] T a72b85dfb34047528b2b28ff0d40f828 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:30.153095 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e6135c51a2164b27bf944da7e525f349"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:30.153712 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6135c51a2164b27bf944da7e525f349 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:30.155062 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_6cbf23397cb948a9981807cb7036eb12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:30.155608 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6cbf23397cb948a9981807cb7036eb12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:30.159185 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 870) took 2909 ms. Trace:
I20250903 20:05:30.159293 27687 rpcz_store.cc:276] 0903 20:05:27.249373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.249914 (+ 541us) service_pool.cc:225] Handling call
0903 20:05:29.745892 (+2495978us) ts_tablet_manager.cc:1145] Deleting tablet 6f8cfe53b91b41d7ba7f9074e1005af3
0903 20:05:29.745901 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:29.858134 (+112233us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:29.952677 (+ 94543us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.159137 (+206460us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2495840}
I20250903 20:05:30.159647 27687 tablet_replica.cc:331] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:30.159819 27687 raft_consensus.cc:2241] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:30.160127 27687 raft_consensus.cc:2270] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:30.160732 27687 ts_tablet_manager.cc:1905] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:30.162642 27353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eeada542d3744728095907ea224e282. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.163223 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a6b64e823243eea2bab26a9fc40190. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.163609 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 271131faf78e4008bb0060ebd15fa415. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.163956 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c47c5fca0ee4349b2667494f45af5d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.165442 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9daec5ba727475bb4ce7eb531424421. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.165732 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3393da8d8d1148d2ba82c0b32728595d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.166520 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e997a321ad0c45dc9258399326f0830d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:30.167189 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e997a321ad0c45dc9258399326f0830d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:30.168149 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca722dfe38ac4e63b024508a3997cedb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.168447 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9425f568adf944c2a1910f20ffc05352. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.169358 27695 tablet_bootstrap.cc:654] T 2059ccbd6d884085886d7d564afe03aa P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:30.170437 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f90a544e3b1472491237845706cf803. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.166149 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_14b6895698244e7097617536094efbd5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:30.179610 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14b6895698244e7097617536094efbd5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:30.186782 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 6f8cfe53b91b41d7ba7f9074e1005af3 (table default.Table_e5def563dd414101b9ee8fbaf7156e29 [id=7e67c0d1d2f34172a987a8ea785c5a6a]) successfully deleted
I20250903 20:05:30.187475 27494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c47c5fca0ee4349b2667494f45af5d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.188510 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca722dfe38ac4e63b024508a3997cedb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.188977 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9425f568adf944c2a1910f20ffc05352. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.189450 27489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9daec5ba727475bb4ce7eb531424421. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.189834 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 271131faf78e4008bb0060ebd15fa415. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.193286 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f90a544e3b1472491237845706cf803. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.193996 28898 raft_consensus.cc:491] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 35bdb0090d8747059b5ee103e786e979)
I20250903 20:05:30.194233 28898 raft_consensus.cc:513] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } }
I20250903 20:05:30.195096 28898 leader_election.cc:290] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8857b5d4ea1e4f45aafb8e029837cca2 (127.26.65.126:39015), 35bdb0090d8747059b5ee103e786e979 (127.26.65.124:40911)
I20250903 20:05:30.197329 27873 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" candidate_term: 4 candidate_status { last_received { term: 3 index: 1159 } } ignore_live_leader: false dest_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" is_pre_election: true
I20250903 20:05:30.197940 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a6b64e823243eea2bab26a9fc40190. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.200832 28902 raft_consensus.cc:491] T 20c25c84c6fb4609a1964a4ff3c526da P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:30.200968 28902 raft_consensus.cc:513] T 20c25c84c6fb4609a1964a4ff3c526da P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:30.201543 28902 leader_election.cc:290] T 20c25c84c6fb4609a1964a4ff3c526da P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:30.202028 27076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" candidate_term: 4 candidate_status { last_received { term: 3 index: 1159 } } ignore_live_leader: false dest_uuid: "35bdb0090d8747059b5ee103e786e979" is_pre_election: true
I20250903 20:05:30.203226 28317 leader_election.cc:304] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35dd69f3fa7a4fd8843b97f8a5d68296; no voters: 35bdb0090d8747059b5ee103e786e979, 8857b5d4ea1e4f45aafb8e029837cca2
I20250903 20:05:30.203496 28770 consensus_queue.cc:1035] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:30.203680 28898 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250903 20:05:30.204566 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20c25c84c6fb4609a1964a4ff3c526da" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:30.205322 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20c25c84c6fb4609a1964a4ff3c526da" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:30.206739 27462 leader_election.cc:304] T 20c25c84c6fb4609a1964a4ff3c526da P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:30.206826 27518 raft_consensus.cc:3058] T 20c25c84c6fb4609a1964a4ff3c526da P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:30.209558 27695 tablet_bootstrap.cc:492] T 2059ccbd6d884085886d7d564afe03aa P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:30.209712 27695 ts_tablet_manager.cc:1397] T 2059ccbd6d884085886d7d564afe03aa P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.101s user 0.003s sys 0.001s
I20250903 20:05:30.210536 27695 raft_consensus.cc:357] T 2059ccbd6d884085886d7d564afe03aa P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.210688 27695 raft_consensus.cc:383] T 2059ccbd6d884085886d7d564afe03aa P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:30.210732 27695 raft_consensus.cc:738] T 2059ccbd6d884085886d7d564afe03aa P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:30.214776 27684 ts_tablet_manager.cc:1918] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:30.214931 27684 log.cc:1199] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/17e0864a32d5452d84ac470b55fbaaf4
I20250903 20:05:30.215356 27684 ts_tablet_manager.cc:1939] T 17e0864a32d5452d84ac470b55fbaaf4 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:30.222528 27685 ts_tablet_manager.cc:1918] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:30.222720 27685 log.cc:1199] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/283a3afed7414a958b2dc4176f00bb01
I20250903 20:05:30.223064 27685 ts_tablet_manager.cc:1939] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:30.224551 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_06e3b212bc67490fbe6c44740bbaf704"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:30.225443 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06e3b212bc67490fbe6c44740bbaf704 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:30.227113 28710 raft_consensus.cc:2747] T 673c45f3f2a14da9bd306011d396eaca P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:30.236733 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 718) took 6416 ms. Trace:
I20250903 20:05:30.228608 27692 ts_tablet_manager.cc:1428] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.099s user 0.002s sys 0.001s
I20250903 20:05:30.236857 27685 rpcz_store.cc:276] 0903 20:05:23.820165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:23.820428 (+ 263us) service_pool.cc:225] Handling call
0903 20:05:30.049453 (+6229025us) ts_tablet_manager.cc:1145] Deleting tablet 283a3afed7414a958b2dc4176f00bb01
0903 20:05:30.049463 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.153412 (+103949us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.222394 (+ 68982us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.236708 (+ 14314us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6228889}
I20250903 20:05:30.237257 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 732) took 5923 ms. Trace:
I20250903 20:05:30.237341 27685 rpcz_store.cc:276] 0903 20:05:24.314316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:24.314392 (+ 76us) service_pool.cc:225] Handling call
0903 20:05:30.237130 (+5922738us) ts_tablet_manager.cc:1145] Deleting tablet aa31f2ddb9f84a8093aa1c6f64b278c9
0903 20:05:30.237139 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.237241 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5922489}
I20250903 20:05:30.237557 27692 tablet_bootstrap.cc:492] T 512908bc4d6941ec81b145ee5795405f P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:30.237635 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 733) took 5923 ms. Trace:
I20250903 20:05:30.237706 27685 rpcz_store.cc:276] 0903 20:05:24.314452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:24.314676 (+ 224us) service_pool.cc:225] Handling call
0903 20:05:30.237548 (+5922872us) ts_tablet_manager.cc:1145] Deleting tablet ebc222df5d1445fea88939e1152142b9
0903 20:05:30.237556 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.237622 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5922736}
I20250903 20:05:30.237990 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 734) took 5923 ms. Trace:
I20250903 20:05:30.242069 28902 raft_consensus.cc:2747] T 20c25c84c6fb4609a1964a4ff3c526da P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:30.241544 28710 raft_consensus.cc:491] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:30.242792 28710 raft_consensus.cc:513] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:30.243391 28710 leader_election.cc:290] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:30.243680 27695 consensus_queue.cc:260] T 2059ccbd6d884085886d7d564afe03aa P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.244441 27695 ts_tablet_manager.cc:1428] T 2059ccbd6d884085886d7d564afe03aa P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.035s user 0.002s sys 0.000s
I20250903 20:05:30.244721 27695 tablet_bootstrap.cc:492] T 22bc87d1a4804b6c80f77f9692500d63 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:30.246414 27695 tablet_bootstrap.cc:654] T 22bc87d1a4804b6c80f77f9692500d63 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:30.246534 28751 raft_consensus.cc:491] T 512908bc4d6941ec81b145ee5795405f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:30.246678 28751 raft_consensus.cc:513] T 512908bc4d6941ec81b145ee5795405f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.247294 28751 leader_election.cc:290] T 512908bc4d6941ec81b145ee5795405f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:30.229637 28817 raft_consensus.cc:513] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:30.247913 28817 leader_election.cc:290] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:30.248153 27184 leader_election.cc:304] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:30.248193 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512908bc4d6941ec81b145ee5795405f" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:30.248384 27184 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 373282us
W20250903 20:05:30.248761 27321 leader_election.cc:343] T 512908bc4d6941ec81b145ee5795405f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:30.249017 27219 tablet_service.cc:1515] Processing DeleteTablet for tablet cc1bc205bf39447bb9c3c7863a0959cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:29 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:30.249708 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet d1c96296a579440c887ba6c2a93f7204 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:29 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:30.250098 27222 tablet_service.cc:1468] Processing CreateTablet for tablet c6a6b64e823243eea2bab26a9fc40190 (DEFAULT_TABLE table=default.Table_7eab6c911ff34aa7a4e2e94cd4087de4 [id=0f86928f520643b0a351c9d8f3cf0afd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.250314 28746 raft_consensus.cc:2802] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:30.250393 28746 raft_consensus.cc:491] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:30.250469 28746 raft_consensus.cc:3058] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:30.250640 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a6b64e823243eea2bab26a9fc40190. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.250599 27227 tablet_service.cc:1468] Processing CreateTablet for tablet 9425f568adf944c2a1910f20ffc05352 (DEFAULT_TABLE table=default.Table_2b9db406a0804f1c8f25adb358424581 [id=6de596de893e474fa9dbf3627f6be37e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.250936 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9425f568adf944c2a1910f20ffc05352. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.251183 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 271131faf78e4008bb0060ebd15fa415 (DEFAULT_TABLE table=default.Table_ffe9c64658a24e47b91b43fe8ae41eba [id=cd3ebf08b0804a63a1077aa335160dc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.251498 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 271131faf78e4008bb0060ebd15fa415. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.252770 27224 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f27f525fdab407280aa23df1ae71b43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:30.253197 28746 raft_consensus.cc:513] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:30.253785 28746 leader_election.cc:290] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:30.254468 27228 tablet_service.cc:1468] Processing CreateTablet for tablet d9daec5ba727475bb4ce7eb531424421 (DEFAULT_TABLE table=default.Table_67b0c9f0d9164596a34c1f4a5a8cfe2f [id=ab99ffeb46d94924b97d7fdde265d09f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.254575 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 8eeada542d3744728095907ea224e282 (DEFAULT_TABLE table=default.Table_c51dfa63afbc4a9aabd9a43355859503 [id=ea7f6cdb93424c91bbb43b5bd98eda06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.254683 27213 tablet_service.cc:1468] Processing CreateTablet for tablet 6f90a544e3b1472491237845706cf803 (DEFAULT_TABLE table=default.Table_5b06e203567741abb0b8790b44715a44 [id=f8d6c4936659476c989fcdc5fd090594]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.254748 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet f2629f46704e4a0fa7922533928ca2ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:30.254828 27228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9daec5ba727475bb4ce7eb531424421. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.255041 27221 tablet_service.cc:1468] Processing CreateTablet for tablet ca722dfe38ac4e63b024508a3997cedb (DEFAULT_TABLE table=default.Table_7197fb56c22c40948ede1c2959901507 [id=7a408d6b802d4dfda68a647d3a38f805]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.255280 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet f9709dfd18654c5bb071e39da7c8889f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:17 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:30.250401 27229 tablet_service.cc:1468] Processing CreateTablet for tablet 3c47c5fca0ee4349b2667494f45af5d0 (DEFAULT_TABLE table=default.Table_8851a4dad8da4bb8907db1ff956dd28f [id=27dbd6d3bb504a82a84b666698c93299]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.254473 27219 tablet_service.cc:1468] Processing CreateTablet for tablet 3393da8d8d1148d2ba82c0b32728595d (DEFAULT_TABLE table=default.Table_158bc2760d3c4441840cb7f7eae71ff0 [id=32919fac502a44dc9227d86251187703]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.258920 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f90a544e3b1472491237845706cf803. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.261468 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "607c9a9cbc5540e5bd0fa436fcba564c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:30.261629 27386 raft_consensus.cc:3058] T 607c9a9cbc5540e5bd0fa436fcba564c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:30.262658 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca722dfe38ac4e63b024508a3997cedb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.263793 27382 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2fed2821d2644f75aff6e0ceb5944a37" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:30.265707 28874 raft_consensus.cc:491] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:30.265836 28874 raft_consensus.cc:513] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:30.266438 28874 leader_election.cc:290] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:30.266439 27229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c47c5fca0ee4349b2667494f45af5d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.266777 27382 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f6cd4d475f147fe8dfcc7e3a73dedf8" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:30.266928 27382 raft_consensus.cc:3058] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:30.268870 28874 raft_consensus.cc:491] T 2059ccbd6d884085886d7d564afe03aa P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:30.268982 28874 raft_consensus.cc:513] T 2059ccbd6d884085886d7d564afe03aa P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.269475 28874 leader_election.cc:290] T 2059ccbd6d884085886d7d564afe03aa P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
W20250903 20:05:30.269989 27183 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 431294us
I20250903 20:05:30.270329 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eeada542d3744728095907ea224e282. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.275036 27219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3393da8d8d1148d2ba82c0b32728595d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.275419 28870 consensus_queue.cc:1035] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:30.281131 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 7cc6cf579e2642e99fefed723b0ee900 (table default.Table_bffb6801f48c4124bfe24c49586d6e48 [id=aa020b54da65482ba59a71768c2bd03b]) successfully deleted
I20250903 20:05:30.281435 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 283a3afed7414a958b2dc4176f00bb01 (table default.Table_37d794e122f2457294c1eb6115be9472 [id=2ed5c84cc5a3466ba8a8cfeda62b643e]) successfully deleted
W20250903 20:05:30.281618 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet aa31f2ddb9f84a8093aa1c6f64b278c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa31f2ddb9f84a8093aa1c6f64b278c9 already in progress: creating tablet
W20250903 20:05:30.281797 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet ebc222df5d1445fea88939e1152142b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ebc222df5d1445fea88939e1152142b9 already in progress: creating tablet
I20250903 20:05:30.283102 27183 leader_election.cc:304] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
W20250903 20:05:30.283900 27183 raft_consensus.cc:2684] T 7cc6cf579e2642e99fefed723b0ee900 P 89f8f0ee68b940db9ce250c0604f934a: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250903 20:05:30.230037 28596 raft_consensus.cc:513] T 9e98cd1d9bd246e182e2969d147e68bf P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:30.288353 28596 leader_election.cc:290] T 9e98cd1d9bd246e182e2969d147e68bf P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:30.292227 28596 raft_consensus.cc:491] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:30.292371 28596 raft_consensus.cc:513] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.292896 28596 leader_election.cc:290] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:30.297418 27687 ts_tablet_manager.cc:1918] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:30.299515 27687 log.cc:1199] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/f0efbb7eefb84ab6bee5b9625b91e298
I20250903 20:05:30.238082 27685 rpcz_store.cc:276] 0903 20:05:24.314567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:24.315317 (+ 750us) service_pool.cc:225] Handling call
0903 20:05:30.237888 (+5922571us) ts_tablet_manager.cc:1145] Deleting tablet 5c42d84a6a5649208b0539a10da21aec
0903 20:05:30.237895 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.237976 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5922396}
I20250903 20:05:30.309167 27685 tablet_replica.cc:331] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:30.309429 27685 raft_consensus.cc:2241] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:30.309651 27685 raft_consensus.cc:2270] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:30.310256 27685 ts_tablet_manager.cc:1905] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:30.310601 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f6cd4d475f147fe8dfcc7e3a73dedf8" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:30.311005 27239 raft_consensus.cc:2391] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 89f8f0ee68b940db9ce250c0604f934a in this term.
I20250903 20:05:30.312506 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f27f525fdab407280aa23df1ae71b43" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:30.312848 27239 raft_consensus.cc:3058] T 5f27f525fdab407280aa23df1ae71b43 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:30.316829 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e98cd1d9bd246e182e2969d147e68bf" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:30.317044 27240 raft_consensus.cc:3058] T 9e98cd1d9bd246e182e2969d147e68bf P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:30.317267 28808 consensus_queue.cc:1035] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:30.317643 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b7a6d5d8a97a4769a81ec7224d591f8c" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:30.317869 27184 leader_election.cc:343] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: f2cc35f4244a42eeb4a65de749824c11
I20250903 20:05:30.319759 27240 raft_consensus.cc:2466] T 9e98cd1d9bd246e182e2969d147e68bf P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
W20250903 20:05:30.319902 27184 leader_election.cc:343] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 283a3afed7414a958b2dc4176f00bb01
W20250903 20:05:30.320091 27184 leader_election.cc:343] T 2fed2821d2644f75aff6e0ceb5944a37 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:30.320765 28808 consensus_queue.cc:1035] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250903 20:05:30.321427 27184 leader_election.cc:343] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 639ecb9caa4a423899ab8eb35bb716d7
W20250903 20:05:30.322751 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 5c42d84a6a5649208b0539a10da21aec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c42d84a6a5649208b0539a10da21aec already in progress: creating tablet
I20250903 20:05:30.332990 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f6cd4d475f147fe8dfcc7e3a73dedf8" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:30.333235 27516 raft_consensus.cc:3058] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:30.333379 27527 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2cc35f4244a42eeb4a65de749824c11" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:30.333537 27527 raft_consensus.cc:3058] T f2cc35f4244a42eeb4a65de749824c11 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Advancing to term 2
I20250903 20:05:30.333681 27525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2fed2821d2644f75aff6e0ceb5944a37" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:30.333837 27525 raft_consensus.cc:2466] T 2fed2821d2644f75aff6e0ceb5944a37 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:30.334278 27515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512908bc4d6941ec81b145ee5795405f" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:30.336141 27462 leader_election.cc:343] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:30.336354 27527 raft_consensus.cc:2466] T f2cc35f4244a42eeb4a65de749824c11 P e72ef1dbcc5143cf9f04a808f101c80f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 2.
I20250903 20:05:30.336549 27517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "607c9a9cbc5540e5bd0fa436fcba564c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:30.336681 27517 raft_consensus.cc:3058] T 607c9a9cbc5540e5bd0fa436fcba564c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:30.336891 27516 raft_consensus.cc:2466] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:30.339033 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2059ccbd6d884085886d7d564afe03aa" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:30.302076 27693 tablet_bootstrap.cc:492] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:30.302800 27386 raft_consensus.cc:2466] T 607c9a9cbc5540e5bd0fa436fcba564c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:30.305198 27382 raft_consensus.cc:2466] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:30.310053 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f6cd4d475f147fe8dfcc7e3a73dedf8" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:30.312150 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f27f525fdab407280aa23df1ae71b43" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:30.313496 28751 consensus_queue.cc:1035] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:30.346951 27385 raft_consensus.cc:3058] T 5f27f525fdab407280aa23df1ae71b43 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:30.346954 27381 raft_consensus.cc:2391] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 89f8f0ee68b940db9ce250c0604f934a in this term.
I20250903 20:05:30.314113 27377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e98cd1d9bd246e182e2969d147e68bf" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:30.316701 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b7a6d5d8a97a4769a81ec7224d591f8c" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:30.347311 27380 raft_consensus.cc:2466] T b7a6d5d8a97a4769a81ec7224d591f8c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:30.347913 27517 raft_consensus.cc:2466] T 607c9a9cbc5540e5bd0fa436fcba564c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:30.348344 27687 ts_tablet_manager.cc:1939] T f0efbb7eefb84ab6bee5b9625b91e298 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:30.349061 27693 tablet_bootstrap.cc:654] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:30.349166 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 866) took 3100 ms. Trace:
I20250903 20:05:30.349256 27687 rpcz_store.cc:276] 0903 20:05:27.249016 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.250079 (+ 1063us) service_pool.cc:225] Handling call
0903 20:05:30.159601 (+2909522us) ts_tablet_manager.cc:1145] Deleting tablet f0efbb7eefb84ab6bee5b9625b91e298
0903 20:05:30.159611 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.234761 (+ 75150us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.297300 (+ 62539us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.349148 (+ 51848us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2909336}
I20250903 20:05:30.349565 27687 tablet_replica.cc:331] T 59a1dd48d76241d7ba817070fbffd853 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:30.349795 27687 raft_consensus.cc:2241] T 59a1dd48d76241d7ba817070fbffd853 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:30.350127 27687 raft_consensus.cc:2270] T 59a1dd48d76241d7ba817070fbffd853 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:30.322861 27384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2059ccbd6d884085886d7d564afe03aa" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:30.350682 27384 raft_consensus.cc:2466] T 2059ccbd6d884085886d7d564afe03aa P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:30.351132 27385 raft_consensus.cc:2466] T 5f27f525fdab407280aa23df1ae71b43 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:30.356379 27239 raft_consensus.cc:2466] T 5f27f525fdab407280aa23df1ae71b43 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:30.336355 27462 leader_election.cc:304] T 9e98cd1d9bd246e182e2969d147e68bf P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
W20250903 20:05:30.365670 27320 leader_election.cc:343] T 512908bc4d6941ec81b145ee5795405f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:30.365825 27320 leader_election.cc:304] T 512908bc4d6941ec81b145ee5795405f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:30.366464 28751 raft_consensus.cc:2747] T 512908bc4d6941ec81b145ee5795405f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:30.368001 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 817) took 3882 ms. Trace:
I20250903 20:05:30.368294 28596 raft_consensus.cc:2802] T 9e98cd1d9bd246e182e2969d147e68bf P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:30.368510 28596 raft_consensus.cc:695] T 9e98cd1d9bd246e182e2969d147e68bf P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:30.368155 27684 rpcz_store.cc:276] 0903 20:05:26.485742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.485803 (+ 61us) service_pool.cc:225] Handling call
0903 20:05:30.088358 (+3602555us) ts_tablet_manager.cc:1145] Deleting tablet 17e0864a32d5452d84ac470b55fbaaf4
0903 20:05:30.088366 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.154940 (+ 66574us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.214674 (+ 59734us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.367977 (+153303us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3602344}
I20250903 20:05:30.368832 28596 consensus_queue.cc:237] T 9e98cd1d9bd246e182e2969d147e68bf P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:30.369064 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 17e0864a32d5452d84ac470b55fbaaf4 (table default.Table_c21ce98311d847f2aaae2ea2c0549856 [id=263d1af690fb4868a4bd92ae090dfd0a]) successfully deleted
I20250903 20:05:30.369294 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 818) took 3882 ms. Trace:
I20250903 20:05:30.369414 27684 rpcz_store.cc:276] 0903 20:05:26.486833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.486885 (+ 52us) service_pool.cc:225] Handling call
0903 20:05:30.369098 (+3882213us) ts_tablet_manager.cc:1145] Deleting tablet ab64604c56914379943c03129ccc2e7c
0903 20:05:30.369112 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.369273 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3882014}
W20250903 20:05:30.370011 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet ab64604c56914379943c03129ccc2e7c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab64604c56914379943c03129ccc2e7c already in progress: creating tablet
I20250903 20:05:30.370507 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 819) took 3883 ms. Trace:
I20250903 20:05:30.370622 27684 rpcz_store.cc:276] 0903 20:05:26.487300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.487347 (+ 47us) service_pool.cc:225] Handling call
0903 20:05:30.370372 (+3883025us) ts_tablet_manager.cc:1145] Deleting tablet ef02f91c0964446daf62ac1dde0330ff
0903 20:05:30.370381 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.370484 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3882864}
W20250903 20:05:30.371143 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet ef02f91c0964446daf62ac1dde0330ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef02f91c0964446daf62ac1dde0330ff already in progress: creating tablet
I20250903 20:05:30.372078 27684 tablet_replica.cc:331] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:30.373332 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet f0efbb7eefb84ab6bee5b9625b91e298 (table default.Table_781d39d6b06f49c28fa93c3692c7b5f7 [id=8d664216ea8e47fdaba51b2d6ca9d0a5]) successfully deleted
I20250903 20:05:30.374070 27684 raft_consensus.cc:2241] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:30.374542 27684 raft_consensus.cc:2270] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:30.375593 27684 ts_tablet_manager.cc:1905] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:30.379117 27462 leader_election.cc:304] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:30.379443 28710 raft_consensus.cc:2747] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:30.379994 27462 leader_election.cc:343] T 9e98cd1d9bd246e182e2969d147e68bf P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:30.380429 27462 leader_election.cc:304] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:30.380715 28710 raft_consensus.cc:2802] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:30.380800 28710 raft_consensus.cc:491] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:30.380877 28710 raft_consensus.cc:3058] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:30.383810 28710 raft_consensus.cc:513] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.384452 28710 leader_election.cc:290] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:30.385304 27462 leader_election.cc:304] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:30.385604 28710 raft_consensus.cc:2802] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:30.385751 28710 raft_consensus.cc:695] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:30.386086 28710 consensus_queue.cc:237] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:30.389482 27692 tablet_bootstrap.cc:654] T 512908bc4d6941ec81b145ee5795405f P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:30.316816 27695 tablet_bootstrap.cc:492] T 22bc87d1a4804b6c80f77f9692500d63 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:30.401125 27695 ts_tablet_manager.cc:1397] T 22bc87d1a4804b6c80f77f9692500d63 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.156s user 0.003s sys 0.000s
I20250903 20:05:30.402100 27695 raft_consensus.cc:357] T 22bc87d1a4804b6c80f77f9692500d63 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.402338 27695 raft_consensus.cc:383] T 22bc87d1a4804b6c80f77f9692500d63 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:30.402395 27695 raft_consensus.cc:738] T 22bc87d1a4804b6c80f77f9692500d63 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:30.402608 27695 consensus_queue.cc:260] T 22bc87d1a4804b6c80f77f9692500d63 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.403070 27687 ts_tablet_manager.cc:1905] T 59a1dd48d76241d7ba817070fbffd853 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:30.403218 27695 ts_tablet_manager.cc:1428] T 22bc87d1a4804b6c80f77f9692500d63 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:30.403525 27695 tablet_bootstrap.cc:492] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:30.404611 27493 tablet_service.cc:1468] Processing CreateTablet for tablet 4ae0446b76d041c98fa76dfeadab5276 (DEFAULT_TABLE table=default.Table_63f21b5e3dd04258a5d189a80a1da7a8 [id=431c1efb538347d1895e3269463c86cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.405086 27695 tablet_bootstrap.cc:654] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:30.405112 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ae0446b76d041c98fa76dfeadab5276. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.405560 27507 tablet_service.cc:1468] Processing CreateTablet for tablet d1bc8582ca68409ea0ee782ee2572c73 (DEFAULT_TABLE table=default.Table_8a37507f7b5a4aa6867f2209f9c94680 [id=6c33b7d6ca5948c7b3f204e76706827a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.405983 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1bc8582ca68409ea0ee782ee2572c73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.407339 27184 leader_election.cc:304] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:30.407886 28874 raft_consensus.cc:2802] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:30.408140 28874 raft_consensus.cc:695] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:30.408460 28874 consensus_queue.cc:237] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:30.413720 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 4ae0446b76d041c98fa76dfeadab5276 (DEFAULT_TABLE table=default.Table_63f21b5e3dd04258a5d189a80a1da7a8 [id=431c1efb538347d1895e3269463c86cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.414309 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ae0446b76d041c98fa76dfeadab5276. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.415396 28808 raft_consensus.cc:491] T a4c582b441f6467c99ba0b090abcf4cc P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:30.415514 28808 raft_consensus.cc:513] T a4c582b441f6467c99ba0b090abcf4cc P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:30.416023 28808 leader_election.cc:290] T a4c582b441f6467c99ba0b090abcf4cc P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:30.417639 27183 leader_election.cc:304] T 2fed2821d2644f75aff6e0ceb5944a37 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:30.417936 28808 raft_consensus.cc:2802] T 2fed2821d2644f75aff6e0ceb5944a37 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:30.418048 28808 raft_consensus.cc:491] T 2fed2821d2644f75aff6e0ceb5944a37 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:30.418179 28808 raft_consensus.cc:3058] T 2fed2821d2644f75aff6e0ceb5944a37 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:30.421132 28808 raft_consensus.cc:513] T 2fed2821d2644f75aff6e0ceb5944a37 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.422086 28808 leader_election.cc:290] T 2fed2821d2644f75aff6e0ceb5944a37 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:30.424790 27184 leader_election.cc:304] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:30.425249 27184 leader_election.cc:304] T 2059ccbd6d884085886d7d564afe03aa P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:30.425933 28745 raft_consensus.cc:2802] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:30.426076 28745 raft_consensus.cc:695] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:30.426296 28870 raft_consensus.cc:2802] T 2059ccbd6d884085886d7d564afe03aa P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:30.426355 28870 raft_consensus.cc:491] T 2059ccbd6d884085886d7d564afe03aa P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:30.426415 28870 raft_consensus.cc:3058] T 2059ccbd6d884085886d7d564afe03aa P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:30.428588 27359 tablet_service.cc:1468] Processing CreateTablet for tablet d1bc8582ca68409ea0ee782ee2572c73 (DEFAULT_TABLE table=default.Table_8a37507f7b5a4aa6867f2209f9c94680 [id=6c33b7d6ca5948c7b3f204e76706827a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.428994 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1bc8582ca68409ea0ee782ee2572c73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.429622 28870 raft_consensus.cc:513] T 2059ccbd6d884085886d7d564afe03aa P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.430493 28745 consensus_queue.cc:237] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:30.432073 28870 leader_election.cc:290] T 2059ccbd6d884085886d7d564afe03aa P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:30.434878 28885 tablet.cc:1722] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250903 20:05:30.436035 27352 tablet_service.cc:1468] Processing CreateTablet for tablet 4ae0446b76d041c98fa76dfeadab5276 (DEFAULT_TABLE table=default.Table_63f21b5e3dd04258a5d189a80a1da7a8 [id=431c1efb538347d1895e3269463c86cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.436582 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ae0446b76d041c98fa76dfeadab5276. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:30.437381 27462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 52090us
I20250903 20:05:30.439361 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b7a6d5d8a97a4769a81ec7224d591f8c" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:30.440026 27383 raft_consensus.cc:3058] T b7a6d5d8a97a4769a81ec7224d591f8c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:30.441202 28751 consensus_queue.cc:1035] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:30.443506 28848 tablet.cc:1722] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250903 20:05:30.446764 27383 raft_consensus.cc:2466] T b7a6d5d8a97a4769a81ec7224d591f8c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:30.449857 27685 ts_tablet_manager.cc:1918] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:30.450073 27685 log.cc:1199] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/de8ac9f984be46ec8f021928de12a090
I20250903 20:05:30.450562 27685 ts_tablet_manager.cc:1939] T de8ac9f984be46ec8f021928de12a090 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:30.453143 28817 raft_consensus.cc:491] T f9709dfd18654c5bb071e39da7c8889f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:30.453311 28817 raft_consensus.cc:513] T f9709dfd18654c5bb071e39da7c8889f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:30.453781 27462 leader_election.cc:304] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:30.454015 28902 raft_consensus.cc:2802] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:30.454128 28902 raft_consensus.cc:695] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:30.454449 28817 leader_election.cc:290] T f9709dfd18654c5bb071e39da7c8889f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:30.454432 28902 consensus_queue.cc:237] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.459789 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 736) took 6145 ms. Trace:
I20250903 20:05:30.464201 27684 ts_tablet_manager.cc:1918] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:30.467599 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2fed2821d2644f75aff6e0ceb5944a37" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:30.467759 27377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2059ccbd6d884085886d7d564afe03aa" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:30.467885 27377 raft_consensus.cc:3058] T 2059ccbd6d884085886d7d564afe03aa P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:30.465224 28746 consensus_queue.cc:1035] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250903 20:05:30.465548 27184 consensus_peers.cc:489] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f8cfe53b91b41d7ba7f9074e1005af3. This is attempt 1: this message will repeat every 5th retry.
W20250903 20:05:30.469359 27184 consensus_peers.cc:489] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:30.469580 28746 consensus_queue.cc:1035] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:30.472445 27349 tablet_service.cc:1468] Processing CreateTablet for tablet 7c6780770f104f669102b051ee7faeb5 (DEFAULT_TABLE table=default.Table_aa4be8035e9441ac8dc4e936724f5e75 [id=163eac86a32e48adb8e3956e40fcd500]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.472875 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c6780770f104f669102b051ee7faeb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.472958 27377 raft_consensus.cc:2466] T 2059ccbd6d884085886d7d564afe03aa P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:30.474936 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 827d186faaa140bdab2f87cf421ca64c (DEFAULT_TABLE table=default.Table_e27e9a52e38149619ab852649bff1d03 [id=14368dfbbc2a4f179dda569e352dabf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.475288 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 827d186faaa140bdab2f87cf421ca64c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.480389 27353 tablet_service.cc:1468] Processing CreateTablet for tablet 12d8846e56c74603891a2c9f0840c007 (DEFAULT_TABLE table=default.Table_b4d6ce0289db42f8aa51e69555580156 [id=4534ff5420864138a7d8d52e8677c65e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.481001 27353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12d8846e56c74603891a2c9f0840c007. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.499866 27695 tablet_bootstrap.cc:492] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:30.500234 27695 ts_tablet_manager.cc:1397] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.097s user 0.003s sys 0.000s
I20250903 20:05:30.500711 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b7a6d5d8a97a4769a81ec7224d591f8c" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:30.472055 27359 tablet_service.cc:1468] Processing CreateTablet for tablet 09a7adab0a524456a59d21b8cfb77f2c (DEFAULT_TABLE table=default.Table_9b7897119b55487d8054d5b6479b7397 [id=e480200de6e74d9c8103ade1ba832091]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.501137 27695 raft_consensus.cc:357] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.501322 27695 raft_consensus.cc:383] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:30.501403 27695 raft_consensus.cc:738] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:30.501747 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09a7adab0a524456a59d21b8cfb77f2c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:30.465840 27183 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 48220.8us
W20250903 20:05:30.502655 27183 leader_election.cc:343] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 283a3afed7414a958b2dc4176f00bb01
I20250903 20:05:30.502921 27183 leader_election.cc:304] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:30.503497 27183 raft_consensus.cc:2684] T 283a3afed7414a958b2dc4176f00bb01 P 89f8f0ee68b940db9ce250c0604f934a: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250903 20:05:30.504848 28746 consensus_queue.cc:1035] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:30.506661 27062 catalog_manager.cc:5582] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66). New cstate: current_term: 1 leader_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:30.507452 27062 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet a9f43781299e47a8aecc37a0b73d7a64 (table default.Table_73a4ac2afbad4654ac623cd581a45eb2 [id=f70d7ea93e154e6199f6766316353862]). Expected version 3 got 0
I20250903 20:05:30.507711 28797 catalog_manager.cc:1261] Loaded cluster ID: 07b8834eda114abdb992ba4c63cadc24
I20250903 20:05:30.507800 27062 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 2059ccbd6d884085886d7d564afe03aa (table default.Table_ea5cb66a574f4466bf7e7f635f22b426 [id=91abc1158ea047849f398926e3695927]). Expected version 1 got 0
I20250903 20:05:30.510608 27228 tablet_service.cc:1468] Processing CreateTablet for tablet 827d186faaa140bdab2f87cf421ca64c (DEFAULT_TABLE table=default.Table_e27e9a52e38149619ab852649bff1d03 [id=14368dfbbc2a4f179dda569e352dabf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.511279 27228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 827d186faaa140bdab2f87cf421ca64c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.511426 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 07439db2c3b8470eb2b797cc68376aa7
I20250903 20:05:30.512149 27065 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:45936 (request call id 1123) took 1286 ms. Trace:
I20250903 20:05:30.512285 27065 rpcz_store.cc:276] 0903 20:05:29.225680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:29.320975 (+ 95295us) service_pool.cc:225] Handling call
0903 20:05:29.329297 (+ 8322us) spinlock_profiling.cc:243] Waited 8.07 ms on lock 0x616000040f54. stack: 00007f4b9d6c2dc6 00007f4b9d6c2b80 00007f4b9c6aa3ad 0000000000972d94 0000000000972d5d 00007f4bac49014d 00007f4bac490243 00007f4ba914f941 00007f4bb0cdaa7d 00007f4bb0e64ea6 00007f4ba7c9d466 00007f4ba7c9d195 00007f4ba44730be 00007f4ba4471f9e 00007f4ba44781d5 00007f4ba4478aff
0903 20:05:29.329679 (+ 382us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0903 20:05:29.329814 (+ 135us) catalog_manager.cc:2790] Modifying in-memory table state
0903 20:05:29.329892 (+ 78us) catalog_manager.cc:2799] Locking tablets
0903 20:05:30.066568 (+736676us) catalog_manager.cc:2813] Removing table and tablets from system table
0903 20:05:30.511195 (+444627us) catalog_manager.cc:2831] Removing table from by-name map
0903 20:05:30.511312 (+ 117us) catalog_manager.cc:2847] Committing in-memory state
0903 20:05:30.511359 (+ 47us) catalog_manager.cc:2851] Aborting table tasks
0903 20:05:30.512055 (+ 696us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:30.069487 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:30.069612 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:30.069618 (+ 6us) write_op.cc:435] Acquired schema lock
0903 20:05:30.069621 (+ 3us) tablet.cc:660] Decoding operations
0903 20:05:30.069738 (+ 117us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:30.069751 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:30.069756 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0903 20:05:30.069875 (+ 119us) tablet.cc:699] Row locks acquired
0903 20:05:30.069878 (+ 3us) write_op.cc:260] PREPARE: finished
0903 20:05:30.069918 (+ 40us) write_op.cc:270] Start()
0903 20:05:30.069952 (+ 34us) write_op.cc:276] Timestamp: P: 1756929930069864 usec, L: 0
0903 20:05:30.069956 (+ 4us) op_driver.cc:348] REPLICATION: starting
0903 20:05:30.156861 (+ 86905us) log.cc:844] Serialized 728 byte log entry
0903 20:05:30.157038 (+ 177us) spinlock_profiling.cc:243] Waited 86.7 ms on lock 0x616000040f54. stack: 00007f4b9d6c2dc6 00007f4b9d6c2b80 00007f4b9c6aa3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f4ba9121d62 00007f4baa4ca2d5 00007f4baa4c8df9 00007f4baa4d1220 00007f4baa4d102d 0000000000a774dc 00007f4b9d70914f 00007f4b9d70c41b 00007f4b9d70c298 0000000000a774dc
0903 20:05:30.508963 (+351925us) op_driver.cc:464] REPLICATION: finished
0903 20:05:30.509278 (+ 315us) write_op.cc:301] APPLY: starting
0903 20:05:30.509326 (+ 48us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:30.509406 (+ 80us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:30.509411 (+ 5us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:30.509553 (+ 142us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:30.509618 (+ 65us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0903 20:05:30.509675 (+ 57us) write_op.cc:312] APPLY: finished
0903 20:05:30.509818 (+ 143us) log.cc:844] Serialized 29 byte log entry
0903 20:05:30.509937 (+ 119us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:30.509970 (+ 33us) write_op.cc:454] Released schema lock
0903 20:05:30.509981 (+ 11us) write_op.cc:341] FINISH: Updating metrics
0903 20:05:30.510117 (+ 136us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:30.510125 (+ 8us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":18561792,"child_traces":[["op",{"apply.queue_time_us":101,"apply.run_cpu_time_us":967,"apply.run_wall_time_us":964,"num_ops":2,"prepare.queue_time_us":1908,"prepare.run_cpu_time_us":1589,"prepare.run_wall_time_us":87612,"raft.queue_time_us":1968,"raft.run_cpu_time_us":404,"raft.run_wall_time_us":404,"replication_time_us":438973,"spinlock_wait_cycles":199479040}]]}
I20250903 20:05:30.514323 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 167b359047ac457c9d165ec199cd9163
I20250903 20:05:30.514281 27695 consensus_queue.cc:260] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.514784 27066 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:45936 (request call id 1131) took 1013 ms. Trace:
I20250903 20:05:30.514914 27695 ts_tablet_manager.cc:1428] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20250903 20:05:30.515205 27695 tablet_bootstrap.cc:492] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:30.514899 27066 rpcz_store.cc:276] 0903 20:05:29.501510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:29.642488 (+140978us) service_pool.cc:225] Handling call
0903 20:05:29.653944 (+ 11456us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x616000040f54. stack: 00007f4b9d6c2dc6 00007f4b9d6c2b80 00007f4b9c6aa3ad 0000000000972d94 0000000000972d5d 00007f4bac49014d 00007f4bac490243 00007f4ba914f941 00007f4bb0cdaa7d 00007f4bb0e64ea6 00007f4ba7c9d466 00007f4ba7c9d195 00007f4ba44730be 00007f4ba4471f9e 00007f4ba44781d5 00007f4ba4478aff
0903 20:05:29.654432 (+ 488us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0903 20:05:29.654567 (+ 135us) catalog_manager.cc:2790] Modifying in-memory table state
0903 20:05:29.654646 (+ 79us) catalog_manager.cc:2799] Locking tablets
0903 20:05:30.068100 (+413454us) catalog_manager.cc:2813] Removing table and tablets from system table
0903 20:05:30.514113 (+446013us) catalog_manager.cc:2831] Removing table from by-name map
0903 20:05:30.514194 (+ 81us) catalog_manager.cc:2847] Committing in-memory state
0903 20:05:30.514253 (+ 59us) catalog_manager.cc:2851] Aborting table tasks
0903 20:05:30.514749 (+ 496us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:30.157130 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:30.157242 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:30.157248 (+ 6us) write_op.cc:435] Acquired schema lock
0903 20:05:30.157250 (+ 2us) tablet.cc:660] Decoding operations
0903 20:05:30.157355 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:30.157366 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:30.157370 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0903 20:05:30.157479 (+ 109us) tablet.cc:699] Row locks acquired
0903 20:05:30.157482 (+ 3us) write_op.cc:260] PREPARE: finished
0903 20:05:30.157521 (+ 39us) write_op.cc:270] Start()
0903 20:05:30.157554 (+ 33us) write_op.cc:276] Timestamp: P: 1756929930157459 usec, L: 0
0903 20:05:30.157557 (+ 3us) op_driver.cc:348] REPLICATION: starting
0903 20:05:30.157692 (+ 135us) log.cc:844] Serialized 728 byte log entry
0903 20:05:30.509044 (+351352us) op_driver.cc:464] REPLICATION: finished
0903 20:05:30.510273 (+ 1229us) write_op.cc:301] APPLY: starting
0903 20:05:30.510289 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:30.510338 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:30.510374 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:30.510444 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:30.510484 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0903 20:05:30.510494 (+ 10us) write_op.cc:312] APPLY: finished
0903 20:05:30.510569 (+ 75us) log.cc:844] Serialized 29 byte log entry
0903 20:05:30.510620 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:30.510639 (+ 19us) write_op.cc:454] Released schema lock
0903 20:05:30.510647 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":25634560,"child_traces":[["op",{"apply.queue_time_us":1181,"apply.run_cpu_time_us":445,"apply.run_wall_time_us":445,"num_ops":2,"prepare.queue_time_us":88382,"prepare.run_cpu_time_us":636,"prepare.run_wall_time_us":635,"raft.queue_time_us":1763,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":351466}]]}
I20250903 20:05:30.516672 27695 tablet_bootstrap.cc:654] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:30.518476 27183 leader_election.cc:304] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:30.518709 28746 consensus_queue.cc:1035] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:30.519376 27227 tablet_service.cc:1468] Processing CreateTablet for tablet 7c6780770f104f669102b051ee7faeb5 (DEFAULT_TABLE table=default.Table_aa4be8035e9441ac8dc4e936724f5e75 [id=163eac86a32e48adb8e3956e40fcd500]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.519601 28746 raft_consensus.cc:2802] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a [term 2 FOLLOWER]: Leader election won for term 2
I20250903 20:05:30.519766 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c6780770f104f669102b051ee7faeb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.519788 28746 raft_consensus.cc:695] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a [term 2 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:30.520012 28746 consensus_queue.cc:237] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:30.522760 27213 tablet_service.cc:1468] Processing CreateTablet for tablet 09a7adab0a524456a59d21b8cfb77f2c (DEFAULT_TABLE table=default.Table_9b7897119b55487d8054d5b6479b7397 [id=e480200de6e74d9c8103ade1ba832091]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.523582 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09a7adab0a524456a59d21b8cfb77f2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.523783 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 167b359047ac457c9d165ec199cd9163 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:29 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:30.524050 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 07439db2c3b8470eb2b797cc68376aa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:29 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:30.524626 27221 tablet_service.cc:1515] Processing DeleteTablet for tablet 07439db2c3b8470eb2b797cc68376aa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:29 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:30.525113 27184 leader_election.cc:343] T 2fed2821d2644f75aff6e0ceb5944a37 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:30.525355 27229 tablet_service.cc:1515] Processing DeleteTablet for tablet 167b359047ac457c9d165ec199cd9163 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:29 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:30.525408 27184 leader_election.cc:304] T 2059ccbd6d884085886d7d564afe03aa P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:30.525867 28808 raft_consensus.cc:2802] T 2059ccbd6d884085886d7d564afe03aa P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:30.525979 28808 raft_consensus.cc:695] T 2059ccbd6d884085886d7d564afe03aa P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:30.526190 28808 consensus_queue.cc:237] T 2059ccbd6d884085886d7d564afe03aa P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.527626 28797 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296: loading cluster ID for follower catalog manager: success
I20250903 20:05:30.528337 27228 tablet_service.cc:1468] Processing CreateTablet for tablet 12d8846e56c74603891a2c9f0840c007 (DEFAULT_TABLE table=default.Table_b4d6ce0289db42f8aa51e69555580156 [id=4534ff5420864138a7d8d52e8677c65e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.528740 27228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12d8846e56c74603891a2c9f0840c007. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.536715 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 1f2124e9baae4c819aeb51cef79c9ff3
I20250903 20:05:30.537453 27061 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:45936 (request call id 1132) took 1020 ms. Trace:
I20250903 20:05:30.537701 28808 consensus_queue.cc:1035] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:30.537568 27061 rpcz_store.cc:276] 0903 20:05:29.517083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:29.827820 (+310737us) service_pool.cc:225] Handling call
0903 20:05:29.829214 (+ 1394us) spinlock_profiling.cc:243] Waited 1.1 ms on lock 0x616000040f54. stack: 00007f4b9d6c2dc6 00007f4b9d6c2b80 00007f4b9c6aa3ad 0000000000972d94 0000000000972d5d 00007f4bac49014d 00007f4bac490243 00007f4ba914f941 00007f4bb0cdaa7d 00007f4bb0e64ea6 00007f4ba7c9d466 00007f4ba7c9d195 00007f4ba44730be 00007f4ba4471f9e 00007f4ba44781d5 00007f4ba4478aff
0903 20:05:29.829658 (+ 444us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0903 20:05:29.829923 (+ 265us) catalog_manager.cc:2790] Modifying in-memory table state
0903 20:05:29.829998 (+ 75us) catalog_manager.cc:2799] Locking tablets
0903 20:05:30.068917 (+238919us) catalog_manager.cc:2813] Removing table and tablets from system table
0903 20:05:30.536523 (+467606us) catalog_manager.cc:2831] Removing table from by-name map
0903 20:05:30.536618 (+ 95us) catalog_manager.cc:2847] Committing in-memory state
0903 20:05:30.536656 (+ 38us) catalog_manager.cc:2851] Aborting table tasks
0903 20:05:30.537388 (+ 732us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:30.157796 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:30.157881 (+ 85us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:30.157885 (+ 4us) write_op.cc:435] Acquired schema lock
0903 20:05:30.157888 (+ 3us) tablet.cc:660] Decoding operations
0903 20:05:30.157972 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:30.157980 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:30.157984 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0903 20:05:30.158069 (+ 85us) tablet.cc:699] Row locks acquired
0903 20:05:30.158072 (+ 3us) write_op.cc:260] PREPARE: finished
0903 20:05:30.158098 (+ 26us) write_op.cc:270] Start()
0903 20:05:30.158121 (+ 23us) write_op.cc:276] Timestamp: P: 1756929930158040 usec, L: 0
0903 20:05:30.158124 (+ 3us) op_driver.cc:348] REPLICATION: starting
0903 20:05:30.158287 (+ 163us) log.cc:844] Serialized 728 byte log entry
0903 20:05:30.535304 (+377017us) op_driver.cc:464] REPLICATION: finished
0903 20:05:30.535466 (+ 162us) write_op.cc:301] APPLY: starting
0903 20:05:30.535480 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:30.535523 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:30.535557 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:30.535626 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:30.535664 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0903 20:05:30.535671 (+ 7us) write_op.cc:312] APPLY: finished
0903 20:05:30.535754 (+ 83us) log.cc:844] Serialized 29 byte log entry
0903 20:05:30.535813 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:30.535829 (+ 16us) write_op.cc:454] Released schema lock
0903 20:05:30.535835 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0903 20:05:30.535900 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:30.535903 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":2533760,"child_traces":[["op",{"apply.queue_time_us":104,"apply.run_cpu_time_us":427,"apply.run_wall_time_us":427,"num_ops":2,"prepare.queue_time_us":88367,"prepare.run_cpu_time_us":576,"prepare.run_wall_time_us":576,"raft.queue_time_us":1181,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":377140}]]}
W20250903 20:05:30.538753 27183 leader_election.cc:343] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 639ecb9caa4a423899ab8eb35bb716d7
I20250903 20:05:30.538874 27183 leader_election.cc:304] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:30.539075 28808 raft_consensus.cc:2747] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:30.540167 27685 rpcz_store.cc:276] 0903 20:05:24.314740 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:24.315529 (+ 789us) service_pool.cc:225] Handling call
0903 20:05:30.308988 (+5993459us) ts_tablet_manager.cc:1145] Deleting tablet de8ac9f984be46ec8f021928de12a090
0903 20:05:30.308998 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.356903 (+ 47905us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.449763 (+ 92860us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.459752 (+ 9989us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5993298}
I20250903 20:05:30.540664 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 737) took 6225 ms. Trace:
I20250903 20:05:30.540824 27685 rpcz_store.cc:276] 0903 20:05:24.314818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:24.316093 (+ 1275us) service_pool.cc:225] Handling call
0903 20:05:30.540533 (+6224440us) ts_tablet_manager.cc:1145] Deleting tablet 548b50819f694fa3ad508da9f442a99c
0903 20:05:30.540545 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.540649 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6224202}
I20250903 20:05:30.541364 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f2124e9baae4c819aeb51cef79c9ff3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:29 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:30.542505 27184 leader_election.cc:343] T a4c582b441f6467c99ba0b090abcf4cc P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: a4c582b441f6467c99ba0b090abcf4cc
W20250903 20:05:30.544360 27183 leader_election.cc:343] T 2059ccbd6d884085886d7d564afe03aa P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:30.545972 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_c623ccc6c3ab480caca560c89b2444ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:30.546677 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c623ccc6c3ab480caca560c89b2444ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:30.547765 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_c642bf5deebb4382b01bc6ceba8b9daa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:30.548334 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c642bf5deebb4382b01bc6ceba8b9daa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:30.550478 27687 ts_tablet_manager.cc:1918] T 59a1dd48d76241d7ba817070fbffd853 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:30.550647 27687 log.cc:1199] T 59a1dd48d76241d7ba817070fbffd853 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/59a1dd48d76241d7ba817070fbffd853
I20250903 20:05:30.551129 27687 ts_tablet_manager.cc:1939] T 59a1dd48d76241d7ba817070fbffd853 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:30.552134 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 867) took 3303 ms. Trace:
I20250903 20:05:30.552644 27687 rpcz_store.cc:276] 0903 20:05:27.249103 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.250290 (+ 1187us) service_pool.cc:225] Handling call
0903 20:05:30.349517 (+3099227us) ts_tablet_manager.cc:1145] Deleting tablet 59a1dd48d76241d7ba817070fbffd853
0903 20:05:30.349527 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.415651 (+ 66124us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.550362 (+134711us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.552109 (+ 1747us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3099065}
I20250903 20:05:30.553321 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 878) took 3150 ms. Trace:
I20250903 20:05:30.554749 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 59a1dd48d76241d7ba817070fbffd853 (table default.Table_d342b3c06e1c4b98bc7f07e88844cdc9 [id=34db41bcdc604ee08da11d0e559b4d35]) successfully deleted
I20250903 20:05:30.557282 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet de8ac9f984be46ec8f021928de12a090 (table default.Table_eaff993b7bb7437580aae238faa74b3e [id=2650c8259ef4489695c56b6a2ba898d3]) successfully deleted
W20250903 20:05:30.557731 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 548b50819f694fa3ad508da9f442a99c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 548b50819f694fa3ad508da9f442a99c already in progress: creating tablet
I20250903 20:05:30.558280 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 735) took 6226 ms. Trace:
I20250903 20:05:30.558434 27685 rpcz_store.cc:276] 0903 20:05:24.314664 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:24.316468 (+ 1804us) service_pool.cc:225] Handling call
0903 20:05:30.541118 (+6224650us) ts_tablet_manager.cc:1145] Deleting tablet ed7c564138cc43e5a3bb4e92fb5060b6
0903 20:05:30.541127 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.541200 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6203702}
I20250903 20:05:30.558931 27684 log.cc:1199] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/4157633525034a019fd6a80185892cec
I20250903 20:05:30.471148 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 09a7adab0a524456a59d21b8cfb77f2c (DEFAULT_TABLE table=default.Table_9b7897119b55487d8054d5b6479b7397 [id=e480200de6e74d9c8103ade1ba832091]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.566457 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09a7adab0a524456a59d21b8cfb77f2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.566552 27684 ts_tablet_manager.cc:1939] T 4157633525034a019fd6a80185892cec P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:30.567420 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 820) took 4069 ms. Trace:
I20250903 20:05:30.567530 27684 rpcz_store.cc:276] 0903 20:05:26.497489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.497545 (+ 56us) service_pool.cc:225] Handling call
0903 20:05:30.371977 (+3874432us) ts_tablet_manager.cc:1145] Deleting tablet 4157633525034a019fd6a80185892cec
0903 20:05:30.371999 (+ 22us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.380246 (+ 8247us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.464069 (+ 83823us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.567396 (+103327us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3874229}
I20250903 20:05:30.568042 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 4157633525034a019fd6a80185892cec (table default.Table_57cd5123dc7042c1887af6e833af4d7a [id=4b42213db8534ac3b3b97318f5a38e61]) successfully deleted
I20250903 20:05:30.568382 27684 tablet_replica.cc:331] T 7cc6cf579e2642e99fefed723b0ee900 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:30.568584 27684 raft_consensus.cc:2241] T 7cc6cf579e2642e99fefed723b0ee900 P e72ef1dbcc5143cf9f04a808f101c80f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:30.568718 27684 raft_consensus.cc:2270] T 7cc6cf579e2642e99fefed723b0ee900 P e72ef1dbcc5143cf9f04a808f101c80f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:30.569332 27684 ts_tablet_manager.cc:1905] T 7cc6cf579e2642e99fefed723b0ee900 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:30.473007 27494 tablet_service.cc:1468] Processing CreateTablet for tablet 7c6780770f104f669102b051ee7faeb5 (DEFAULT_TABLE table=default.Table_aa4be8035e9441ac8dc4e936724f5e75 [id=163eac86a32e48adb8e3956e40fcd500]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.475484 27488 tablet_service.cc:1468] Processing CreateTablet for tablet 827d186faaa140bdab2f87cf421ca64c (DEFAULT_TABLE table=default.Table_e27e9a52e38149619ab852649bff1d03 [id=14368dfbbc2a4f179dda569e352dabf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.476689 27499 tablet_service.cc:1468] Processing CreateTablet for tablet 12d8846e56c74603891a2c9f0840c007 (DEFAULT_TABLE table=default.Table_b4d6ce0289db42f8aa51e69555580156 [id=4534ff5420864138a7d8d52e8677c65e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.570708 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 827d186faaa140bdab2f87cf421ca64c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.570858 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12d8846e56c74603891a2c9f0840c007. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.571079 27685 tablet_replica.cc:331] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:30.571364 27685 raft_consensus.cc:2241] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:30.571547 27685 raft_consensus.cc:2270] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:30.572096 27685 ts_tablet_manager.cc:1905] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:30.519817 27489 tablet_service.cc:1515] Processing DeleteTablet for tablet 07439db2c3b8470eb2b797cc68376aa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:29 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:30.519949 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet 167b359047ac457c9d165ec199cd9163 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:29 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:30.574435 27494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c6780770f104f669102b051ee7faeb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.542433 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f2124e9baae4c819aeb51cef79c9ff3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:29 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:30.584287 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2fed2821d2644f75aff6e0ceb5944a37" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:30.584600 27518 raft_consensus.cc:3058] T 2fed2821d2644f75aff6e0ceb5944a37 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:30.469147 28710 raft_consensus.cc:491] T 4af8c03c9d834405b4e25d35441db961 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:30.566051 27692 tablet_bootstrap.cc:492] T 512908bc4d6941ec81b145ee5795405f P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:30.589915 27518 raft_consensus.cc:2466] T 2fed2821d2644f75aff6e0ceb5944a37 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
W20250903 20:05:30.559716 27462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 105940us
I20250903 20:05:30.589944 28710 raft_consensus.cc:513] T 4af8c03c9d834405b4e25d35441db961 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:30.592298 28710 leader_election.cc:290] T 4af8c03c9d834405b4e25d35441db961 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
W20250903 20:05:30.596994 27183 leader_election.cc:343] T a4c582b441f6467c99ba0b090abcf4cc P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: a4c582b441f6467c99ba0b090abcf4cc
I20250903 20:05:30.597355 27183 leader_election.cc:304] T a4c582b441f6467c99ba0b090abcf4cc P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:30.590163 27692 ts_tablet_manager.cc:1397] T 512908bc4d6941ec81b145ee5795405f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.353s user 0.004s sys 0.000s
I20250903 20:05:30.597887 27183 leader_election.cc:304] T 2fed2821d2644f75aff6e0ceb5944a37 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:30.598358 27692 raft_consensus.cc:357] T 512908bc4d6941ec81b145ee5795405f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.598554 27692 raft_consensus.cc:383] T 512908bc4d6941ec81b145ee5795405f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:30.598624 27692 raft_consensus.cc:738] T 512908bc4d6941ec81b145ee5795405f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:30.598621 28947 raft_consensus.cc:2747] T a4c582b441f6467c99ba0b090abcf4cc P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:30.598826 28947 raft_consensus.cc:2802] T 2fed2821d2644f75aff6e0ceb5944a37 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:30.598976 28947 raft_consensus.cc:695] T 2fed2821d2644f75aff6e0ceb5944a37 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:30.598871 27692 consensus_queue.cc:260] T 512908bc4d6941ec81b145ee5795405f P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.599184 28746 consensus_queue.cc:1035] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:30.599273 28947 consensus_queue.cc:237] T 2fed2821d2644f75aff6e0ceb5944a37 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.599706 27692 ts_tablet_manager.cc:1428] T 512908bc4d6941ec81b145ee5795405f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:30.599658 28848 tablet.cc:1722] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:30.599979 27692 tablet_bootstrap.cc:492] T 22bc87d1a4804b6c80f77f9692500d63 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:30.600215 28808 raft_consensus.cc:491] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:30.600420 28808 raft_consensus.cc:513] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.601635 28808 leader_election.cc:290] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:30.601667 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9709dfd18654c5bb071e39da7c8889f" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:30.601989 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2059ccbd6d884085886d7d564afe03aa" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:30.618323 28596 raft_consensus.cc:491] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:30.618548 28596 raft_consensus.cc:513] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.619406 28596 leader_election.cc:290] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:30.620128 27462 leader_election.cc:343] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:30.620741 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet ed7c564138cc43e5a3bb4e92fb5060b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed7c564138cc43e5a3bb4e92fb5060b6 already in progress: creating tablet
I20250903 20:05:30.623489 27518 raft_consensus.cc:3058] T f9709dfd18654c5bb071e39da7c8889f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:30.626310 27687 rpcz_store.cc:276] 0903 20:05:27.403098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.403175 (+ 77us) service_pool.cc:225] Handling call
0903 20:05:30.553177 (+3150002us) ts_tablet_manager.cc:1145] Deleting tablet edac2355f67045a6acffaae723d9186f
0903 20:05:30.553187 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.553306 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3149775}
I20250903 20:05:30.622318 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4af8c03c9d834405b4e25d35441db961" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:30.626863 27378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "726f4d920f3e47fea41b4e12c021c988" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:30.622903 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63a1f26fac3b4b939ebb79c6a07d8f8e" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:30.627089 27378 raft_consensus.cc:2466] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:30.627210 27381 raft_consensus.cc:2466] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:30.594197 27384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9709dfd18654c5bb071e39da7c8889f" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:30.627722 28874 raft_consensus.cc:491] T 120ca61b3da742dc9a2efa483ead7e97 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:30.627923 28874 raft_consensus.cc:513] T 120ca61b3da742dc9a2efa483ead7e97 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:30.628500 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet edac2355f67045a6acffaae723d9186f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet edac2355f67045a6acffaae723d9186f already in progress: creating tablet
I20250903 20:05:30.628618 28874 leader_election.cc:290] T 120ca61b3da742dc9a2efa483ead7e97 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:30.629012 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 879) took 3224 ms. Trace:
I20250903 20:05:30.629168 27687 rpcz_store.cc:276] 0903 20:05:27.404130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.404183 (+ 53us) service_pool.cc:225] Handling call
0903 20:05:30.628868 (+3224685us) ts_tablet_manager.cc:1145] Deleting tablet 91169fff8a6649828648ac97ccf65fdc
0903 20:05:30.628880 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.628994 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3224492}
W20250903 20:05:30.629674 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 91169fff8a6649828648ac97ccf65fdc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91169fff8a6649828648ac97ccf65fdc already in progress: creating tablet
I20250903 20:05:30.635324 27227 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f2124e9baae4c819aeb51cef79c9ff3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:29 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:30.636885 27184 leader_election.cc:304] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:30.637179 28870 consensus_queue.cc:1035] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:30.638171 27215 tablet_service.cc:1468] Processing CreateTablet for tablet d1bc8582ca68409ea0ee782ee2572c73 (DEFAULT_TABLE table=default.Table_8a37507f7b5a4aa6867f2209f9c94680 [id=6c33b7d6ca5948c7b3f204e76706827a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.638546 28870 raft_consensus.cc:2802] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:30.638675 28870 raft_consensus.cc:491] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:30.638767 28870 raft_consensus.cc:3058] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:30.638892 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1bc8582ca68409ea0ee782ee2572c73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.641783 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63a1f26fac3b4b939ebb79c6a07d8f8e" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:30.641180 27376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "120ca61b3da742dc9a2efa483ead7e97" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:30.642771 27513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "726f4d920f3e47fea41b4e12c021c988" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:30.643481 27183 leader_election.cc:343] T 2059ccbd6d884085886d7d564afe03aa P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:30.643831 27183 leader_election.cc:343] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:30.644151 27526 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "120ca61b3da742dc9a2efa483ead7e97" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:30.644850 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 1157920321454d8790b2620690878579 (DEFAULT_TABLE table=default.Table_3bcd6f6a18284a278b1467fe1c2374a3 [id=6bf15d11e18d4290904de9faba2af69e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.645277 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1157920321454d8790b2620690878579. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:30.647121 27184 leader_election.cc:343] T 120ca61b3da742dc9a2efa483ead7e97 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:30.646154 27359 tablet_service.cc:1468] Processing CreateTablet for tablet 1157920321454d8790b2620690878579 (DEFAULT_TABLE table=default.Table_3bcd6f6a18284a278b1467fe1c2374a3 [id=6bf15d11e18d4290904de9faba2af69e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.647862 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1157920321454d8790b2620690878579. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.648216 27227 tablet_service.cc:1468] Processing CreateTablet for tablet 1157920321454d8790b2620690878579 (DEFAULT_TABLE table=default.Table_3bcd6f6a18284a278b1467fe1c2374a3 [id=6bf15d11e18d4290904de9faba2af69e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:30.648811 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1157920321454d8790b2620690878579. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:30.641155 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 881) took 3191 ms. Trace:
I20250903 20:05:30.652426 27687 rpcz_store.cc:276] 0903 20:05:27.449943 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.450018 (+ 75us) service_pool.cc:225] Handling call
0903 20:05:30.641026 (+3191008us) ts_tablet_manager.cc:1145] Deleting tablet 112c0dccb60c4c7c9efc164d9c8443ea
0903 20:05:30.641036 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.641134 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3190771}
I20250903 20:05:30.653235 28885 tablet.cc:1722] T b0d79c78e0024b7ba3bb3356be7c8ceb P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250903 20:05:30.653458 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 112c0dccb60c4c7c9efc164d9c8443ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 112c0dccb60c4c7c9efc164d9c8443ea already in progress: creating tablet
I20250903 20:05:30.655397 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 880) took 3206 ms. Trace:
W20250903 20:05:30.659348 27462 leader_election.cc:343] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:30.660707 27462 leader_election.cc:304] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:30.661230 27462 leader_election.cc:304] T f9709dfd18654c5bb071e39da7c8889f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:30.661429 28596 raft_consensus.cc:2802] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:30.661515 28596 raft_consensus.cc:491] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:30.661572 28596 raft_consensus.cc:3058] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:30.662266 27687 rpcz_store.cc:276] 0903 20:05:27.449276 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.450271 (+ 995us) service_pool.cc:225] Handling call
0903 20:05:30.655239 (+3204968us) ts_tablet_manager.cc:1145] Deleting tablet a686ea49a29a4dd7b6e5dce5e1f9935a
0903 20:05:30.655252 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.655376 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3204754}
I20250903 20:05:30.663115 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 882) took 3212 ms. Trace:
I20250903 20:05:30.663208 27687 rpcz_store.cc:276] 0903 20:05:27.450633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.450728 (+ 95us) service_pool.cc:225] Handling call
0903 20:05:30.662988 (+3212260us) ts_tablet_manager.cc:1145] Deleting tablet cf99a74acfdb4a17bbbce3a07467e9c1
0903 20:05:30.663002 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.663099 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3212093}
W20250903 20:05:30.663640 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet cf99a74acfdb4a17bbbce3a07467e9c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf99a74acfdb4a17bbbce3a07467e9c1 already in progress: creating tablet
I20250903 20:05:30.664170 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 883) took 3213 ms. Trace:
I20250903 20:05:30.664793 28596 raft_consensus.cc:513] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:30.665550 27183 leader_election.cc:343] T 120ca61b3da742dc9a2efa483ead7e97 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:30.665748 27183 leader_election.cc:304] T 120ca61b3da742dc9a2efa483ead7e97 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:30.666029 28745 raft_consensus.cc:2747] T 120ca61b3da742dc9a2efa483ead7e97 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:30.667003 28808 consensus_queue.cc:1035] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:30.667516 27058 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:45936 (request call id 1136) took 1081 ms. Trace:
I20250903 20:05:30.667491 28710 raft_consensus.cc:2747] T f9709dfd18654c5bb071e39da7c8889f P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:30.667665 27058 rpcz_store.cc:276] 0903 20:05:29.586452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:29.873493 (+287041us) service_pool.cc:225] Handling call
0903 20:05:30.166470 (+292977us) spinlock_profiling.cc:243] Waited 293 ms on lock 0x616000040f54. stack: 00007f4b9d6c2dc6 00007f4b9d6c2b80 00007f4b9c6aa3ad 0000000000972d94 0000000000972d5d 00007f4bac49014d 00007f4bac490243 00007f4ba914f941 00007f4bb0cdaa7d 00007f4bb0e63eb3 00007f4ba7c9b7a6 00007f4ba7c9b4d5 00007f4ba44730be 00007f4ba4471f9e 00007f4ba44781d5 00007f4ba4478aff
0903 20:05:30.167322 (+ 852us) catalog_manager.cc:2076] Acquired catalog manager lock
0903 20:05:30.167805 (+ 483us) catalog_manager.cc:2122] Created new table and tablet info
0903 20:05:30.667118 (+499313us) catalog_manager.cc:2182] Wrote table and tablets to system table
0903 20:05:30.667395 (+ 277us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0903 20:05:30.667476 (+ 81us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:30.604146 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:30.604354 (+ 208us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:30.604364 (+ 10us) write_op.cc:435] Acquired schema lock
0903 20:05:30.604369 (+ 5us) tablet.cc:660] Decoding operations
0903 20:05:30.604546 (+ 177us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:30.604564 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:30.604572 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0903 20:05:30.604701 (+ 129us) tablet.cc:699] Row locks acquired
0903 20:05:30.604705 (+ 4us) write_op.cc:260] PREPARE: finished
0903 20:05:30.604888 (+ 183us) write_op.cc:270] Start()
0903 20:05:30.604970 (+ 82us) write_op.cc:276] Timestamp: P: 1756929930604697 usec, L: 0
0903 20:05:30.604977 (+ 7us) op_driver.cc:348] REPLICATION: starting
0903 20:05:30.605261 (+ 284us) log.cc:844] Serialized 433 byte log entry
0903 20:05:30.664793 (+ 59532us) op_driver.cc:464] REPLICATION: finished
0903 20:05:30.665611 (+ 818us) write_op.cc:301] APPLY: starting
0903 20:05:30.665623 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:30.665661 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:30.665690 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:30.665812 (+ 122us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:30.665848 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0903 20:05:30.665855 (+ 7us) write_op.cc:312] APPLY: finished
0903 20:05:30.665919 (+ 64us) log.cc:844] Serialized 29 byte log entry
0903 20:05:30.665995 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:30.666011 (+ 16us) write_op.cc:454] Released schema lock
0903 20:05:30.666017 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0903 20:05:30.666084 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:30.666087 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":673206528,"child_traces":[["op",{"apply.queue_time_us":765,"apply.run_cpu_time_us":466,"apply.run_wall_time_us":466,"num_ops":2,"prepare.queue_time_us":435790,"prepare.run_cpu_time_us":1172,"prepare.run_wall_time_us":3588,"raft.queue_time_us":934,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":59797}]]}
I20250903 20:05:30.668416 27058 master_service.cc:948] ReplaceTablet: received request to replace tablet 17e7deb74389448db37289563b86c253 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:30.669056 27692 tablet_bootstrap.cc:654] T 22bc87d1a4804b6c80f77f9692500d63 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:30.665629 28596 leader_election.cc:290] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:30.552786 27695 tablet_bootstrap.cc:492] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:30.671164 27695 ts_tablet_manager.cc:1397] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.156s user 0.003s sys 0.001s
I20250903 20:05:30.664250 27687 rpcz_store.cc:276] 0903 20:05:27.450987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.451420 (+ 433us) service_pool.cc:225] Handling call
0903 20:05:30.664064 (+3212644us) ts_tablet_manager.cc:1145] Deleting tablet 30cc0d4ba8d444f1b98232f906ea0c7b
0903 20:05:30.664074 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.664154 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3212505}
W20250903 20:05:30.673362 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 30cc0d4ba8d444f1b98232f906ea0c7b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30cc0d4ba8d444f1b98232f906ea0c7b already in progress: creating tablet
I20250903 20:05:30.674489 27687 tablet_replica.cc:331] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:30.637238 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4af8c03c9d834405b4e25d35441db961" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:30.674839 27249 raft_consensus.cc:2391] T 4af8c03c9d834405b4e25d35441db961 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
I20250903 20:05:30.674990 27687 raft_consensus.cc:2241] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:30.675643 27687 raft_consensus.cc:2270] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:30.676689 27687 ts_tablet_manager.cc:1905] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:30.685302 27695 raft_consensus.cc:357] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.685542 27695 raft_consensus.cc:383] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:30.685628 27695 raft_consensus.cc:738] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:30.685892 27695 consensus_queue.cc:260] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.687094 27695 ts_tablet_manager.cc:1428] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20250903 20:05:30.687440 27695 tablet_bootstrap.cc:492] T b7a6d5d8a97a4769a81ec7224d591f8c P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:30.690413 27687 ts_tablet_manager.cc:1918] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:30.690629 27687 log.cc:1199] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/1fbe54dc40dd47daa109ceeb07546b0a
I20250903 20:05:30.692358 27687 ts_tablet_manager.cc:1939] T 1fbe54dc40dd47daa109ceeb07546b0a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:30.695802 28770 consensus_queue.cc:1035] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:30.693447 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 889) took 3120 ms. Trace:
I20250903 20:05:30.697240 27687 rpcz_store.cc:276] 0903 20:05:27.573162 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.573222 (+ 60us) service_pool.cc:225] Handling call
0903 20:05:30.674429 (+3101207us) ts_tablet_manager.cc:1145] Deleting tablet 1fbe54dc40dd47daa109ceeb07546b0a
0903 20:05:30.674441 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.683014 (+ 8573us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.690288 (+ 7274us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.693424 (+ 3136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3100995}
I20250903 20:05:30.697842 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 1fbe54dc40dd47daa109ceeb07546b0a (table default.Table_975e53d37e054938a5b63957252ff50d [id=f261e84232234244824d0232efbb4399]) successfully deleted
I20250903 20:05:30.698333 27687 tablet_replica.cc:331] T e0347eab6c8642beabdc2d7e7bbfdc9b P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:30.699162 27687 raft_consensus.cc:2241] T e0347eab6c8642beabdc2d7e7bbfdc9b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:30.699399 27687 raft_consensus.cc:2270] T e0347eab6c8642beabdc2d7e7bbfdc9b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:30.700402 27685 ts_tablet_manager.cc:1918] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:30.700546 27685 log.cc:1199] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/639ecb9caa4a423899ab8eb35bb716d7
I20250903 20:05:30.700933 27685 ts_tablet_manager.cc:1939] T 639ecb9caa4a423899ab8eb35bb716d7 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:30.701467 27687 ts_tablet_manager.cc:1905] T e0347eab6c8642beabdc2d7e7bbfdc9b P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:30.706025 28958 tablet.cc:1722] T b0d79c78e0024b7ba3bb3356be7c8ceb P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:30.706537 27684 ts_tablet_manager.cc:1918] T 7cc6cf579e2642e99fefed723b0ee900 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:30.706686 27684 log.cc:1199] T 7cc6cf579e2642e99fefed723b0ee900 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/7cc6cf579e2642e99fefed723b0ee900
I20250903 20:05:30.707171 27684 ts_tablet_manager.cc:1939] T 7cc6cf579e2642e99fefed723b0ee900 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:30.714121 28948 raft_consensus.cc:491] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51578fb7bceb4e119b3ab1e58c4f5528)
I20250903 20:05:30.714308 28948 raft_consensus.cc:513] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:30.715020 28948 leader_election.cc:290] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:30.716063 27513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685cba2f73b412382a69dab2c7f50c1" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:30.718258 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_eaff993b7bb7437580aae238faa74b3e" } modify_external_catalogs: true
I20250903 20:05:30.718495 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_eaff993b7bb7437580aae238faa74b3e" } modify_external_catalogs: true
I20250903 20:05:30.719424 27057 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:45936 (request call id 1137) took 1109 ms. Trace:
I20250903 20:05:30.719542 27057 rpcz_store.cc:276] 0903 20:05:29.610050 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.064846 (+454796us) service_pool.cc:225] Handling call
0903 20:05:30.166074 (+101228us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x616000040f54. stack: 00007f4b9d6c2dc6 00007f4b9d6c2b80 00007f4b9c6aa3ad 0000000000972d94 0000000000972d5d 00007f4bac49014d 00007f4bac490243 00007f4ba914f941 00007f4bb0cdaa7d 00007f4bb0e63eb3 00007f4ba7c9b7a6 00007f4ba7c9b4d5 00007f4ba44730be 00007f4ba4471f9e 00007f4ba44781d5 00007f4ba4478aff
0903 20:05:30.179746 (+ 13672us) catalog_manager.cc:2076] Acquired catalog manager lock
0903 20:05:30.180277 (+ 531us) catalog_manager.cc:2122] Created new table and tablet info
0903 20:05:30.719150 (+538873us) catalog_manager.cc:2182] Wrote table and tablets to system table
0903 20:05:30.719283 (+ 133us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0903 20:05:30.719380 (+ 97us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:30.607787 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:30.607970 (+ 183us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:30.607980 (+ 10us) write_op.cc:435] Acquired schema lock
0903 20:05:30.607985 (+ 5us) tablet.cc:660] Decoding operations
0903 20:05:30.608126 (+ 141us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:30.608143 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:30.608151 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0903 20:05:30.608262 (+ 111us) tablet.cc:699] Row locks acquired
0903 20:05:30.608267 (+ 5us) write_op.cc:260] PREPARE: finished
0903 20:05:30.608324 (+ 57us) write_op.cc:270] Start()
0903 20:05:30.608373 (+ 49us) write_op.cc:276] Timestamp: P: 1756929930608255 usec, L: 0
0903 20:05:30.608379 (+ 6us) op_driver.cc:348] REPLICATION: starting
0903 20:05:30.608600 (+ 221us) log.cc:844] Serialized 433 byte log entry
0903 20:05:30.716613 (+108013us) op_driver.cc:464] REPLICATION: finished
0903 20:05:30.717655 (+ 1042us) write_op.cc:301] APPLY: starting
0903 20:05:30.717677 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:30.717729 (+ 52us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:30.717777 (+ 48us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:30.718041 (+ 264us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:30.718081 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0903 20:05:30.718089 (+ 8us) write_op.cc:312] APPLY: finished
0903 20:05:30.718312 (+ 223us) log.cc:844] Serialized 29 byte log entry
0903 20:05:30.718408 (+ 96us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:30.718440 (+ 32us) write_op.cc:454] Released schema lock
0903 20:05:30.718451 (+ 11us) write_op.cc:341] FINISH: Updating metrics
0903 20:05:30.718561 (+ 110us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:30.718568 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":231920000,"child_traces":[["op",{"apply.queue_time_us":935,"apply.run_cpu_time_us":895,"apply.run_wall_time_us":894,"num_ops":2,"prepare.queue_time_us":426992,"prepare.run_cpu_time_us":998,"prepare.run_wall_time_us":55625,"raft.queue_time_us":50,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":111,"replication_time_us":108192}]]}
I20250903 20:05:30.720338 27057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_28fd9ad363e94c149f59785ca4cdeb7b" } new_table_name: "default.Table_120fea21398e41afaa4f552bbd492f38" modify_external_catalogs: true
I20250903 20:05:30.720893 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a534288cb6194102ade0cbb0b87630ee" } modify_external_catalogs: true
I20250903 20:05:30.721079 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a534288cb6194102ade0cbb0b87630ee" } modify_external_catalogs: true
I20250903 20:05:30.721642 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet e8bc99de4ffe43d58351d170b7c61d8a
I20250903 20:05:30.722167 27059 catalog_manager.cc:6652] ReplaceTablet: tablet e8bc99de4ffe43d58351d170b7c61d8a deleted and replaced by tablet e9b5ab312f8744af99d557c0733bf71d
I20250903 20:05:30.722326 27059 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:45936 (request call id 1139) took 1003 ms. Trace:
I20250903 20:05:30.722470 27059 rpcz_store.cc:276] 0903 20:05:29.719220 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.111865 (+392645us) service_pool.cc:225] Handling call
0903 20:05:30.209420 (+ 97555us) spinlock_profiling.cc:243] Waited 97.1 ms on lock 0x616000040f54. stack: 00007f4b9d6c2dc6 00007f4b9d6c2b80 00007f4b9c6aa3ad 0000000000972d94 0000000000972d5d 00007f4bac49014d 00007f4bac490243 00007f4ba914f941 00007f4bb0cdaa7d 00007f4bb0e70bfb 00007f4ba7ca80e7 00007f4ba7ca7e15 00007f4ba44730be 00007f4ba4471f9e 00007f4ba44781d5 00007f4ba4478aff
0903 20:05:30.722285 (+512865us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:30.663451 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:30.663617 (+ 166us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:30.663628 (+ 11us) write_op.cc:435] Acquired schema lock
0903 20:05:30.663631 (+ 3us) tablet.cc:660] Decoding operations
0903 20:05:30.663793 (+ 162us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:30.663806 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:30.663813 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0903 20:05:30.663906 (+ 93us) tablet.cc:699] Row locks acquired
0903 20:05:30.663908 (+ 2us) write_op.cc:260] PREPARE: finished
0903 20:05:30.663945 (+ 37us) write_op.cc:270] Start()
0903 20:05:30.663981 (+ 36us) write_op.cc:276] Timestamp: P: 1756929930663885 usec, L: 0
0903 20:05:30.663985 (+ 4us) op_driver.cc:348] REPLICATION: starting
0903 20:05:30.664182 (+ 197us) log.cc:844] Serialized 659 byte log entry
0903 20:05:30.716886 (+ 52704us) op_driver.cc:464] REPLICATION: finished
0903 20:05:30.718637 (+ 1751us) write_op.cc:301] APPLY: starting
0903 20:05:30.718648 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:30.718684 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:30.718707 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:30.718799 (+ 92us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:30.718866 (+ 67us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0903 20:05:30.718879 (+ 13us) write_op.cc:312] APPLY: finished
0903 20:05:30.718957 (+ 78us) log.cc:844] Serialized 29 byte log entry
0903 20:05:30.719002 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:30.719018 (+ 16us) write_op.cc:454] Released schema lock
0903 20:05:30.719024 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":223232896,"child_traces":[["op",{"apply.queue_time_us":1713,"apply.run_cpu_time_us":438,"apply.run_wall_time_us":437,"num_ops":2,"prepare.queue_time_us":440591,"prepare.run_cpu_time_us":978,"prepare.run_wall_time_us":59435,"raft.queue_time_us":86,"raft.run_cpu_time_us":379,"raft.run_wall_time_us":569,"replication_time_us":52878}]]}
I20250903 20:05:30.723457 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_20e9f856d5e24febbdb3b95098a17f0c" } new_table_name: "default.Table_cb85e96534984e88a664c7183a732fe4" modify_external_catalogs: true
I20250903 20:05:30.727787 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 821) took 4229 ms. Trace:
I20250903 20:05:30.727910 27684 rpcz_store.cc:276] 0903 20:05:26.497982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.498033 (+ 51us) service_pool.cc:225] Handling call
0903 20:05:30.568332 (+4070299us) ts_tablet_manager.cc:1145] Deleting tablet 7cc6cf579e2642e99fefed723b0ee900
0903 20:05:30.568342 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.660908 (+ 92566us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.706391 (+ 45483us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.727761 (+ 21370us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4070147}
I20250903 20:05:30.728392 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 7cc6cf579e2642e99fefed723b0ee900 (table default.Table_bffb6801f48c4124bfe24c49586d6e48 [id=aa020b54da65482ba59a71768c2bd03b]) successfully deleted
I20250903 20:05:30.729174 27352 tablet_service.cc:1515] Processing DeleteTablet for tablet e8bc99de4ffe43d58351d170b7c61d8a with delete_type TABLET_DATA_DELETED (replaced by tablet e9b5ab312f8744af99d557c0733bf71d) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:30.729321 27184 leader_election.cc:343] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 9685cba2f73b412382a69dab2c7f50c1
I20250903 20:05:30.729480 27184 leader_election.cc:304] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:30.730001 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 816) took 4324 ms. Trace:
I20250903 20:05:30.730086 27228 tablet_service.cc:1515] Processing DeleteTablet for tablet e8bc99de4ffe43d58351d170b7c61d8a with delete_type TABLET_DATA_DELETED (replaced by tablet e9b5ab312f8744af99d557c0733bf71d) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:30.730428 28746 raft_consensus.cc:2747] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:30.730571 27684 rpcz_store.cc:276] 0903 20:05:26.405967 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.531453 (+125486us) service_pool.cc:225] Handling call
0903 20:05:30.729839 (+4198386us) ts_tablet_manager.cc:1145] Deleting tablet 55b2cdee20264ae6b64ebe3607fe1ca6
0903 20:05:30.729852 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.729979 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4197928}
W20250903 20:05:30.731124 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 55b2cdee20264ae6b64ebe3607fe1ca6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55b2cdee20264ae6b64ebe3607fe1ca6 already in progress: creating tablet
I20250903 20:05:30.733644 27684 tablet_replica.cc:331] T a0293a69f1c246d08d414e95729b4ab4 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:30.733927 27684 raft_consensus.cc:2241] T a0293a69f1c246d08d414e95729b4ab4 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:30.734407 27684 raft_consensus.cc:2270] T a0293a69f1c246d08d414e95729b4ab4 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:30.734822 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63a1f26fac3b4b939ebb79c6a07d8f8e" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:30.734835 27376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63a1f26fac3b4b939ebb79c6a07d8f8e" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:30.735014 27376 raft_consensus.cc:3058] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:30.737780 27376 raft_consensus.cc:2466] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:30.738879 27684 ts_tablet_manager.cc:1905] T a0293a69f1c246d08d414e95729b4ab4 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:30.740370 27462 leader_election.cc:304] T 4af8c03c9d834405b4e25d35441db961 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:30.740741 28902 raft_consensus.cc:491] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:30.740953 28902 raft_consensus.cc:513] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:30.741714 28902 leader_election.cc:290] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:30.742554 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet e8bc99de4ffe43d58351d170b7c61d8a with delete_type TABLET_DATA_DELETED (replaced by tablet e9b5ab312f8744af99d557c0733bf71d) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:30.743639 28902 raft_consensus.cc:3058] T 4af8c03c9d834405b4e25d35441db961 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:30.746174 28902 raft_consensus.cc:2747] T 4af8c03c9d834405b4e25d35441db961 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:30.758071 27695 tablet_bootstrap.cc:654] T b7a6d5d8a97a4769a81ec7224d591f8c P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:30.760381 27693 tablet_bootstrap.cc:492] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:30.760563 27693 ts_tablet_manager.cc:1397] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.459s user 0.057s sys 0.056s
I20250903 20:05:30.761554 27693 raft_consensus.cc:357] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.761746 27693 raft_consensus.cc:383] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:30.761797 27693 raft_consensus.cc:738] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:30.763957 27693 consensus_queue.cc:260] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.766681 27687 ts_tablet_manager.cc:1918] T e0347eab6c8642beabdc2d7e7bbfdc9b P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:30.769081 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a3ccbeb42cef4247a4f9b6b349af9371" } new_table_name: "default.Table_af252d2e4c2f40469f1e0b5cd3fa20ff" modify_external_catalogs: true
I20250903 20:05:30.769443 27687 log.cc:1199] T e0347eab6c8642beabdc2d7e7bbfdc9b P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/e0347eab6c8642beabdc2d7e7bbfdc9b
I20250903 20:05:30.769953 27687 ts_tablet_manager.cc:1939] T e0347eab6c8642beabdc2d7e7bbfdc9b P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:30.771118 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 891) took 3186 ms. Trace:
I20250903 20:05:30.772111 27687 rpcz_store.cc:276] 0903 20:05:27.584998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.585074 (+ 76us) service_pool.cc:225] Handling call
0903 20:05:30.698257 (+3113183us) ts_tablet_manager.cc:1145] Deleting tablet e0347eab6c8642beabdc2d7e7bbfdc9b
0903 20:05:30.698274 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.761680 (+ 63406us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.766577 (+ 4897us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.771092 (+ 4515us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3112945}
I20250903 20:05:30.772930 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet e0347eab6c8642beabdc2d7e7bbfdc9b (table default.Table_187e391a6d6d4d0d96d4bb8a0581face [id=54ebade293b7466aa39f00ec876f27eb]) successfully deleted
I20250903 20:05:30.773505 27687 tablet_replica.cc:331] T a4f3230f50ec4c95b528fead8c191224 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:30.773900 27687 raft_consensus.cc:2241] T a4f3230f50ec4c95b528fead8c191224 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:30.774386 27687 raft_consensus.cc:2270] T a4f3230f50ec4c95b528fead8c191224 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:30.780682 27687 ts_tablet_manager.cc:1905] T a4f3230f50ec4c95b528fead8c191224 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:30.781632 27684 ts_tablet_manager.cc:1918] T a0293a69f1c246d08d414e95729b4ab4 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:30.781785 27684 log.cc:1199] T a0293a69f1c246d08d414e95729b4ab4 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/a0293a69f1c246d08d414e95729b4ab4
I20250903 20:05:30.771517 27693 ts_tablet_manager.cc:1428] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250903 20:05:30.782538 27693 tablet_bootstrap.cc:492] T 2fed2821d2644f75aff6e0ceb5944a37 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:30.783272 27066 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 2059ccbd6d884085886d7d564afe03aa (table default.Table_ea5cb66a574f4466bf7e7f635f22b426 [id=91abc1158ea047849f398926e3695927]). Expected version 1 got 0
I20250903 20:05:30.783545 27066 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 3b3bfc63aa0b4e56bec189e834dc6a8d (table default.Table_9b20469406c94986ab496ea3e678b586 [id=6d6f5e8cadf9445d8695c075cfabb2ea]). Expected version 1 got 0
I20250903 20:05:30.783931 27066 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 9e98cd1d9bd246e182e2969d147e68bf (table default.Table_28a829a9988442049d50e6ae254e6274 [id=b27f08d56c3c43e6a09a465fdda4d3db]). Expected version 1 got 0
I20250903 20:05:30.784155 27693 tablet_bootstrap.cc:654] T 2fed2821d2644f75aff6e0ceb5944a37 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:30.788985 27065 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 9e98cd1d9bd246e182e2969d147e68bf (table default.Table_28a829a9988442049d50e6ae254e6274 [id=b27f08d56c3c43e6a09a465fdda4d3db]). Expected version 1 got 0
I20250903 20:05:30.789423 27065 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet a9f43781299e47a8aecc37a0b73d7a64 (table default.Table_73a4ac2afbad4654ac623cd581a45eb2 [id=f70d7ea93e154e6199f6766316353862]). Expected version 3 got 0
I20250903 20:05:30.789934 27687 ts_tablet_manager.cc:1918] T a4f3230f50ec4c95b528fead8c191224 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:30.790077 27687 log.cc:1199] T a4f3230f50ec4c95b528fead8c191224 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/a4f3230f50ec4c95b528fead8c191224
I20250903 20:05:30.790489 27687 ts_tablet_manager.cc:1939] T a4f3230f50ec4c95b528fead8c191224 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:30.790983 27693 tablet_bootstrap.cc:492] T 2fed2821d2644f75aff6e0ceb5944a37 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:30.791118 27693 ts_tablet_manager.cc:1397] T 2fed2821d2644f75aff6e0ceb5944a37 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20250903 20:05:30.791589 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 892) took 3206 ms. Trace:
I20250903 20:05:30.791846 27687 rpcz_store.cc:276] 0903 20:05:27.585520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.585574 (+ 54us) service_pool.cc:225] Handling call
0903 20:05:30.773453 (+3187879us) ts_tablet_manager.cc:1145] Deleting tablet a4f3230f50ec4c95b528fead8c191224
0903 20:05:30.773464 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.786717 (+ 13253us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.789822 (+ 3105us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.791562 (+ 1740us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3187716}
I20250903 20:05:30.792618 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet a4f3230f50ec4c95b528fead8c191224 (table default.Table_be001b7aec594b93a1b45e9313b772ca [id=7a045d80e0eb42cb95fdca51d339da0d]) successfully deleted
I20250903 20:05:30.793803 27687 tablet_replica.cc:331] T 7e2f98f760cf4df4bc2a10530e2157c6 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:30.794070 27687 raft_consensus.cc:2241] T 7e2f98f760cf4df4bc2a10530e2157c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:30.795070 27687 raft_consensus.cc:2270] T 7e2f98f760cf4df4bc2a10530e2157c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:30.798770 27684 ts_tablet_manager.cc:1939] T a0293a69f1c246d08d414e95729b4ab4 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
W20250903 20:05:30.798930 27462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 58592.6us
W20250903 20:05:30.799240 27462 leader_election.cc:343] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:30.799736 27687 ts_tablet_manager.cc:1905] T 7e2f98f760cf4df4bc2a10530e2157c6 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:30.799774 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 824) took 4169 ms. Trace:
I20250903 20:05:30.799891 27684 rpcz_store.cc:276] 0903 20:05:26.629816 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.634444 (+ 4628us) service_pool.cc:225] Handling call
0903 20:05:30.731635 (+4097191us) ts_tablet_manager.cc:1145] Deleting tablet a0293a69f1c246d08d414e95729b4ab4
0903 20:05:30.733578 (+ 1943us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.742038 (+ 8460us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.781485 (+ 39447us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.799726 (+ 18241us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4095908}
I20250903 20:05:30.800101 27462 leader_election.cc:304] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:30.800420 28596 raft_consensus.cc:2802] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:30.800392 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet a0293a69f1c246d08d414e95729b4ab4 (table default.Table_61bb115ae97549f7ab9c44e52b75acf0 [id=4ce7195efb6e432e9b232b8a4749179f]) successfully deleted
I20250903 20:05:30.800556 28596 raft_consensus.cc:695] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:30.791929 27693 raft_consensus.cc:357] T 2fed2821d2644f75aff6e0ceb5944a37 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.800823 28596 consensus_queue.cc:237] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.800930 27693 raft_consensus.cc:383] T 2fed2821d2644f75aff6e0ceb5944a37 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:30.800997 27693 raft_consensus.cc:738] T 2fed2821d2644f75aff6e0ceb5944a37 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:30.802356 27693 consensus_queue.cc:260] T 2fed2821d2644f75aff6e0ceb5944a37 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.803323 27693 ts_tablet_manager.cc:1428] T 2fed2821d2644f75aff6e0ceb5944a37 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250903 20:05:30.803597 27693 tablet_bootstrap.cc:492] T 9e98cd1d9bd246e182e2969d147e68bf P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:30.805380 27693 tablet_bootstrap.cc:654] T 9e98cd1d9bd246e182e2969d147e68bf P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:30.806972 28745 consensus_queue.cc:1035] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20250903 20:05:30.807503 28958 tablet.cc:1722] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250903 20:05:30.810602 27684 tablet_replica.cc:331] T 9685cba2f73b412382a69dab2c7f50c1 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:30.810982 27684 raft_consensus.cc:2241] T 9685cba2f73b412382a69dab2c7f50c1 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:30.811340 27684 raft_consensus.cc:2270] T 9685cba2f73b412382a69dab2c7f50c1 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:30.812268 27684 ts_tablet_manager.cc:1905] T 9685cba2f73b412382a69dab2c7f50c1 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:30.813928 27692 tablet_bootstrap.cc:492] T 22bc87d1a4804b6c80f77f9692500d63 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:30.814108 27692 ts_tablet_manager.cc:1397] T 22bc87d1a4804b6c80f77f9692500d63 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.214s user 0.003s sys 0.001s
I20250903 20:05:30.815032 27692 raft_consensus.cc:357] T 22bc87d1a4804b6c80f77f9692500d63 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.815241 27692 raft_consensus.cc:383] T 22bc87d1a4804b6c80f77f9692500d63 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:30.815356 27692 raft_consensus.cc:738] T 22bc87d1a4804b6c80f77f9692500d63 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:30.815654 27692 consensus_queue.cc:260] T 22bc87d1a4804b6c80f77f9692500d63 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.816614 27692 ts_tablet_manager.cc:1428] T 22bc87d1a4804b6c80f77f9692500d63 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250903 20:05:30.817066 27692 tablet_bootstrap.cc:492] T 1bf69315f07a470bb2b897ad7c590729 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:30.819653 27687 ts_tablet_manager.cc:1918] T 7e2f98f760cf4df4bc2a10530e2157c6 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:30.819795 27687 log.cc:1199] T 7e2f98f760cf4df4bc2a10530e2157c6 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/7e2f98f760cf4df4bc2a10530e2157c6
I20250903 20:05:30.820194 27687 ts_tablet_manager.cc:1939] T 7e2f98f760cf4df4bc2a10530e2157c6 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:30.821060 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 893) took 3234 ms. Trace:
I20250903 20:05:30.821188 27687 rpcz_store.cc:276] 0903 20:05:27.586784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.588517 (+ 1733us) service_pool.cc:225] Handling call
0903 20:05:30.793751 (+3205234us) ts_tablet_manager.cc:1145] Deleting tablet 7e2f98f760cf4df4bc2a10530e2157c6
0903 20:05:30.793762 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.804282 (+ 10520us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.819510 (+ 15228us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.821016 (+ 1506us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3203122}
I20250903 20:05:30.821800 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 7e2f98f760cf4df4bc2a10530e2157c6 (table default.Table_3185c25ec3784288bd52ccf75830589d [id=4d1a27b6dfb74dd69ba27d0f1e767dac]) successfully deleted
I20250903 20:05:30.828003 27692 tablet_bootstrap.cc:654] T 1bf69315f07a470bb2b897ad7c590729 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:30.828310 28870 raft_consensus.cc:513] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.829071 28870 leader_election.cc:290] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:30.829581 27687 tablet_replica.cc:331] T 466e501e424e46bc98e8bbc64f97005e P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:30.829792 27687 raft_consensus.cc:2241] T 466e501e424e46bc98e8bbc64f97005e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:30.829963 27687 raft_consensus.cc:2270] T 466e501e424e46bc98e8bbc64f97005e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:30.830009 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 741) took 6179 ms. Trace:
I20250903 20:05:30.830132 27685 rpcz_store.cc:276] 0903 20:05:24.650898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:24.656034 (+ 5136us) service_pool.cc:225] Handling call
0903 20:05:30.570922 (+5914888us) ts_tablet_manager.cc:1145] Deleting tablet 639ecb9caa4a423899ab8eb35bb716d7
0903 20:05:30.570940 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.656705 (+ 85765us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.700230 (+ 43525us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.829955 (+129725us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5907113}
I20250903 20:05:30.830586 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 753) took 6178 ms. Trace:
I20250903 20:05:30.830711 27685 rpcz_store.cc:276] 0903 20:05:24.651844 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:24.893063 (+241219us) service_pool.cc:225] Handling call
0903 20:05:30.830424 (+5937361us) ts_tablet_manager.cc:1145] Deleting tablet 849eb453d3e04b16bbb5df9017bfb2fd
0903 20:05:30.830434 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.830545 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5937173}
I20250903 20:05:30.831074 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 752) took 6179 ms. Trace:
I20250903 20:05:30.831178 27685 rpcz_store.cc:276] 0903 20:05:24.651776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:24.892745 (+240969us) service_pool.cc:225] Handling call
0903 20:05:30.830926 (+5938181us) ts_tablet_manager.cc:1145] Deleting tablet 0060e6c4699a4bff8faddefb4ee50add
0903 20:05:30.830935 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.831028 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5916957}
I20250903 20:05:30.831502 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 754) took 6179 ms. Trace:
I20250903 20:05:30.831599 27685 rpcz_store.cc:276] 0903 20:05:24.651936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:24.893265 (+241329us) service_pool.cc:225] Handling call
0903 20:05:30.831388 (+5938123us) ts_tablet_manager.cc:1145] Deleting tablet 97d261080b8b479ca12267b6aa2e2ab0
0903 20:05:30.831396 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.831459 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5908619}
I20250903 20:05:30.831916 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 755) took 6179 ms. Trace:
I20250903 20:05:30.832022 27685 rpcz_store.cc:276] 0903 20:05:24.652033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:24.893564 (+241531us) service_pool.cc:225] Handling call
0903 20:05:30.831799 (+5938235us) ts_tablet_manager.cc:1145] Deleting tablet 15072aa10c5540a09f3ec4592e377d2a
0903 20:05:30.831806 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.831872 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5883500}
I20250903 20:05:30.832294 27685 tablet_replica.cc:331] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:30.836824 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 639ecb9caa4a423899ab8eb35bb716d7 (table default.Table_3da0f03ae8114e348baf44301196b2e5 [id=ad8b3a9bedef4c188b89640ab5e2473a]) successfully deleted
W20250903 20:05:30.837231 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 849eb453d3e04b16bbb5df9017bfb2fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 849eb453d3e04b16bbb5df9017bfb2fd already in progress: creating tablet
W20250903 20:05:30.837497 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 0060e6c4699a4bff8faddefb4ee50add because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0060e6c4699a4bff8faddefb4ee50add already in progress: creating tablet
W20250903 20:05:30.837708 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 97d261080b8b479ca12267b6aa2e2ab0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97d261080b8b479ca12267b6aa2e2ab0 already in progress: creating tablet
W20250903 20:05:30.837953 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 15072aa10c5540a09f3ec4592e377d2a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15072aa10c5540a09f3ec4592e377d2a already in progress: creating tablet
I20250903 20:05:30.839120 27693 tablet_bootstrap.cc:492] T 9e98cd1d9bd246e182e2969d147e68bf P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:30.839277 27693 ts_tablet_manager.cc:1397] T 9e98cd1d9bd246e182e2969d147e68bf P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.036s user 0.001s sys 0.002s
I20250903 20:05:30.839532 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "726f4d920f3e47fea41b4e12c021c988" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:30.839963 27693 raft_consensus.cc:357] T 9e98cd1d9bd246e182e2969d147e68bf P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:30.840127 27693 raft_consensus.cc:383] T 9e98cd1d9bd246e182e2969d147e68bf P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:30.840173 27693 raft_consensus.cc:738] T 9e98cd1d9bd246e182e2969d147e68bf P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
W20250903 20:05:30.840183 27183 leader_election.cc:343] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:30.842594 27184 consensus_peers.cc:489] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0347eab6c8642beabdc2d7e7bbfdc9b. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:30.843483 27685 raft_consensus.cc:2241] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:30.843672 27685 raft_consensus.cc:2270] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:30.844275 27685 ts_tablet_manager.cc:1905] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:30.846474 27693 consensus_queue.cc:260] T 9e98cd1d9bd246e182e2969d147e68bf P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:30.847548 27687 ts_tablet_manager.cc:1905] T 466e501e424e46bc98e8bbc64f97005e P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:30.848775 27693 ts_tablet_manager.cc:1428] T 9e98cd1d9bd246e182e2969d147e68bf P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250903 20:05:30.849084 27693 tablet_bootstrap.cc:492] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:30.850795 27693 tablet_bootstrap.cc:654] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:30.851079 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "726f4d920f3e47fea41b4e12c021c988" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:30.851377 27385 raft_consensus.cc:3058] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:30.854754 27385 raft_consensus.cc:2466] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:30.855980 27685 ts_tablet_manager.cc:1918] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:30.856143 27684 ts_tablet_manager.cc:1918] T 9685cba2f73b412382a69dab2c7f50c1 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:30.856375 27684 log.cc:1199] T 9685cba2f73b412382a69dab2c7f50c1 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/9685cba2f73b412382a69dab2c7f50c1
W20250903 20:05:30.856739 27462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 56621.5us
I20250903 20:05:30.855283 27687 ts_tablet_manager.cc:1918] T 466e501e424e46bc98e8bbc64f97005e P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:30.857913 27687 log.cc:1199] T 466e501e424e46bc98e8bbc64f97005e P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/466e501e424e46bc98e8bbc64f97005e
I20250903 20:05:30.858314 27687 ts_tablet_manager.cc:1939] T 466e501e424e46bc98e8bbc64f97005e P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:30.859027 28567 consensus_queue.cc:1035] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:30.860010 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 895) took 3267 ms. Trace:
I20250903 20:05:30.860127 27687 rpcz_store.cc:276] 0903 20:05:27.592710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.592755 (+ 45us) service_pool.cc:225] Handling call
0903 20:05:30.829505 (+3236750us) ts_tablet_manager.cc:1145] Deleting tablet 466e501e424e46bc98e8bbc64f97005e
0903 20:05:30.829517 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.851618 (+ 22101us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.855171 (+ 3553us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.859982 (+ 4811us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":678,"tablet-delete.queue_time_us":3236595}
W20250903 20:05:30.860518 27052 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 989545be137341ef8d742bb1119cc51f (table default.Table_412e2b595d4e4e83b26634356476d069 [id=1caafc550c7d4f478cb019f7e01b2aee], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 989545be137341ef8d742bb1119cc51f consensus state has no leader
I20250903 20:05:30.860677 27687 tablet_replica.cc:331] T a72b85dfb34047528b2b28ff0d40f828 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:30.860896 27687 raft_consensus.cc:2241] T a72b85dfb34047528b2b28ff0d40f828 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:30.861074 27687 raft_consensus.cc:2270] T a72b85dfb34047528b2b28ff0d40f828 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:30.861896 27687 ts_tablet_manager.cc:1905] T a72b85dfb34047528b2b28ff0d40f828 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:30.862636 28797 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296: acquiring CA information for follower catalog manager: success
I20250903 20:05:30.858618 27184 leader_election.cc:304] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:30.867007 27692 tablet_bootstrap.cc:492] T 1bf69315f07a470bb2b897ad7c590729 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:30.862272 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "514e382479014fa7bf1f404efff11c86" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:30.867218 27692 ts_tablet_manager.cc:1397] T 1bf69315f07a470bb2b897ad7c590729 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.050s user 0.001s sys 0.003s
I20250903 20:05:30.867409 27240 raft_consensus.cc:2466] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:30.868362 27687 ts_tablet_manager.cc:1918] T a72b85dfb34047528b2b28ff0d40f828 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:30.868487 28870 raft_consensus.cc:2802] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:30.868422 27692 raft_consensus.cc:357] T 1bf69315f07a470bb2b897ad7c590729 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.868640 28870 raft_consensus.cc:695] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:30.868680 27692 raft_consensus.cc:383] T 1bf69315f07a470bb2b897ad7c590729 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:30.868770 27692 raft_consensus.cc:738] T 1bf69315f07a470bb2b897ad7c590729 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:30.868973 28870 consensus_queue.cc:237] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.872648 27687 log.cc:1199] T a72b85dfb34047528b2b28ff0d40f828 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/a72b85dfb34047528b2b28ff0d40f828
I20250903 20:05:30.856837 27684 ts_tablet_manager.cc:1939] T 9685cba2f73b412382a69dab2c7f50c1 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:30.874441 27693 tablet_bootstrap.cc:492] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:30.875056 27693 ts_tablet_manager.cc:1397] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20250903 20:05:30.876004 27693 raft_consensus.cc:357] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.876251 27693 raft_consensus.cc:383] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:30.876329 27693 raft_consensus.cc:738] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:30.876837 27687 ts_tablet_manager.cc:1939] T a72b85dfb34047528b2b28ff0d40f828 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:30.877645 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 899) took 3229 ms. Trace:
I20250903 20:05:30.877748 27687 rpcz_store.cc:276] 0903 20:05:27.648442 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.648507 (+ 65us) service_pool.cc:225] Handling call
0903 20:05:30.860614 (+3212107us) ts_tablet_manager.cc:1145] Deleting tablet a72b85dfb34047528b2b28ff0d40f828
0903 20:05:30.860628 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.865023 (+ 4395us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.868150 (+ 3127us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.877623 (+ 9473us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3211855}
I20250903 20:05:30.878887 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 823) took 4249 ms. Trace:
I20250903 20:05:30.879007 27684 rpcz_store.cc:276] 0903 20:05:26.629699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.639114 (+ 9415us) service_pool.cc:225] Handling call
0903 20:05:30.810479 (+4171365us) ts_tablet_manager.cc:1145] Deleting tablet 9685cba2f73b412382a69dab2c7f50c1
0903 20:05:30.810496 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.837210 (+ 26714us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.855990 (+ 18780us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.878859 (+ 22869us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4170815}
I20250903 20:05:30.879331 27684 tablet_replica.cc:331] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:30.879506 27684 raft_consensus.cc:2241] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f [term 2 LEADER]: Raft consensus shutting down.
I20250903 20:05:30.879729 27684 pending_rounds.cc:70] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f: Trying to abort 1 pending ops.
I20250903 20:05:30.879773 27684 pending_rounds.cc:77] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f: Aborting op as it isn't in flight: id { term: 2 index: 1 } timestamp: 7196384993530519552 op_type: NO_OP noop_request { }
I20250903 20:05:30.879886 27684 raft_consensus.cc:2887] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:30.879963 27684 raft_consensus.cc:2270] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:30.880698 27685 log.cc:1199] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/9685cba2f73b412382a69dab2c7f50c1
I20250903 20:05:30.881189 27685 ts_tablet_manager.cc:1939] T 9685cba2f73b412382a69dab2c7f50c1 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:30.881206 27693 consensus_queue.cc:260] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.881755 27687 tablet_replica.cc:331] T 8a223f75045345ae833f52bcc8f239f9 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:30.881986 27687 raft_consensus.cc:2241] T 8a223f75045345ae833f52bcc8f239f9 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:30.883208 27687 raft_consensus.cc:2270] T 8a223f75045345ae833f52bcc8f239f9 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:30.882809 27693 ts_tablet_manager.cc:1428] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250903 20:05:30.883764 27693 tablet_bootstrap.cc:492] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:30.884809 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 466e501e424e46bc98e8bbc64f97005e (table default.Table_83ea6dfcee144a4fbc86931567b676bb [id=f321a32c1ac043238c2fc92bfd5ec1ae]) successfully deleted
I20250903 20:05:30.885264 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 9685cba2f73b412382a69dab2c7f50c1 (table default.Table_0cd65e00bd5a40bd9dc9db12fd0bda4d [id=15369700921549be8faa450d2ca98949]) successfully deleted
I20250903 20:05:30.885366 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet a72b85dfb34047528b2b28ff0d40f828 (table default.Table_2fa1d4a13c1840cd88c36e5fa2c6475f [id=d7b6b14b360e45a59034d2eed00e2e62]) successfully deleted
I20250903 20:05:30.889086 27060 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:45936 (request call id 1142) took 1054 ms. Trace:
I20250903 20:05:30.890014 27687 ts_tablet_manager.cc:1905] T 8a223f75045345ae833f52bcc8f239f9 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:30.889266 27060 rpcz_store.cc:276] 0903 20:05:29.834822 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.539347 (+704525us) service_pool.cc:225] Handling call
0903 20:05:30.547713 (+ 8366us) spinlock_profiling.cc:243] Waited 8.11 ms on lock 0x616000040f54. stack: 00007f4b9d6c2dc6 00007f4b9d6c2b80 00007f4b9c6aa3ad 0000000000972d94 0000000000972d5d 00007f4bac49014d 00007f4bac490243 00007f4ba914f941 00007f4bb0cdaa7d 00007f4bb0e63eb3 00007f4ba7c9b7a6 00007f4ba7c9b4d5 00007f4ba44730be 00007f4ba4471f9e 00007f4ba44781d5 00007f4ba4478aff
0903 20:05:30.548424 (+ 711us) catalog_manager.cc:2076] Acquired catalog manager lock
0903 20:05:30.548820 (+ 396us) catalog_manager.cc:2122] Created new table and tablet info
0903 20:05:30.888744 (+339924us) catalog_manager.cc:2182] Wrote table and tablets to system table
0903 20:05:30.888883 (+ 139us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0903 20:05:30.889010 (+ 127us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:30.826484 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:30.826630 (+ 146us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:30.826635 (+ 5us) write_op.cc:435] Acquired schema lock
0903 20:05:30.826638 (+ 3us) tablet.cc:660] Decoding operations
0903 20:05:30.826744 (+ 106us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:30.826759 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:30.826764 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0903 20:05:30.826874 (+ 110us) tablet.cc:699] Row locks acquired
0903 20:05:30.826877 (+ 3us) write_op.cc:260] PREPARE: finished
0903 20:05:30.826935 (+ 58us) write_op.cc:270] Start()
0903 20:05:30.826970 (+ 35us) write_op.cc:276] Timestamp: P: 1756929930826889 usec, L: 0
0903 20:05:30.826974 (+ 4us) op_driver.cc:348] REPLICATION: starting
0903 20:05:30.827217 (+ 243us) log.cc:844] Serialized 433 byte log entry
0903 20:05:30.887768 (+ 60551us) op_driver.cc:464] REPLICATION: finished
0903 20:05:30.888008 (+ 240us) write_op.cc:301] APPLY: starting
0903 20:05:30.888031 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:30.888093 (+ 62us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:30.888137 (+ 44us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:30.888288 (+ 151us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:30.888347 (+ 59us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0903 20:05:30.888359 (+ 12us) write_op.cc:312] APPLY: finished
0903 20:05:30.888489 (+ 130us) log.cc:844] Serialized 29 byte log entry
0903 20:05:30.888567 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:30.888598 (+ 31us) write_op.cc:454] Released schema lock
0903 20:05:30.888607 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":18644736,"child_traces":[["op",{"apply.queue_time_us":174,"apply.run_cpu_time_us":675,"apply.run_wall_time_us":675,"num_ops":2,"prepare.queue_time_us":277150,"raft.queue_time_us":19945,"raft.run_cpu_time_us":921,"raft.run_wall_time_us":4587,"replication_time_us":60751}]]}
I20250903 20:05:30.891100 27684 ts_tablet_manager.cc:1905] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:30.895758 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e672f826c7de4a0eae169f4c4002f34a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:30.896684 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e672f826c7de4a0eae169f4c4002f34a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:30.903331 27693 tablet_bootstrap.cc:654] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:30.905915 27061 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:45936 (request call id 1141) took 1071 ms. Trace:
I20250903 20:05:30.906124 27061 rpcz_store.cc:276] 0903 20:05:29.834644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.538643 (+703999us) service_pool.cc:225] Handling call
0903 20:05:30.545899 (+ 7256us) spinlock_profiling.cc:243] Waited 6.89 ms on lock 0x616000040f54. stack: 00007f4b9d6c2dc6 00007f4b9d6c2b80 00007f4b9c6aa3ad 0000000000972d94 0000000000972d5d 00007f4bac49014d 00007f4bac490243 00007f4ba914f941 00007f4bb0cdaa7d 00007f4bb0e63eb3 00007f4ba7c9b7a6 00007f4ba7c9b4d5 00007f4ba44730be 00007f4ba4471f9e 00007f4ba44781d5 00007f4ba4478aff
0903 20:05:30.546771 (+ 872us) catalog_manager.cc:2076] Acquired catalog manager lock
0903 20:05:30.547247 (+ 476us) catalog_manager.cc:2122] Created new table and tablet info
0903 20:05:30.905545 (+358298us) catalog_manager.cc:2182] Wrote table and tablets to system table
0903 20:05:30.905702 (+ 157us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0903 20:05:30.905833 (+ 131us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:30.898633 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:30.898785 (+ 152us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:30.898792 (+ 7us) write_op.cc:435] Acquired schema lock
0903 20:05:30.898797 (+ 5us) tablet.cc:660] Decoding operations
0903 20:05:30.898922 (+ 125us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:30.898940 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:30.898947 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0903 20:05:30.899054 (+ 107us) tablet.cc:699] Row locks acquired
0903 20:05:30.899056 (+ 2us) write_op.cc:260] PREPARE: finished
0903 20:05:30.899097 (+ 41us) write_op.cc:270] Start()
0903 20:05:30.899138 (+ 41us) write_op.cc:276] Timestamp: P: 1756929930899057 usec, L: 0
0903 20:05:30.899143 (+ 5us) op_driver.cc:348] REPLICATION: starting
0903 20:05:30.899366 (+ 223us) log.cc:844] Serialized 433 byte log entry
0903 20:05:30.904471 (+ 5105us) op_driver.cc:464] REPLICATION: finished
0903 20:05:30.904749 (+ 278us) write_op.cc:301] APPLY: starting
0903 20:05:30.904774 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:30.904841 (+ 67us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:30.904876 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:30.905084 (+ 208us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:30.905136 (+ 52us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0903 20:05:30.905148 (+ 12us) write_op.cc:312] APPLY: finished
0903 20:05:30.905256 (+ 108us) log.cc:844] Serialized 29 byte log entry
0903 20:05:30.905335 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:30.905364 (+ 29us) write_op.cc:454] Released schema lock
0903 20:05:30.905375 (+ 11us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":15850880,"child_traces":[["op",{"apply.queue_time_us":215,"apply.run_cpu_time_us":707,"apply.run_wall_time_us":707,"num_ops":2,"prepare.queue_time_us":349114,"raft.queue_time_us":34,"replication_time_us":5282}]]}
I20250903 20:05:30.911425 27684 ts_tablet_manager.cc:1918] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20250903 20:05:30.911561 27684 log.cc:1199] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/6788b1683f984949816edfc5b68de8db
I20250903 20:05:30.911942 27684 ts_tablet_manager.cc:1939] T 6788b1683f984949816edfc5b68de8db P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:30.869148 27692 consensus_queue.cc:260] T 1bf69315f07a470bb2b897ad7c590729 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.912792 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 822) took 4283 ms. Trace:
I20250903 20:05:30.912907 27684 rpcz_store.cc:276] 0903 20:05:26.629571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.636659 (+ 7088us) service_pool.cc:225] Handling call
0903 20:05:30.879283 (+4242624us) ts_tablet_manager.cc:1145] Deleting tablet 6788b1683f984949816edfc5b68de8db
0903 20:05:30.879292 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.908253 (+ 28961us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.911312 (+ 3059us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.912772 (+ 1460us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4239481}
I20250903 20:05:30.913393 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 6788b1683f984949816edfc5b68de8db (table default.Table_48f65d1e5e0b4836bb4df770bd94b4e0 [id=45702f222451484c80915421a0cdeb68]) successfully deleted
I20250903 20:05:30.914945 27692 ts_tablet_manager.cc:1428] T 1bf69315f07a470bb2b897ad7c590729 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.048s user 0.022s sys 0.000s
I20250903 20:05:30.915282 27692 tablet_bootstrap.cc:492] T 67ca1dcf6fc7435db555392235f9900f P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:30.918841 27693 tablet_bootstrap.cc:492] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:30.919131 27693 ts_tablet_manager.cc:1397] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.002s
I20250903 20:05:30.921731 27693 raft_consensus.cc:357] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.922048 27693 raft_consensus.cc:383] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:30.922147 27693 raft_consensus.cc:738] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:30.922461 27693 consensus_queue.cc:260] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.922837 27692 tablet_bootstrap.cc:654] T 67ca1dcf6fc7435db555392235f9900f P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:30.923480 27693 ts_tablet_manager.cc:1428] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:30.925709 27693 tablet_bootstrap.cc:492] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:30.926005 27687 ts_tablet_manager.cc:1918] T 8a223f75045345ae833f52bcc8f239f9 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:30.927523 27687 log.cc:1199] T 8a223f75045345ae833f52bcc8f239f9 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/8a223f75045345ae833f52bcc8f239f9
I20250903 20:05:30.928006 28751 consensus_queue.cc:1035] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250903 20:05:30.928808 27462 leader_election.cc:343] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 514e382479014fa7bf1f404efff11c86
I20250903 20:05:30.929029 27687 ts_tablet_manager.cc:1939] T 8a223f75045345ae833f52bcc8f239f9 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:30.929044 27462 leader_election.cc:304] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:30.929977 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 900) took 3279 ms. Trace:
I20250903 20:05:30.930081 28710 raft_consensus.cc:2802] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250903 20:05:30.930172 28710 raft_consensus.cc:491] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:30.930267 28710 raft_consensus.cc:3058] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Advancing to term 2
I20250903 20:05:30.932976 28710 raft_consensus.cc:513] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:30.933625 28710 leader_election.cc:290] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 2 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:30.933910 27687 rpcz_store.cc:276] 0903 20:05:27.650404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.650458 (+ 54us) service_pool.cc:225] Handling call
0903 20:05:30.881695 (+3231237us) ts_tablet_manager.cc:1145] Deleting tablet 8a223f75045345ae833f52bcc8f239f9
0903 20:05:30.881708 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.901931 (+ 20223us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.918193 (+ 16262us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.929954 (+ 11761us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1044,"tablet-delete.queue_time_us":3231035}
I20250903 20:05:30.934571 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 8a223f75045345ae833f52bcc8f239f9 (table default.Table_04bd610b6ae24395b30dc33c4d118258 [id=02a2c253dfb64f2cac5f70bde132aceb]) successfully deleted
I20250903 20:05:30.935137 27684 tablet_replica.cc:331] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:30.935472 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 901) took 3267 ms. Trace:
I20250903 20:05:30.935757 27684 raft_consensus.cc:2241] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:30.936136 27684 raft_consensus.cc:2270] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:30.936281 27687 rpcz_store.cc:276] 0903 20:05:27.667922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.668025 (+ 103us) service_pool.cc:225] Handling call
0903 20:05:30.935330 (+3267305us) ts_tablet_manager.cc:1145] Deleting tablet 733f8024e1fa401ca9614a97db91a19b
0903 20:05:30.935339 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.935451 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3267077}
W20250903 20:05:30.937129 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 733f8024e1fa401ca9614a97db91a19b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 733f8024e1fa401ca9614a97db91a19b already in progress: creating tablet
I20250903 20:05:30.933844 27693 tablet_bootstrap.cc:654] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:30.938483 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 903) took 3029 ms. Trace:
I20250903 20:05:30.937907 27692 tablet_bootstrap.cc:492] T 67ca1dcf6fc7435db555392235f9900f P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:30.938766 27692 ts_tablet_manager.cc:1397] T 67ca1dcf6fc7435db555392235f9900f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.001s
I20250903 20:05:30.939261 27684 ts_tablet_manager.cc:1905] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:30.939391 27687 rpcz_store.cc:276] 0903 20:05:27.908677 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.909028 (+ 351us) service_pool.cc:225] Handling call
0903 20:05:30.938346 (+3029318us) ts_tablet_manager.cc:1145] Deleting tablet 475d05d087c34159b093b92cc308bcaf
0903 20:05:30.938357 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.938464 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3029107}
W20250903 20:05:30.939985 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 475d05d087c34159b093b92cc308bcaf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 475d05d087c34159b093b92cc308bcaf already in progress: creating tablet
I20250903 20:05:30.940455 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 904) took 3030 ms. Trace:
I20250903 20:05:30.939502 27692 raft_consensus.cc:357] T 67ca1dcf6fc7435db555392235f9900f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:30.940797 27692 raft_consensus.cc:383] T 67ca1dcf6fc7435db555392235f9900f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:30.940872 27692 raft_consensus.cc:738] T 67ca1dcf6fc7435db555392235f9900f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:30.941082 27692 consensus_queue.cc:260] T 67ca1dcf6fc7435db555392235f9900f P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:30.941303 27687 rpcz_store.cc:276] 0903 20:05:27.909849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.909902 (+ 53us) service_pool.cc:225] Handling call
0903 20:05:30.940334 (+3030432us) ts_tablet_manager.cc:1145] Deleting tablet 5fe1de01bf7a4d418e765ea3e8cbecc1
0903 20:05:30.940345 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.940438 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3030244}
I20250903 20:05:30.941650 27692 ts_tablet_manager.cc:1428] T 67ca1dcf6fc7435db555392235f9900f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20250903 20:05:30.941872 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 5fe1de01bf7a4d418e765ea3e8cbecc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5fe1de01bf7a4d418e765ea3e8cbecc1 already in progress: creating tablet
I20250903 20:05:30.942339 27692 tablet_bootstrap.cc:492] T 726f4d920f3e47fea41b4e12c021c988 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:30.942633 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 905) took 3030 ms. Trace:
I20250903 20:05:30.943776 27692 tablet_bootstrap.cc:654] T 726f4d920f3e47fea41b4e12c021c988 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:30.943907 27687 rpcz_store.cc:276] 0903 20:05:27.912215 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.912266 (+ 51us) service_pool.cc:225] Handling call
0903 20:05:30.942511 (+3030245us) ts_tablet_manager.cc:1145] Deleting tablet 3e930dbaac0e40ee8f175ccf87592d31
0903 20:05:30.942521 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.942616 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3030089}
W20250903 20:05:30.949752 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 3e930dbaac0e40ee8f175ccf87592d31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e930dbaac0e40ee8f175ccf87592d31 already in progress: creating tablet
I20250903 20:05:30.950330 27687 tablet_replica.cc:331] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:30.950637 27687 raft_consensus.cc:2241] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:30.950875 27687 raft_consensus.cc:2270] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:30.951102 27692 tablet_bootstrap.cc:492] T 726f4d920f3e47fea41b4e12c021c988 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:30.951257 27692 ts_tablet_manager.cc:1397] T 726f4d920f3e47fea41b4e12c021c988 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.002s
I20250903 20:05:30.952008 27692 raft_consensus.cc:357] T 726f4d920f3e47fea41b4e12c021c988 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.952183 27692 raft_consensus.cc:383] T 726f4d920f3e47fea41b4e12c021c988 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:30.952236 27692 raft_consensus.cc:738] T 726f4d920f3e47fea41b4e12c021c988 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:30.952550 27692 consensus_queue.cc:260] T 726f4d920f3e47fea41b4e12c021c988 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.952807 27687 ts_tablet_manager.cc:1905] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:30.953689 27692 ts_tablet_manager.cc:1428] T 726f4d920f3e47fea41b4e12c021c988 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:30.953996 27692 tablet_bootstrap.cc:492] T 2059ccbd6d884085886d7d564afe03aa P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:30.955399 27692 tablet_bootstrap.cc:654] T 2059ccbd6d884085886d7d564afe03aa P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:30.956614 27693 tablet_bootstrap.cc:492] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:30.956818 27693 ts_tablet_manager.cc:1397] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20250903 20:05:30.957612 27693 raft_consensus.cc:357] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.957846 27693 raft_consensus.cc:383] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:30.957929 27693 raft_consensus.cc:738] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:30.958189 27693 consensus_queue.cc:260] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.958729 27693 ts_tablet_manager.cc:1428] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:30.959054 27693 tablet_bootstrap.cc:492] T 567630e2a93645e58f9bf05ffff87b32 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:30.959964 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_dbaa47c36c6f43fab8828747862ebbaa" } new_table_name: "default.Table_10275f9babd149008387d2c114944d27" modify_external_catalogs: true
I20250903 20:05:30.963526 27684 ts_tablet_manager.cc:1918] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:30.963650 27684 log.cc:1199] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/c9e90abef9814338b70a3550ab5686e0
I20250903 20:05:30.964005 27684 ts_tablet_manager.cc:1939] T c9e90abef9814338b70a3550ab5686e0 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:30.964787 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 840) took 4038 ms. Trace:
I20250903 20:05:30.964922 27684 rpcz_store.cc:276] 0903 20:05:26.926749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.926814 (+ 65us) service_pool.cc:225] Handling call
0903 20:05:30.935065 (+4008251us) ts_tablet_manager.cc:1145] Deleting tablet c9e90abef9814338b70a3550ab5686e0
0903 20:05:30.935083 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.960287 (+ 25204us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.963413 (+ 3126us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.964763 (+ 1350us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4007984}
I20250903 20:05:30.965713 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet c9e90abef9814338b70a3550ab5686e0 (table default.Table_5b44669e91a8427c8e7d96b866fdb3b5 [id=96167b0d9f9b47168a9fc59635b7136d]) successfully deleted
I20250903 20:05:30.966497 27695 tablet_bootstrap.cc:492] T b7a6d5d8a97a4769a81ec7224d591f8c P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:30.966687 27695 ts_tablet_manager.cc:1397] T b7a6d5d8a97a4769a81ec7224d591f8c P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.279s user 0.018s sys 0.000s
I20250903 20:05:30.967535 27695 raft_consensus.cc:357] T b7a6d5d8a97a4769a81ec7224d591f8c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.967752 27695 raft_consensus.cc:383] T b7a6d5d8a97a4769a81ec7224d591f8c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:30.967837 27695 raft_consensus.cc:738] T b7a6d5d8a97a4769a81ec7224d591f8c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:30.968024 27692 tablet_bootstrap.cc:492] T 2059ccbd6d884085886d7d564afe03aa P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:30.968189 27692 ts_tablet_manager.cc:1397] T 2059ccbd6d884085886d7d564afe03aa P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.002s
I20250903 20:05:30.968843 27692 raft_consensus.cc:357] T 2059ccbd6d884085886d7d564afe03aa P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.969081 27692 raft_consensus.cc:383] T 2059ccbd6d884085886d7d564afe03aa P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:30.969149 27692 raft_consensus.cc:738] T 2059ccbd6d884085886d7d564afe03aa P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:30.969388 27693 tablet_bootstrap.cc:654] T 567630e2a93645e58f9bf05ffff87b32 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:30.969364 27692 consensus_queue.cc:260] T 2059ccbd6d884085886d7d564afe03aa P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.969980 27692 ts_tablet_manager.cc:1428] T 2059ccbd6d884085886d7d564afe03aa P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:30.970322 27692 tablet_bootstrap.cc:492] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:30.971905 27692 tablet_bootstrap.cc:654] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:30.972538 28848 tablet.cc:1722] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:30.981326 27692 tablet_bootstrap.cc:492] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:30.981490 27692 ts_tablet_manager.cc:1397] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20250903 20:05:30.981921 27687 ts_tablet_manager.cc:1918] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:30.982062 27687 log.cc:1199] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/673c45f3f2a14da9bd306011d396eaca
I20250903 20:05:30.982427 27687 ts_tablet_manager.cc:1939] T 673c45f3f2a14da9bd306011d396eaca P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:30.982323 27692 raft_consensus.cc:357] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.982494 27692 raft_consensus.cc:383] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:30.982553 27692 raft_consensus.cc:738] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:30.982806 27692 consensus_queue.cc:260] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.983238 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 911) took 2544 ms. Trace:
I20250903 20:05:30.983417 27692 ts_tablet_manager.cc:1428] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:30.983342 27687 rpcz_store.cc:276] 0903 20:05:28.438556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.438721 (+ 165us) service_pool.cc:225] Handling call
0903 20:05:30.950268 (+2511547us) ts_tablet_manager.cc:1145] Deleting tablet 673c45f3f2a14da9bd306011d396eaca
0903 20:05:30.950280 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.971432 (+ 21152us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.981794 (+ 10362us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.983216 (+ 1422us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2511358}
I20250903 20:05:30.983844 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 673c45f3f2a14da9bd306011d396eaca (table default.Table_43ea308240ac4628a72434608a22b975 [id=8de1ad04c15648d9bcccfc56efc9e7fb]) successfully deleted
I20250903 20:05:30.984494 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 919) took 2202 ms. Trace:
I20250903 20:05:30.984622 27687 rpcz_store.cc:276] 0903 20:05:28.782005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.782088 (+ 83us) service_pool.cc:225] Handling call
0903 20:05:30.984363 (+2202275us) ts_tablet_manager.cc:1145] Deleting tablet 4739f10128d64e2c986257b1d9deb90f
0903 20:05:30.984372 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.984476 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2202007}
W20250903 20:05:30.985213 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 4739f10128d64e2c986257b1d9deb90f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4739f10128d64e2c986257b1d9deb90f already in progress: creating tablet
I20250903 20:05:30.985859 28885 tablet.cc:1722] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:30.985949 27692 tablet_bootstrap.cc:492] T 541278ca9ffb4e41b341139c2faa337f P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:30.986256 27693 tablet_bootstrap.cc:492] T 567630e2a93645e58f9bf05ffff87b32 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:30.986384 27693 ts_tablet_manager.cc:1397] T 567630e2a93645e58f9bf05ffff87b32 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20250903 20:05:30.987534 27693 raft_consensus.cc:357] T 567630e2a93645e58f9bf05ffff87b32 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.987738 27692 tablet_bootstrap.cc:654] T 541278ca9ffb4e41b341139c2faa337f P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:30.987742 27693 raft_consensus.cc:383] T 567630e2a93645e58f9bf05ffff87b32 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:30.987797 27693 raft_consensus.cc:738] T 567630e2a93645e58f9bf05ffff87b32 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:30.988058 27693 consensus_queue.cc:260] T 567630e2a93645e58f9bf05ffff87b32 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:30.988647 27693 ts_tablet_manager.cc:1428] T 567630e2a93645e58f9bf05ffff87b32 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:30.988921 27693 tablet_bootstrap.cc:492] T a0f79831866b40c49c5b27ce2014c3f6 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:30.989589 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 845) took 3965 ms. Trace:
I20250903 20:05:30.989683 27684 rpcz_store.cc:276] 0903 20:05:27.024569 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.024632 (+ 63us) service_pool.cc:225] Handling call
0903 20:05:30.984212 (+3959580us) ts_tablet_manager.cc:1145] Deleting tablet 44fef873a0644dd988322847f333ec95
0903 20:05:30.989455 (+ 5243us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.989566 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3959348}
I20250903 20:05:30.989818 27695 consensus_queue.cc:260] T b7a6d5d8a97a4769a81ec7224d591f8c P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:30.990195 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 44fef873a0644dd988322847f333ec95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44fef873a0644dd988322847f333ec95 already in progress: creating tablet
I20250903 20:05:30.990310 28967 tablet.cc:1722] T 3b3bfc63aa0b4e56bec189e834dc6a8d P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:30.991772 27693 tablet_bootstrap.cc:654] T a0f79831866b40c49c5b27ce2014c3f6 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:30.992223 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 843) took 3967 ms. Trace:
I20250903 20:05:30.992367 27684 rpcz_store.cc:276] 0903 20:05:27.024322 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.025083 (+ 761us) service_pool.cc:225] Handling call
0903 20:05:30.991977 (+3966894us) ts_tablet_manager.cc:1145] Deleting tablet 553443eda16c44fa84b1db22b1363808
0903 20:05:30.991987 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.992209 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3966725}
W20250903 20:05:30.992976 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 553443eda16c44fa84b1db22b1363808 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 553443eda16c44fa84b1db22b1363808 already in progress: creating tablet
I20250903 20:05:30.993434 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 844) took 3968 ms. Trace:
I20250903 20:05:30.993552 27684 rpcz_store.cc:276] 0903 20:05:27.024482 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.025284 (+ 802us) service_pool.cc:225] Handling call
0903 20:05:30.993323 (+3968039us) ts_tablet_manager.cc:1145] Deleting tablet e0b9172ebb484382adb86e597f5cbc3b
0903 20:05:30.993333 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.993418 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3967900}
W20250903 20:05:30.994035 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet e0b9172ebb484382adb86e597f5cbc3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0b9172ebb484382adb86e597f5cbc3b already in progress: creating tablet
I20250903 20:05:30.994606 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 846) took 3969 ms. Trace:
I20250903 20:05:30.995611 27684 rpcz_store.cc:276] 0903 20:05:27.024690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.025440 (+ 750us) service_pool.cc:225] Handling call
0903 20:05:30.994501 (+3969061us) ts_tablet_manager.cc:1145] Deleting tablet ea3f82f4e4244219ab0919f3e341aead
0903 20:05:30.994512 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.994590 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3968964}
W20250903 20:05:30.996143 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet ea3f82f4e4244219ab0919f3e341aead because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea3f82f4e4244219ab0919f3e341aead already in progress: creating tablet
I20250903 20:05:30.996150 27695 ts_tablet_manager.cc:1428] T b7a6d5d8a97a4769a81ec7224d591f8c P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20250903 20:05:30.996569 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 847) took 3970 ms. Trace:
I20250903 20:05:30.996685 27684 rpcz_store.cc:276] 0903 20:05:27.026408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.026451 (+ 43us) service_pool.cc:225] Handling call
0903 20:05:30.996463 (+3970012us) ts_tablet_manager.cc:1145] Deleting tablet 81852d8e06ae4c6bb715a88d8c398bfc
0903 20:05:30.996473 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.996555 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3969847}
W20250903 20:05:30.997241 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 81852d8e06ae4c6bb715a88d8c398bfc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81852d8e06ae4c6bb715a88d8c398bfc already in progress: creating tablet
I20250903 20:05:30.997627 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 848) took 3783 ms. Trace:
I20250903 20:05:30.997745 27684 rpcz_store.cc:276] 0903 20:05:27.214149 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.214232 (+ 83us) service_pool.cc:225] Handling call
0903 20:05:30.997520 (+3783288us) ts_tablet_manager.cc:1145] Deleting tablet ce3be9d44147420d953e716a347352c9
0903 20:05:30.997529 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.997612 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3783094}
W20250903 20:05:30.998266 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet ce3be9d44147420d953e716a347352c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce3be9d44147420d953e716a347352c9 already in progress: creating tablet
I20250903 20:05:30.998656 27684 tablet_replica.cc:331] T 1bcb9d5f010f4297bab78c0183889238 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:30.998901 27684 raft_consensus.cc:2241] T 1bcb9d5f010f4297bab78c0183889238 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:30.999071 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 920) took 2216 ms. Trace:
I20250903 20:05:30.999337 27684 raft_consensus.cc:2270] T 1bcb9d5f010f4297bab78c0183889238 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:30.999449 27687 rpcz_store.cc:276] 0903 20:05:28.782176 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.782404 (+ 228us) service_pool.cc:225] Handling call
0903 20:05:30.985566 (+2203162us) ts_tablet_manager.cc:1145] Deleting tablet b3bb55085af044ec8bb2368c845e7404
0903 20:05:30.985577 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.999038 (+ 13461us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2202992}
I20250903 20:05:31.001893 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 762) took 5536 ms. Trace:
I20250903 20:05:31.002050 27685 rpcz_store.cc:276] 0903 20:05:25.465199 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:25.465252 (+ 53us) service_pool.cc:225] Handling call
0903 20:05:30.832211 (+5366959us) ts_tablet_manager.cc:1145] Deleting tablet 9685cba2f73b412382a69dab2c7f50c1
0903 20:05:30.832230 (+ 19us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:30.848423 (+ 16193us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:30.855843 (+ 7420us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.001827 (+145984us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10933,"tablet-delete.queue_time_us":5366761}
I20250903 20:05:31.002627 27685 tablet_replica.cc:331] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:31.002743 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 9685cba2f73b412382a69dab2c7f50c1 (table default.Table_0cd65e00bd5a40bd9dc9db12fd0bda4d [id=15369700921549be8faa450d2ca98949]) successfully deleted
I20250903 20:05:31.002912 27685 raft_consensus.cc:2241] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a [term 2 LEADER]: Raft consensus shutting down.
I20250903 20:05:31.003227 27685 pending_rounds.cc:70] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a: Trying to abort 1 pending ops.
I20250903 20:05:31.003298 27685 pending_rounds.cc:77] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a: Aborting op as it isn't in flight: id { term: 2 index: 1 } timestamp: 7196384995411906560 op_type: NO_OP noop_request { }
W20250903 20:05:31.003405 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet b3bb55085af044ec8bb2368c845e7404 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3bb55085af044ec8bb2368c845e7404 already in progress: creating tablet
I20250903 20:05:31.003504 27685 raft_consensus.cc:2887] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:31.003607 27685 raft_consensus.cc:2270] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:31.004281 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 922) took 2221 ms. Trace:
I20250903 20:05:31.005056 27687 rpcz_store.cc:276] 0903 20:05:28.782495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.782597 (+ 102us) service_pool.cc:225] Handling call
0903 20:05:31.004124 (+2221527us) ts_tablet_manager.cc:1145] Deleting tablet 5962685d79044081a75532e19bfa2bce
0903 20:05:31.004137 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.004257 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2221406}
I20250903 20:05:31.005434 27684 ts_tablet_manager.cc:1905] T 1bcb9d5f010f4297bab78c0183889238 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:31.006458 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 5962685d79044081a75532e19bfa2bce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5962685d79044081a75532e19bfa2bce already in progress: creating tablet
I20250903 20:05:31.008116 27695 tablet_bootstrap.cc:492] T 67ca1dcf6fc7435db555392235f9900f P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:31.008342 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 921) took 2226 ms. Trace:
I20250903 20:05:31.009904 27695 tablet_bootstrap.cc:654] T 67ca1dcf6fc7435db555392235f9900f P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:31.010527 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 90776f6369744cec8ce380b7695c44f1
I20250903 20:05:31.010533 27687 rpcz_store.cc:276] 0903 20:05:28.782352 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.784264 (+ 1912us) service_pool.cc:225] Handling call
0903 20:05:31.007069 (+2222805us) ts_tablet_manager.cc:1145] Deleting tablet de20d18d7f264811a0631aa1144aef2c
0903 20:05:31.007082 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.008317 (+ 1235us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2222651}
I20250903 20:05:31.011139 27058 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:45936 (request call id 1145) took 1012 ms. Trace:
W20250903 20:05:31.011170 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet de20d18d7f264811a0631aa1144aef2c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de20d18d7f264811a0631aa1144aef2c already in progress: creating tablet
I20250903 20:05:31.011263 27058 rpcz_store.cc:276] 0903 20:05:29.999096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.717989 (+718893us) service_pool.cc:225] Handling call
0903 20:05:30.718597 (+ 608us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0903 20:05:30.718719 (+ 122us) catalog_manager.cc:2790] Modifying in-memory table state
0903 20:05:30.718826 (+ 107us) catalog_manager.cc:2799] Locking tablets
0903 20:05:30.719103 (+ 277us) catalog_manager.cc:2813] Removing table and tablets from system table
0903 20:05:31.010306 (+291203us) catalog_manager.cc:2831] Removing table from by-name map
0903 20:05:31.010402 (+ 96us) catalog_manager.cc:2847] Committing in-memory state
0903 20:05:31.010448 (+ 46us) catalog_manager.cc:2851] Aborting table tasks
0903 20:05:31.011086 (+ 638us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:30.962556 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:30.962670 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:30.962674 (+ 4us) write_op.cc:435] Acquired schema lock
0903 20:05:30.962676 (+ 2us) tablet.cc:660] Decoding operations
0903 20:05:30.962804 (+ 128us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:30.962819 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:30.962824 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0903 20:05:30.962942 (+ 118us) tablet.cc:699] Row locks acquired
0903 20:05:30.962944 (+ 2us) write_op.cc:260] PREPARE: finished
0903 20:05:30.962986 (+ 42us) write_op.cc:270] Start()
0903 20:05:30.963021 (+ 35us) write_op.cc:276] Timestamp: P: 1756929930962942 usec, L: 0
0903 20:05:30.963025 (+ 4us) op_driver.cc:348] REPLICATION: starting
0903 20:05:30.963246 (+ 221us) log.cc:844] Serialized 759 byte log entry
0903 20:05:31.009314 (+ 46068us) op_driver.cc:464] REPLICATION: finished
0903 20:05:31.009498 (+ 184us) write_op.cc:301] APPLY: starting
0903 20:05:31.009516 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:31.009566 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:31.009599 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:31.009674 (+ 75us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:31.009717 (+ 43us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0903 20:05:31.009729 (+ 12us) write_op.cc:312] APPLY: finished
0903 20:05:31.009821 (+ 92us) log.cc:844] Serialized 29 byte log entry
0903 20:05:31.009889 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:31.009912 (+ 23us) write_op.cc:454] Released schema lock
0903 20:05:31.009920 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":129,"apply.run_cpu_time_us":493,"apply.run_wall_time_us":493,"num_ops":2,"prepare.queue_time_us":242700,"raft.queue_time_us":51,"raft.run_cpu_time_us":372,"raft.run_wall_time_us":372,"replication_time_us":46254}]]}
I20250903 20:05:31.011746 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet 90776f6369744cec8ce380b7695c44f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:30 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:31.011973 27685 ts_tablet_manager.cc:1905] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:31.013993 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 90776f6369744cec8ce380b7695c44f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:30 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:31.014475 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 923) took 2202 ms. Trace:
I20250903 20:05:31.014571 27687 rpcz_store.cc:276] 0903 20:05:28.811912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.811981 (+ 69us) service_pool.cc:225] Handling call
0903 20:05:31.014331 (+2202350us) ts_tablet_manager.cc:1145] Deleting tablet 13496efe34944b7280e7a684f180e6e0
0903 20:05:31.014343 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.014455 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2202140}
W20250903 20:05:31.015033 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 13496efe34944b7280e7a684f180e6e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13496efe34944b7280e7a684f180e6e0 already in progress: creating tablet
I20250903 20:05:31.017210 27687 tablet_replica.cc:331] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:31.017460 27687 raft_consensus.cc:2241] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:31.018393 27227 tablet_service.cc:1515] Processing DeleteTablet for tablet 90776f6369744cec8ce380b7695c44f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:30 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:31.022616 27687 raft_consensus.cc:2270] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:31.024583 27687 ts_tablet_manager.cc:1905] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:31.025854 26885 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskH7eH49/build/asan/bin/kudu pid 28184
I20250903 20:05:31.031167 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_182ace2fdd7f4619b244f455e93d4df9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:31.032137 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_182ace2fdd7f4619b244f455e93d4df9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:31.040606 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "514e382479014fa7bf1f404efff11c86" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:31.040711 27685 ts_tablet_manager.cc:1918] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20250903 20:05:31.040889 27240 raft_consensus.cc:3058] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Advancing to term 2
W20250903 20:05:31.042747 27183 consensus_peers.cc:489] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:31.043939 27240 raft_consensus.cc:2466] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 2.
I20250903 20:05:31.045208 27684 ts_tablet_manager.cc:1918] T 1bcb9d5f010f4297bab78c0183889238 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:31.045372 27684 log.cc:1199] T 1bcb9d5f010f4297bab78c0183889238 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/1bcb9d5f010f4297bab78c0183889238
I20250903 20:05:31.045823 27684 ts_tablet_manager.cc:1939] T 1bcb9d5f010f4297bab78c0183889238 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:31.053412 27695 tablet_bootstrap.cc:492] T 67ca1dcf6fc7435db555392235f9900f P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:31.053604 27695 ts_tablet_manager.cc:1397] T 67ca1dcf6fc7435db555392235f9900f P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.001s
W20250903 20:05:31.053601 27057 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fb9e90d6e97d4ab4b27bd5efb2ee4b55 consensus state has no leader
I20250903 20:05:31.053781 27057 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:45936 (request call id 1146) took 1040 ms. Trace:
I20250903 20:05:31.053882 27057 rpcz_store.cc:276] 0903 20:05:30.012941 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.720166 (+707225us) service_pool.cc:225] Handling call
0903 20:05:30.720495 (+ 329us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0903 20:05:30.720700 (+ 205us) catalog_manager.cc:3683] Apply alter schema
0903 20:05:30.720907 (+ 207us) catalog_manager.cc:3715] Acquired catalog manager lock
0903 20:05:30.720985 (+ 78us) catalog_manager.cc:3873] Updating metadata on disk
0903 20:05:31.053322 (+332337us) catalog_manager.cc:3904] Committing alterations to in-memory state
0903 20:05:31.053743 (+ 421us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:31.047875 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:31.047984 (+ 109us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:31.047988 (+ 4us) write_op.cc:435] Acquired schema lock
0903 20:05:31.047990 (+ 2us) tablet.cc:660] Decoding operations
0903 20:05:31.048097 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:31.048108 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:31.048115 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0903 20:05:31.048176 (+ 61us) tablet.cc:699] Row locks acquired
0903 20:05:31.048179 (+ 3us) write_op.cc:260] PREPARE: finished
0903 20:05:31.048221 (+ 42us) write_op.cc:270] Start()
0903 20:05:31.048254 (+ 33us) write_op.cc:276] Timestamp: P: 1756929931048169 usec, L: 0
0903 20:05:31.048257 (+ 3us) op_driver.cc:348] REPLICATION: starting
0903 20:05:31.048484 (+ 227us) log.cc:844] Serialized 405 byte log entry
0903 20:05:31.052597 (+ 4113us) op_driver.cc:464] REPLICATION: finished
0903 20:05:31.052802 (+ 205us) write_op.cc:301] APPLY: starting
0903 20:05:31.052820 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:31.052860 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:31.052886 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:31.052956 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:31.052989 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0903 20:05:31.052996 (+ 7us) write_op.cc:312] APPLY: finished
0903 20:05:31.053081 (+ 85us) log.cc:844] Serialized 23 byte log entry
0903 20:05:31.053145 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:31.053164 (+ 19us) write_op.cc:454] Released schema lock
0903 20:05:31.053170 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":149,"apply.run_cpu_time_us":435,"apply.run_wall_time_us":435,"num_ops":1,"prepare.queue_time_us":326431,"raft.queue_time_us":42,"replication_time_us":4307}]]}
I20250903 20:05:31.055306 27685 log.cc:1199] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/f2cc35f4244a42eeb4a65de749824c11
I20250903 20:05:31.055807 27685 ts_tablet_manager.cc:1939] T f2cc35f4244a42eeb4a65de749824c11 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:31.056437 27695 raft_consensus.cc:357] T 67ca1dcf6fc7435db555392235f9900f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:31.059078 27695 raft_consensus.cc:383] T 67ca1dcf6fc7435db555392235f9900f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:31.059237 27695 raft_consensus.cc:738] T 67ca1dcf6fc7435db555392235f9900f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:31.059569 27695 consensus_queue.cc:260] T 67ca1dcf6fc7435db555392235f9900f P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:31.063205 27695 ts_tablet_manager.cc:1428] T 67ca1dcf6fc7435db555392235f9900f P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250903 20:05:31.063644 27695 tablet_bootstrap.cc:492] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:31.065399 27695 tablet_bootstrap.cc:654] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:31.069563 27687 ts_tablet_manager.cc:1918] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:31.070184 27687 log.cc:1199] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/20c25c84c6fb4609a1964a4ff3c526da
I20250903 20:05:31.070832 28751 raft_consensus.cc:491] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:31.070971 28751 raft_consensus.cc:513] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.071098 27687 ts_tablet_manager.cc:1939] T 20c25c84c6fb4609a1964a4ff3c526da P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:31.071615 28751 leader_election.cc:290] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:31.072036 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 924) took 2109 ms. Trace:
I20250903 20:05:31.072121 27687 rpcz_store.cc:276] 0903 20:05:28.962598 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.962664 (+ 66us) service_pool.cc:225] Handling call
0903 20:05:31.017144 (+2054480us) ts_tablet_manager.cc:1145] Deleting tablet 20c25c84c6fb4609a1964a4ff3c526da
0903 20:05:31.017156 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.046952 (+ 29796us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.068740 (+ 21788us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.072012 (+ 3272us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2054236}
I20250903 20:05:31.072705 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 20c25c84c6fb4609a1964a4ff3c526da (table default.Table_6b435f3c61214c74a632c94e955abcdf [id=c2cf9ce651fc41a9b2b223b998e45d9b]) successfully deleted
I20250903 20:05:31.075979 28463 generic_service.cc:196] Checking for leaks (request via RPC)
I20250903 20:05:31.078874 27462 leader_election.cc:304] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:31.079296 28596 raft_consensus.cc:2802] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f [term 2 FOLLOWER]: Leader election won for term 2
I20250903 20:05:31.079416 28596 raft_consensus.cc:695] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f [term 2 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:31.079641 28596 consensus_queue.cc:237] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:31.082564 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 857) took 3697 ms. Trace:
I20250903 20:05:31.082669 27684 rpcz_store.cc:276] 0903 20:05:27.386159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.386257 (+ 98us) service_pool.cc:225] Handling call
0903 20:05:30.998612 (+3612355us) ts_tablet_manager.cc:1145] Deleting tablet 1bcb9d5f010f4297bab78c0183889238
0903 20:05:30.998621 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.027569 (+ 28948us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.045041 (+ 17472us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.082540 (+ 37499us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24732,"tablet-delete.queue_time_us":3612068}
I20250903 20:05:31.085638 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 1bcb9d5f010f4297bab78c0183889238 (table default.Table_adf05e59a7be478b89666c3162ac83f1 [id=e69321e60a3a4e71a463a0011c0c6d65]) successfully deleted
I20250903 20:05:31.088505 27057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ee99c37551924b8fb11237c20dc777de" } new_table_name: "default.Table_27ad52e9c3794a6f8b8959abe991f5f0" modify_external_catalogs: true
I20250903 20:05:31.094640 27687 tablet_replica.cc:331] T 0a9b7770432942da9373d5100c71bd6a P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:31.094924 27687 raft_consensus.cc:2241] T 0a9b7770432942da9373d5100c71bd6a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:31.095155 27687 raft_consensus.cc:2270] T 0a9b7770432942da9373d5100c71bd6a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:31.098094 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 96c9875830b246b3a9117536254cbc4e
I20250903 20:05:31.098431 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 765) took 5350 ms. Trace:
I20250903 20:05:31.098551 27685 rpcz_store.cc:276] 0903 20:05:25.747663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:25.747721 (+ 58us) service_pool.cc:225] Handling call
0903 20:05:31.002533 (+5254812us) ts_tablet_manager.cc:1145] Deleting tablet f2cc35f4244a42eeb4a65de749824c11
0903 20:05:31.002545 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.031906 (+ 29361us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.040526 (+ 8620us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.098400 (+ 57874us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5254637}
I20250903 20:05:31.099048 27063 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:45936 (request call id 1143) took 1243 ms. Trace:
I20250903 20:05:31.099110 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet f2cc35f4244a42eeb4a65de749824c11 (table default.Table_78e7a5b76ebc469396d8a3d670ab63ee [id=fb43441ca8f2415d99a628e81ef1642f]) successfully deleted
I20250903 20:05:31.099298 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 96c9875830b246b3a9117536254cbc4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:30 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:31.099164 27063 rpcz_store.cc:276] 0903 20:05:29.855331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.666875 (+811544us) service_pool.cc:225] Handling call
0903 20:05:30.720826 (+ 53951us) spinlock_profiling.cc:243] Waited 53.7 ms on lock 0x616000040f54. stack: 00007f4b9d6c2dc6 00007f4b9d6c2b80 00007f4b9c6aa3ad 0000000000972d94 0000000000972d5d 00007f4bac49014d 00007f4bac490243 00007f4ba914f941 00007f4bb0cdaa7d 00007f4bb0e64ea6 00007f4ba7c9d466 00007f4ba7c9d195 00007f4ba44730be 00007f4ba4471f9e 00007f4ba44781d5 00007f4ba4478aff
0903 20:05:30.721191 (+ 365us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0903 20:05:30.721333 (+ 142us) catalog_manager.cc:2790] Modifying in-memory table state
0903 20:05:30.721394 (+ 61us) catalog_manager.cc:2799] Locking tablets
0903 20:05:30.721535 (+ 141us) catalog_manager.cc:2813] Removing table and tablets from system table
0903 20:05:31.097834 (+376299us) catalog_manager.cc:2831] Removing table from by-name map
0903 20:05:31.097945 (+ 111us) catalog_manager.cc:2847] Committing in-memory state
0903 20:05:31.097998 (+ 53us) catalog_manager.cc:2851] Aborting table tasks
0903 20:05:31.098962 (+ 964us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:31.090961 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:31.091125 (+ 164us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:31.091132 (+ 7us) write_op.cc:435] Acquired schema lock
0903 20:05:31.091136 (+ 4us) tablet.cc:660] Decoding operations
0903 20:05:31.091284 (+ 148us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:31.091301 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:31.091308 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0903 20:05:31.091446 (+ 138us) tablet.cc:699] Row locks acquired
0903 20:05:31.091450 (+ 4us) write_op.cc:260] PREPARE: finished
0903 20:05:31.091489 (+ 39us) write_op.cc:270] Start()
0903 20:05:31.091531 (+ 42us) write_op.cc:276] Timestamp: P: 1756929931091435 usec, L: 0
0903 20:05:31.091537 (+ 6us) op_driver.cc:348] REPLICATION: starting
0903 20:05:31.091788 (+ 251us) log.cc:844] Serialized 728 byte log entry
0903 20:05:31.096934 (+ 5146us) op_driver.cc:464] REPLICATION: finished
0903 20:05:31.097170 (+ 236us) write_op.cc:301] APPLY: starting
0903 20:05:31.097191 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:31.097253 (+ 62us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:31.097306 (+ 53us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:31.097398 (+ 92us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:31.097447 (+ 49us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0903 20:05:31.097458 (+ 11us) write_op.cc:312] APPLY: finished
0903 20:05:31.097564 (+ 106us) log.cc:844] Serialized 29 byte log entry
0903 20:05:31.097642 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:31.097670 (+ 28us) write_op.cc:454] Released schema lock
0903 20:05:31.097680 (+ 10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":123504000,"child_traces":[["op",{"apply.queue_time_us":175,"apply.run_cpu_time_us":582,"apply.run_wall_time_us":581,"num_ops":2,"prepare.queue_time_us":368565,"raft.queue_time_us":79,"raft.run_cpu_time_us":680,"raft.run_wall_time_us":1637,"replication_time_us":5358}]]}
I20250903 20:05:31.099646 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet 96c9875830b246b3a9117536254cbc4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:30 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:31.099818 27063 master_service.cc:948] ReplaceTablet: received request to replace tablet 64d53f2071b0449ca61caa6c264a3641 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:31.100994 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "120ca61b3da742dc9a2efa483ead7e97" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:31.101233 27240 raft_consensus.cc:2466] T 120ca61b3da742dc9a2efa483ead7e97 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:31.101253 27687 ts_tablet_manager.cc:1905] T 0a9b7770432942da9373d5100c71bd6a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:31.101615 27321 leader_election.cc:304] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:31.102123 28751 raft_consensus.cc:2802] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:31.102195 28751 raft_consensus.cc:491] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:31.102270 28751 raft_consensus.cc:3058] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.104744 28751 raft_consensus.cc:513] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.105862 28751 leader_election.cc:290] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:31.106436 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "120ca61b3da742dc9a2efa483ead7e97" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:31.106624 27240 raft_consensus.cc:3058] T 120ca61b3da742dc9a2efa483ead7e97 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.109186 27240 raft_consensus.cc:2466] T 120ca61b3da742dc9a2efa483ead7e97 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:31.109697 27321 leader_election.cc:304] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:31.110234 28751 raft_consensus.cc:2802] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:31.110366 28751 raft_consensus.cc:695] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:31.110615 28751 consensus_queue.cc:237] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.114841 27227 tablet_service.cc:1515] Processing DeleteTablet for tablet 96c9875830b246b3a9117536254cbc4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:30 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:31.115803 28870 raft_consensus.cc:491] T 961ade36f7d14eba91bb16d5eae2e254 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51578fb7bceb4e119b3ab1e58c4f5528)
I20250903 20:05:31.115940 28870 raft_consensus.cc:513] T 961ade36f7d14eba91bb16d5eae2e254 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:31.116510 28870 leader_election.cc:290] T 961ade36f7d14eba91bb16d5eae2e254 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:31.117276 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "961ade36f7d14eba91bb16d5eae2e254" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:31.119031 27184 leader_election.cc:343] T 961ade36f7d14eba91bb16d5eae2e254 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 961ade36f7d14eba91bb16d5eae2e254
I20250903 20:05:31.119165 27184 leader_election.cc:304] T 961ade36f7d14eba91bb16d5eae2e254 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:31.119570 28870 raft_consensus.cc:2747] T 961ade36f7d14eba91bb16d5eae2e254 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:31.120913 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 766) took 5372 ms. Trace:
I20250903 20:05:31.121004 27685 rpcz_store.cc:276] 0903 20:05:25.748835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:25.748879 (+ 44us) service_pool.cc:225] Handling call
0903 20:05:31.099022 (+5350143us) ts_tablet_manager.cc:1145] Deleting tablet 005d7bc2c7534f288caec8326fdf2416
0903 20:05:31.120783 (+ 21761us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.120893 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5349978}
I20250903 20:05:31.121479 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 767) took 5370 ms. Trace:
I20250903 20:05:31.121558 27685 rpcz_store.cc:276] 0903 20:05:25.751414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:25.751467 (+ 53us) service_pool.cc:225] Handling call
0903 20:05:31.121265 (+5369798us) ts_tablet_manager.cc:1145] Deleting tablet 3d875d29e66d48ed872e38d20f30d9c0
0903 20:05:31.121379 (+ 114us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.121462 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5369673}
W20250903 20:05:31.121675 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 005d7bc2c7534f288caec8326fdf2416 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 005d7bc2c7534f288caec8326fdf2416 already in progress: creating tablet
I20250903 20:05:31.121858 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 768) took 5358 ms. Trace:
I20250903 20:05:31.121917 27685 rpcz_store.cc:276] 0903 20:05:25.763400 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:25.763450 (+ 50us) service_pool.cc:225] Handling call
0903 20:05:31.121769 (+5358319us) ts_tablet_manager.cc:1145] Deleting tablet 24ddae9179cd4240914cbbae31d8b851
0903 20:05:31.121777 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.121845 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5358144}
W20250903 20:05:31.122568 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 3d875d29e66d48ed872e38d20f30d9c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d875d29e66d48ed872e38d20f30d9c0 already in progress: creating tablet
W20250903 20:05:31.122834 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 24ddae9179cd4240914cbbae31d8b851 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24ddae9179cd4240914cbbae31d8b851 already in progress: creating tablet
I20250903 20:05:31.122203 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 769) took 5318 ms. Trace:
I20250903 20:05:31.123183 27685 rpcz_store.cc:276] 0903 20:05:25.803487 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:25.803540 (+ 53us) service_pool.cc:225] Handling call
0903 20:05:31.122124 (+5318584us) ts_tablet_manager.cc:1145] Deleting tablet 45f7068bafaf48d290ded9d1272c0e46
0903 20:05:31.122131 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.122193 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5318412}
I20250903 20:05:31.123497 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 770) took 5254 ms. Trace:
I20250903 20:05:31.123584 27685 rpcz_store.cc:276] 0903 20:05:25.869393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:25.869450 (+ 57us) service_pool.cc:225] Handling call
0903 20:05:31.123409 (+5253959us) ts_tablet_manager.cc:1145] Deleting tablet b6663ba96faa44b88fe8b6def4ff6951
0903 20:05:31.123417 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.123484 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5084763}
I20250903 20:05:31.123822 27685 tablet_replica.cc:331] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:31.123999 27685 raft_consensus.cc:2241] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:31.124177 27685 raft_consensus.cc:2270] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a [term 2 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:31.124600 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 45f7068bafaf48d290ded9d1272c0e46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 45f7068bafaf48d290ded9d1272c0e46 already in progress: creating tablet
W20250903 20:05:31.124836 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet b6663ba96faa44b88fe8b6def4ff6951 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6663ba96faa44b88fe8b6def4ff6951 already in progress: creating tablet
I20250903 20:05:31.125474 27685 ts_tablet_manager.cc:1905] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:31.127351 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "120ca61b3da742dc9a2efa483ead7e97" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:31.128412 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "120ca61b3da742dc9a2efa483ead7e97" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:31.129750 27687 ts_tablet_manager.cc:1918] T 0a9b7770432942da9373d5100c71bd6a P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:31.129882 27687 log.cc:1199] T 0a9b7770432942da9373d5100c71bd6a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/0a9b7770432942da9373d5100c71bd6a
I20250903 20:05:31.130261 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 859) took 3722 ms. Trace:
I20250903 20:05:31.130352 27684 rpcz_store.cc:276] 0903 20:05:27.407539 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.407706 (+ 167us) service_pool.cc:225] Handling call
0903 20:05:31.087493 (+3679787us) ts_tablet_manager.cc:1145] Deleting tablet 91169fff8a6649828648ac97ccf65fdc
0903 20:05:31.130064 (+ 42571us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.130192 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3679457}
I20250903 20:05:31.130776 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 858) took 3723 ms. Trace:
I20250903 20:05:31.130853 27684 rpcz_store.cc:276] 0903 20:05:27.407399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.408078 (+ 679us) service_pool.cc:225] Handling call
0903 20:05:31.130678 (+3722600us) ts_tablet_manager.cc:1145] Deleting tablet edac2355f67045a6acffaae723d9186f
0903 20:05:31.130688 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.130761 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3722453}
I20250903 20:05:31.131161 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 862) took 3690 ms. Trace:
I20250903 20:05:31.131246 27684 rpcz_store.cc:276] 0903 20:05:27.440820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.447875 (+ 7055us) service_pool.cc:225] Handling call
0903 20:05:31.131075 (+3683200us) ts_tablet_manager.cc:1145] Deleting tablet cf99a74acfdb4a17bbbce3a07467e9c1
0903 20:05:31.131084 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.131148 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3682889}
I20250903 20:05:31.131525 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 861) took 3690 ms. Trace:
I20250903 20:05:31.131601 27684 rpcz_store.cc:276] 0903 20:05:27.440710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.448250 (+ 7540us) service_pool.cc:225] Handling call
0903 20:05:31.131433 (+3683183us) ts_tablet_manager.cc:1145] Deleting tablet 112c0dccb60c4c7c9efc164d9c8443ea
0903 20:05:31.131443 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.131509 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3683010}
I20250903 20:05:31.131892 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 860) took 3702 ms. Trace:
I20250903 20:05:31.131969 27684 rpcz_store.cc:276] 0903 20:05:27.429043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.448479 (+ 19436us) service_pool.cc:225] Handling call
0903 20:05:31.131804 (+3683325us) ts_tablet_manager.cc:1145] Deleting tablet a686ea49a29a4dd7b6e5dce5e1f9935a
0903 20:05:31.131812 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.131877 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3683127}
I20250903 20:05:31.132272 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 863) took 3691 ms. Trace:
I20250903 20:05:31.132347 27684 rpcz_store.cc:276] 0903 20:05:27.440953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.448731 (+ 7778us) service_pool.cc:225] Handling call
0903 20:05:31.132167 (+3683436us) ts_tablet_manager.cc:1145] Deleting tablet 30cc0d4ba8d444f1b98232f906ea0c7b
0903 20:05:31.132175 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.132256 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3682777}
I20250903 20:05:31.132577 27684 tablet_replica.cc:331] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:31.132771 27684 raft_consensus.cc:2241] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f [term 2 LEADER]: Raft consensus shutting down.
I20250903 20:05:31.133019 27684 pending_rounds.cc:70] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f: Trying to abort 1 pending ops.
W20250903 20:05:31.133850 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 91169fff8a6649828648ac97ccf65fdc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91169fff8a6649828648ac97ccf65fdc already in progress: creating tablet
W20250903 20:05:31.134130 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet edac2355f67045a6acffaae723d9186f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet edac2355f67045a6acffaae723d9186f already in progress: creating tablet
W20250903 20:05:31.134393 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet cf99a74acfdb4a17bbbce3a07467e9c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf99a74acfdb4a17bbbce3a07467e9c1 already in progress: creating tablet
W20250903 20:05:31.134585 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 112c0dccb60c4c7c9efc164d9c8443ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 112c0dccb60c4c7c9efc164d9c8443ea already in progress: creating tablet
W20250903 20:05:31.134900 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 30cc0d4ba8d444f1b98232f906ea0c7b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30cc0d4ba8d444f1b98232f906ea0c7b already in progress: creating tablet
I20250903 20:05:31.133074 27684 pending_rounds.cc:77] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f: Aborting op as it isn't in flight: id { term: 2 index: 1 } timestamp: 7196384997704769536 op_type: NO_OP noop_request { }
I20250903 20:05:31.135140 27684 raft_consensus.cc:2887] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:31.135255 27684 raft_consensus.cc:2270] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:31.136297 27684 ts_tablet_manager.cc:1905] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:31.139906 27063 master_service.cc:948] ReplaceTablet: received request to replace tablet bc98762790f441038972366b1cc2c55b from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:31.140476 27063 master_service.cc:948] ReplaceTablet: received request to replace tablet 17514a2af39b402abbe8263d7c1abc7f from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:31.140914 28596 raft_consensus.cc:491] T 22bc87d1a4804b6c80f77f9692500d63 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:31.141085 28596 raft_consensus.cc:513] T 22bc87d1a4804b6c80f77f9692500d63 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.142554 28596 leader_election.cc:290] T 22bc87d1a4804b6c80f77f9692500d63 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:31.143272 28751 consensus_queue.cc:1035] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:31.146243 27684 ts_tablet_manager.cc:1918] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20250903 20:05:31.146417 27684 log.cc:1199] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/514e382479014fa7bf1f404efff11c86
I20250903 20:05:31.146850 27684 ts_tablet_manager.cc:1939] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:31.147460 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 864) took 3640 ms. Trace:
I20250903 20:05:31.147555 27684 rpcz_store.cc:276] 0903 20:05:27.507441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.507499 (+ 58us) service_pool.cc:225] Handling call
0903 20:05:31.132538 (+3625039us) ts_tablet_manager.cc:1145] Deleting tablet 514e382479014fa7bf1f404efff11c86
0903 20:05:31.132546 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.139648 (+ 7102us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.146062 (+ 6414us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.147433 (+ 1371us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3624833}
I20250903 20:05:31.154578 27687 ts_tablet_manager.cc:1939] T 0a9b7770432942da9373d5100c71bd6a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:31.157236 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 925) took 2182 ms. Trace:
I20250903 20:05:31.157361 28751 consensus_queue.cc:1035] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250903 20:05:31.157716 27320 leader_election.cc:343] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:31.162441 27320 leader_election.cc:343] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:31.162387 28916 consensus_queue.cc:1035] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:31.162997 28596 raft_consensus.cc:491] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51578fb7bceb4e119b3ab1e58c4f5528)
I20250903 20:05:31.163193 28596 raft_consensus.cc:513] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:31.163852 28596 leader_election.cc:290] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:31.159668 28965 raft_consensus.cc:491] T 9e98cd1d9bd246e182e2969d147e68bf P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:31.164237 28965 raft_consensus.cc:513] T 9e98cd1d9bd246e182e2969d147e68bf P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:31.165335 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e98cd1d9bd246e182e2969d147e68bf" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:31.165550 27249 raft_consensus.cc:2391] T 9e98cd1d9bd246e182e2969d147e68bf P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 51578fb7bceb4e119b3ab1e58c4f5528 in current term 1: Already voted for candidate e72ef1dbcc5143cf9f04a808f101c80f in this term.
I20250903 20:05:31.166343 28965 leader_election.cc:290] T 9e98cd1d9bd246e182e2969d147e68bf P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:31.161104 27687 rpcz_store.cc:276] 0903 20:05:28.974639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.974703 (+ 64us) service_pool.cc:225] Handling call
0903 20:05:31.094552 (+2119849us) ts_tablet_manager.cc:1145] Deleting tablet 0a9b7770432942da9373d5100c71bd6a
0903 20:05:31.094571 (+ 19us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.104388 (+ 9817us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.129628 (+ 25240us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.157211 (+ 27583us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2119594}
I20250903 20:05:31.168056 27059 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:45936 (request call id 1147) took 1138 ms. Trace:
I20250903 20:05:31.168567 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 926) took 2173 ms. Trace:
I20250903 20:05:31.168675 27687 rpcz_store.cc:276] 0903 20:05:28.994723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.994770 (+ 47us) service_pool.cc:225] Handling call
0903 20:05:31.168383 (+2173613us) ts_tablet_manager.cc:1145] Deleting tablet 86e33bd235c14042a578a4a1c70f11b7
0903 20:05:31.168402 (+ 19us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.168542 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2173412}
I20250903 20:05:31.169973 27684 tablet_replica.cc:331] T f2cc35f4244a42eeb4a65de749824c11 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:31.170184 27684 raft_consensus.cc:2241] T f2cc35f4244a42eeb4a65de749824c11 P e72ef1dbcc5143cf9f04a808f101c80f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:31.170392 27684 raft_consensus.cc:2270] T f2cc35f4244a42eeb4a65de749824c11 P e72ef1dbcc5143cf9f04a808f101c80f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:31.170881 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 0a9b7770432942da9373d5100c71bd6a (table default.Table_6656b728e12a482fa9e7893ef4842ac7 [id=6788b524a8654e34a4d9ee8cc8fc84d3]) successfully deleted
I20250903 20:05:31.170928 27684 ts_tablet_manager.cc:1905] T f2cc35f4244a42eeb4a65de749824c11 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:31.171178 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 86e33bd235c14042a578a4a1c70f11b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86e33bd235c14042a578a4a1c70f11b7 already in progress: creating tablet
I20250903 20:05:31.172792 27687 tablet_replica.cc:331] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:31.173712 27687 raft_consensus.cc:2241] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:31.174921 27687 raft_consensus.cc:2270] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:31.181136 27687 ts_tablet_manager.cc:1905] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:31.191231 28870 raft_consensus.cc:491] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:31.191417 28870 raft_consensus.cc:513] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.192075 28870 leader_election.cc:290] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:31.192909 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bf69315f07a470bb2b897ad7c590729" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:31.193118 27511 raft_consensus.cc:2466] T 1bf69315f07a470bb2b897ad7c590729 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:31.193651 27183 leader_election.cc:304] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:31.193688 28751 consensus_queue.cc:1035] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:31.194015 28870 raft_consensus.cc:2802] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:31.194098 28870 raft_consensus.cc:491] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:31.194156 28870 raft_consensus.cc:3058] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.196893 28870 raft_consensus.cc:513] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.198518 28751 raft_consensus.cc:491] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:31.198527 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bf69315f07a470bb2b897ad7c590729" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:31.198729 27511 raft_consensus.cc:3058] T 1bf69315f07a470bb2b897ad7c590729 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.202961 27511 raft_consensus.cc:2466] T 1bf69315f07a470bb2b897ad7c590729 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:31.203747 27183 leader_election.cc:304] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:31.204181 28947 raft_consensus.cc:2802] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:31.204342 28947 raft_consensus.cc:695] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:31.204649 28947 consensus_queue.cc:237] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.206727 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bf69315f07a470bb2b897ad7c590729" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:31.207062 27385 raft_consensus.cc:2466] T 1bf69315f07a470bb2b897ad7c590729 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:31.207423 28870 leader_election.cc:290] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:31.211300 28770 raft_consensus.cc:491] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:31.211503 28770 raft_consensus.cc:513] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.198835 28751 raft_consensus.cc:513] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.212256 28770 leader_election.cc:290] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:31.212488 28751 leader_election.cc:290] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:31.213572 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "541278ca9ffb4e41b341139c2faa337f" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:31.214016 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22bc87d1a4804b6c80f77f9692500d63" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:31.215054 27248 raft_consensus.cc:2466] T 22bc87d1a4804b6c80f77f9692500d63 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
W20250903 20:05:31.216701 27321 leader_election.cc:343] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:31.216912 27321 leader_election.cc:304] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:31.217698 28751 raft_consensus.cc:2802] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:31.217782 28751 raft_consensus.cc:491] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:31.217839 28751 raft_consensus.cc:3058] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.221487 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bf69315f07a470bb2b897ad7c590729" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:31.222405 27385 raft_consensus.cc:3058] T 1bf69315f07a470bb2b897ad7c590729 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.225171 28751 raft_consensus.cc:513] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.229045 27684 ts_tablet_manager.cc:1918] T f2cc35f4244a42eeb4a65de749824c11 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:31.229175 27684 log.cc:1199] T f2cc35f4244a42eeb4a65de749824c11 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/f2cc35f4244a42eeb4a65de749824c11
I20250903 20:05:31.229455 27684 ts_tablet_manager.cc:1939] T f2cc35f4244a42eeb4a65de749824c11 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:31.229737 28848 tablet.cc:1722] T ac1b0dc95fca49b4af6697c0a70e9730 P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:31.228749 28885 tablet.cc:1722] T ac1b0dc95fca49b4af6697c0a70e9730 P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:31.232157 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22bc87d1a4804b6c80f77f9692500d63" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:31.232481 27248 raft_consensus.cc:3058] T 22bc87d1a4804b6c80f77f9692500d63 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.235091 27248 raft_consensus.cc:2466] T 22bc87d1a4804b6c80f77f9692500d63 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:31.235399 28751 leader_election.cc:290] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:31.235715 27321 leader_election.cc:304] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:31.235987 27385 raft_consensus.cc:2466] T 1bf69315f07a470bb2b897ad7c590729 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:31.236292 28770 raft_consensus.cc:2802] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:31.236445 28770 raft_consensus.cc:695] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:31.236688 28770 consensus_queue.cc:237] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:31.239595 27462 leader_election.cc:343] T 514e382479014fa7bf1f404efff11c86 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 514e382479014fa7bf1f404efff11c86
I20250903 20:05:31.240815 27376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22bc87d1a4804b6c80f77f9692500d63" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:31.241081 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22bc87d1a4804b6c80f77f9692500d63" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:31.241554 27248 raft_consensus.cc:2391] T 22bc87d1a4804b6c80f77f9692500d63 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
W20250903 20:05:31.242399 27184 consensus_peers.cc:489] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4f3230f50ec4c95b528fead8c191224. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:31.247385 27692 tablet_bootstrap.cc:492] T 541278ca9ffb4e41b341139c2faa337f P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:31.247551 27693 tablet_bootstrap.cc:492] T a0f79831866b40c49c5b27ce2014c3f6 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:31.247701 27692 ts_tablet_manager.cc:1397] T 541278ca9ffb4e41b341139c2faa337f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.262s user 0.029s sys 0.025s
I20250903 20:05:31.247785 27693 ts_tablet_manager.cc:1397] T a0f79831866b40c49c5b27ce2014c3f6 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.259s user 0.017s sys 0.013s
I20250903 20:05:31.248605 27693 raft_consensus.cc:357] T a0f79831866b40c49c5b27ce2014c3f6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.248870 27693 raft_consensus.cc:383] T a0f79831866b40c49c5b27ce2014c3f6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:31.248762 27692 raft_consensus.cc:357] T 541278ca9ffb4e41b341139c2faa337f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.248942 27692 raft_consensus.cc:383] T 541278ca9ffb4e41b341139c2faa337f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:31.248989 27693 raft_consensus.cc:738] T a0f79831866b40c49c5b27ce2014c3f6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:31.249060 27692 raft_consensus.cc:738] T 541278ca9ffb4e41b341139c2faa337f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:31.249322 27693 consensus_queue.cc:260] T a0f79831866b40c49c5b27ce2014c3f6 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.249322 27692 consensus_queue.cc:260] T 541278ca9ffb4e41b341139c2faa337f P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.249851 27687 ts_tablet_manager.cc:1918] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:31.249941 27692 ts_tablet_manager.cc:1428] T 541278ca9ffb4e41b341139c2faa337f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:31.250388 27692 tablet_bootstrap.cc:492] T 567630e2a93645e58f9bf05ffff87b32 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:31.251752 27687 log.cc:1199] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/f9709dfd18654c5bb071e39da7c8889f
I20250903 20:05:31.252512 27692 tablet_bootstrap.cc:654] T 567630e2a93645e58f9bf05ffff87b32 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:31.252980 27687 ts_tablet_manager.cc:1939] T f9709dfd18654c5bb071e39da7c8889f P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:31.253221 27695 tablet_bootstrap.cc:492] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:31.254060 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 927) took 2236 ms. Trace:
I20250903 20:05:31.253471 28870 consensus_queue.cc:1035] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:31.254932 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 872) took 3637 ms. Trace:
I20250903 20:05:31.255158 27695 ts_tablet_manager.cc:1397] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.192s user 0.007s sys 0.019s
I20250903 20:05:31.256911 27684 rpcz_store.cc:276] 0903 20:05:27.617210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.617264 (+ 54us) service_pool.cc:225] Handling call
0903 20:05:31.169911 (+3552647us) ts_tablet_manager.cc:1145] Deleting tablet f2cc35f4244a42eeb4a65de749824c11
0903 20:05:31.169924 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.202897 (+ 32973us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.228927 (+ 26030us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.254908 (+ 25981us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3552202}
I20250903 20:05:31.258579 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet f2cc35f4244a42eeb4a65de749824c11 (table default.Table_78e7a5b76ebc469396d8a3d670ab63ee [id=fb43441ca8f2415d99a628e81ef1642f]) successfully deleted
I20250903 20:05:31.258716 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 875) took 3598 ms. Trace:
I20250903 20:05:31.258795 27684 rpcz_store.cc:276] 0903 20:05:27.660338 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.660416 (+ 78us) service_pool.cc:225] Handling call
0903 20:05:31.258569 (+3598153us) ts_tablet_manager.cc:1145] Deleting tablet 733f8024e1fa401ca9614a97db91a19b
0903 20:05:31.258582 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.258700 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3597939}
W20250903 20:05:31.259210 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 733f8024e1fa401ca9614a97db91a19b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 733f8024e1fa401ca9614a97db91a19b already in progress: creating tablet
I20250903 20:05:31.259529 27687 rpcz_store.cc:276] 0903 20:05:29.017415 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:29.017474 (+ 59us) service_pool.cc:225] Handling call
0903 20:05:31.172740 (+2155266us) ts_tablet_manager.cc:1145] Deleting tablet f9709dfd18654c5bb071e39da7c8889f
0903 20:05:31.172751 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.187065 (+ 14314us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.249734 (+ 62669us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.254039 (+ 4305us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2155074}
I20250903 20:05:31.260002 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 878) took 3353 ms. Trace:
I20250903 20:05:31.260155 27695 raft_consensus.cc:357] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.249882 27693 ts_tablet_manager.cc:1428] T a0f79831866b40c49c5b27ce2014c3f6 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:31.260637 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet f9709dfd18654c5bb071e39da7c8889f (table default.Table_876161e56d1d4c4c9d6726471530d965 [id=39cbb7f0d8e145c6b5f283133017b80f]) successfully deleted
I20250903 20:05:31.260811 27693 tablet_bootstrap.cc:492] T 67ca1dcf6fc7435db555392235f9900f P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:31.263656 27693 tablet_bootstrap.cc:654] T 67ca1dcf6fc7435db555392235f9900f P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:31.264935 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 928) took 2228 ms. Trace:
I20250903 20:05:31.265255 27695 raft_consensus.cc:383] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:31.265455 27684 rpcz_store.cc:276] 0903 20:05:27.906794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.907104 (+ 310us) service_pool.cc:225] Handling call
0903 20:05:31.259891 (+3352787us) ts_tablet_manager.cc:1145] Deleting tablet 475d05d087c34159b093b92cc308bcaf
0903 20:05:31.259902 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.259986 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3352571}
I20250903 20:05:31.265900 27695 raft_consensus.cc:738] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:31.266644 27687 rpcz_store.cc:276] 0903 20:05:29.036589 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:29.040466 (+ 3877us) service_pool.cc:225] Handling call
0903 20:05:31.264803 (+2224337us) ts_tablet_manager.cc:1145] Deleting tablet 9766cea9d5bb407a9900c7e3bb4d2641
0903 20:05:31.264811 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.264915 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2224144}
W20250903 20:05:31.267678 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 475d05d087c34159b093b92cc308bcaf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 475d05d087c34159b093b92cc308bcaf already in progress: creating tablet
I20250903 20:05:31.268270 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 879) took 3358 ms. Trace:
W20250903 20:05:31.268707 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 9766cea9d5bb407a9900c7e3bb4d2641 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9766cea9d5bb407a9900c7e3bb4d2641 already in progress: creating tablet
I20250903 20:05:31.269001 27695 consensus_queue.cc:260] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.269246 27693 tablet_bootstrap.cc:492] T 67ca1dcf6fc7435db555392235f9900f P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:31.269419 27693 ts_tablet_manager.cc:1397] T 67ca1dcf6fc7435db555392235f9900f P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20250903 20:05:31.269588 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 929) took 2224 ms. Trace:
I20250903 20:05:31.269740 27684 rpcz_store.cc:276] 0903 20:05:27.909954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.910471 (+ 517us) service_pool.cc:225] Handling call
0903 20:05:31.268122 (+3357651us) ts_tablet_manager.cc:1145] Deleting tablet 5fe1de01bf7a4d418e765ea3e8cbecc1
0903 20:05:31.268133 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.268248 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3357445}
W20250903 20:05:31.269716 27183 consensus_peers.cc:489] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0293a69f1c246d08d414e95729b4ab4. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:31.270609 27695 ts_tablet_manager.cc:1428] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250903 20:05:31.270730 27687 rpcz_store.cc:276] 0903 20:05:29.044880 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:29.044970 (+ 90us) service_pool.cc:225] Handling call
0903 20:05:31.269407 (+2224437us) ts_tablet_manager.cc:1145] Deleting tablet bf459db3327640f88574fca7b5f2ca98
0903 20:05:31.269418 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.269570 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2224197}
W20250903 20:05:31.271275 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 5fe1de01bf7a4d418e765ea3e8cbecc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5fe1de01bf7a4d418e765ea3e8cbecc1 already in progress: creating tablet
I20250903 20:05:31.271744 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 880) took 3359 ms. Trace:
W20250903 20:05:31.272172 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet bf459db3327640f88574fca7b5f2ca98 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf459db3327640f88574fca7b5f2ca98 already in progress: creating tablet
W20250903 20:05:31.272573 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4aea507c5e214a57a80bd4de1a6ccd67 consensus state has no leader
I20250903 20:05:31.272651 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 937) took 1971 ms. Trace:
I20250903 20:05:31.272754 27064 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:45936 (request call id 1148) took 1237 ms. Trace:
I20250903 20:05:31.272730 27687 rpcz_store.cc:276] 0903 20:05:29.301373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:29.301438 (+ 65us) service_pool.cc:225] Handling call
0903 20:05:31.272526 (+1971088us) ts_tablet_manager.cc:1145] Deleting tablet 53b00774b745462091321ea47fcc2114
0903 20:05:31.272537 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.272632 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1970881}
W20250903 20:05:31.273226 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 53b00774b745462091321ea47fcc2114 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53b00774b745462091321ea47fcc2114 already in progress: creating tablet
I20250903 20:05:31.272833 27064 rpcz_store.cc:276] 0903 20:05:30.035700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.751420 (+715720us) service_pool.cc:225] Handling call
0903 20:05:30.769015 (+ 17595us) spinlock_profiling.cc:243] Waited 17.3 ms on lock 0x616000040f54. stack: 00007f4b9d6c2dc6 00007f4b9d6c2b80 00007f4b9c6aa3ad 0000000000972d94 0000000000972d5d 00007f4bac49014d 00007f4bac490243 00007f4ba914f941 00007f4bb0cdaa7d 00007f4bb0e66146 00007f4ba7c9f126 00007f4ba7c9ee55 00007f4ba44730be 00007f4ba4471f9e 00007f4ba44781d5 00007f4ba4478aff
0903 20:05:30.769317 (+ 302us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0903 20:05:30.769582 (+ 265us) catalog_manager.cc:3683] Apply alter schema
0903 20:05:30.769786 (+ 204us) catalog_manager.cc:3715] Acquired catalog manager lock
0903 20:05:30.769867 (+ 81us) catalog_manager.cc:3873] Updating metadata on disk
0903 20:05:31.272332 (+502465us) catalog_manager.cc:3904] Committing alterations to in-memory state
0903 20:05:31.272734 (+ 402us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:31.237696 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:31.237822 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:31.237827 (+ 5us) write_op.cc:435] Acquired schema lock
0903 20:05:31.237829 (+ 2us) tablet.cc:660] Decoding operations
0903 20:05:31.237941 (+ 112us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:31.237954 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:31.237959 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0903 20:05:31.238028 (+ 69us) tablet.cc:699] Row locks acquired
0903 20:05:31.238031 (+ 3us) write_op.cc:260] PREPARE: finished
0903 20:05:31.238065 (+ 34us) write_op.cc:270] Start()
0903 20:05:31.238098 (+ 33us) write_op.cc:276] Timestamp: P: 1756929931238045 usec, L: 0
0903 20:05:31.238102 (+ 4us) op_driver.cc:348] REPLICATION: starting
0903 20:05:31.262701 (+ 24599us) log.cc:844] Serialized 405 byte log entry
0903 20:05:31.270902 (+ 8201us) op_driver.cc:464] REPLICATION: finished
0903 20:05:31.271249 (+ 347us) write_op.cc:301] APPLY: starting
0903 20:05:31.271272 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:31.271321 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:31.271353 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:31.271425 (+ 72us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:31.271465 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0903 20:05:31.271474 (+ 9us) write_op.cc:312] APPLY: finished
0903 20:05:31.271593 (+ 119us) log.cc:844] Serialized 23 byte log entry
0903 20:05:31.271664 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:31.271686 (+ 22us) write_op.cc:454] Released schema lock
0903 20:05:31.271694 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0903 20:05:31.271778 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:31.271783 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":39838976,"child_traces":[["op",{"apply.queue_time_us":266,"apply.run_cpu_time_us":535,"apply.run_wall_time_us":535,"num_ops":1,"prepare.queue_time_us":467232,"prepare.run_cpu_time_us":808,"prepare.run_wall_time_us":25128,"raft.queue_time_us":974,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":250,"replication_time_us":32761}]]}
I20250903 20:05:31.273861 27684 rpcz_store.cc:276] 0903 20:05:27.912086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.912146 (+ 60us) service_pool.cc:225] Handling call
0903 20:05:31.271614 (+3359468us) ts_tablet_manager.cc:1145] Deleting tablet 3e930dbaac0e40ee8f175ccf87592d31
0903 20:05:31.271627 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.271726 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3359271}
I20250903 20:05:31.274361 27062 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_24e55a7c820c413f8514bd3ad7bd3269 [id=0904dc5d0c594bc3bbf32de2508c6195] alter complete (version 3)
I20250903 20:05:31.274487 27684 tablet_replica.cc:331] T 33fb6b357bda43e7abeaed5634183ef6 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:31.274791 27684 raft_consensus.cc:2241] T 33fb6b357bda43e7abeaed5634183ef6 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:31.275036 27684 raft_consensus.cc:2270] T 33fb6b357bda43e7abeaed5634183ef6 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:31.275784 27695 tablet_bootstrap.cc:492] T 541278ca9ffb4e41b341139c2faa337f P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
W20250903 20:05:31.276530 27062 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.26.65.66:53547 (request call id 129) took 1131 ms (client timeout 1000 ms). Trace:
W20250903 20:05:31.276645 27062 rpcz_store.cc:269] 0903 20:05:30.144691 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.235461 (+ 90770us) service_pool.cc:225] Handling call
0903 20:05:30.503529 (+268068us) spinlock_profiling.cc:243] Waited 267 ms on lock 0x616000040f54. stack: 00007f4b9d6c2dc6 00007f4b9d6c2b80 00007f4b9c6aa3ad 0000000000972d94 0000000000972d5d 00007f4bac49014d 00007f4bac490243 00007f4ba914f941 00007f4bb0cdaa7d 00007f4bb0e5e6f8 00007f4ba7c98c86 00007f4ba7c989b5 00007f4ba44730be 00007f4ba4471f9e 00007f4ba44781d5 00007f4ba4478aff
0903 20:05:31.276501 (+772972us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:30.770620 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:30.770742 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:30.770749 (+ 7us) write_op.cc:435] Acquired schema lock
0903 20:05:30.770751 (+ 2us) tablet.cc:660] Decoding operations
0903 20:05:30.770868 (+ 117us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:30.770884 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:30.770889 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0903 20:05:30.770956 (+ 67us) tablet.cc:699] Row locks acquired
0903 20:05:30.770958 (+ 2us) write_op.cc:260] PREPARE: finished
0903 20:05:30.771002 (+ 44us) write_op.cc:270] Start()
0903 20:05:30.771038 (+ 36us) write_op.cc:276] Timestamp: P: 1756929930770972 usec, L: 0
0903 20:05:30.771042 (+ 4us) op_driver.cc:348] REPLICATION: starting
0903 20:05:30.773889 (+ 2847us) log.cc:844] Serialized 530 byte log entry
0903 20:05:30.777852 (+ 3963us) op_driver.cc:464] REPLICATION: finished
0903 20:05:30.778097 (+ 245us) write_op.cc:301] APPLY: starting
0903 20:05:30.778122 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:30.778183 (+ 61us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:30.778248 (+ 65us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:30.778316 (+ 68us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:30.778353 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0903 20:05:30.778361 (+ 8us) write_op.cc:312] APPLY: finished
0903 20:05:30.778447 (+ 86us) log.cc:844] Serialized 23 byte log entry
0903 20:05:30.778507 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:30.778531 (+ 24us) write_op.cc:454] Released schema lock
0903 20:05:30.778539 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0903 20:05:30.826347 (+ 47808us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:30.826357 (+ 10us) write_op.cc:454] Released schema lock
Related trace 'op':
0903 20:05:31.262873 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:31.263023 (+ 150us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:31.263032 (+ 9us) write_op.cc:435] Acquired schema lock
0903 20:05:31.263036 (+ 4us) tablet.cc:660] Decoding operations
0903 20:05:31.263177 (+ 141us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:31.263189 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:31.263194 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0903 20:05:31.263270 (+ 76us) tablet.cc:699] Row locks acquired
0903 20:05:31.263273 (+ 3us) write_op.cc:260] PREPARE: finished
0903 20:05:31.263312 (+ 39us) write_op.cc:270] Start()
0903 20:05:31.263352 (+ 40us) write_op.cc:276] Timestamp: P: 1756929931263286 usec, L: 0
0903 20:05:31.263355 (+ 3us) op_driver.cc:348] REPLICATION: starting
0903 20:05:31.263544 (+ 189us) log.cc:844] Serialized 356 byte log entry
0903 20:05:31.270987 (+ 7443us) op_driver.cc:464] REPLICATION: finished
0903 20:05:31.271882 (+ 895us) write_op.cc:301] APPLY: starting
0903 20:05:31.271894 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:31.271929 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:31.271968 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:31.272016 (+ 48us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:31.272045 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0903 20:05:31.272053 (+ 8us) write_op.cc:312] APPLY: finished
0903 20:05:31.272115 (+ 62us) log.cc:844] Serialized 23 byte log entry
0903 20:05:31.272162 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:31.272177 (+ 15us) write_op.cc:454] Released schema lock
0903 20:05:31.272183 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":15,"spinlock_wait_cycles":614346112,"child_traces":[["op",{"apply.queue_time_us":180,"apply.run_cpu_time_us":499,"apply.run_wall_time_us":507,"num_ops":1,"prepare.queue_time_us":261778,"prepare.run_cpu_time_us":872,"prepare.run_wall_time_us":55688,"raft.queue_time_us":164,"raft.run_cpu_time_us":873,"raft.run_wall_time_us":7152,"replication_time_us":6766}],["op",{"apply.queue_time_us":855,"apply.run_cpu_time_us":378,"apply.run_wall_time_us":377,"num_ops":1,"prepare.queue_time_us":482122,"prepare.run_cpu_time_us":765,"prepare.run_wall_time_us":763,"raft.queue_time_us":449,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":3,"replication_time_us":7612,"threads_started":1}]]}
I20250903 20:05:31.278025 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 938) took 1961 ms. Trace:
I20250903 20:05:31.278117 27687 rpcz_store.cc:276] 0903 20:05:29.316304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:29.316362 (+ 58us) service_pool.cc:225] Handling call
0903 20:05:31.277874 (+1961512us) ts_tablet_manager.cc:1145] Deleting tablet 31186292b25f4c3cb23597f66dcbd282
0903 20:05:31.277886 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.278000 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1961350}
I20250903 20:05:31.270361 27693 raft_consensus.cc:357] T 67ca1dcf6fc7435db555392235f9900f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:31.279058 27693 raft_consensus.cc:383] T 67ca1dcf6fc7435db555392235f9900f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250903 20:05:31.280385 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 31186292b25f4c3cb23597f66dcbd282 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31186292b25f4c3cb23597f66dcbd282 already in progress: creating tablet
I20250903 20:05:31.282529 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 939) took 1629 ms. Trace:
I20250903 20:05:31.282617 27687 rpcz_store.cc:276] 0903 20:05:29.652875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:29.653423 (+ 548us) service_pool.cc:225] Handling call
0903 20:05:31.282374 (+1628951us) ts_tablet_manager.cc:1145] Deleting tablet cc1bc205bf39447bb9c3c7863a0959cf
0903 20:05:31.282387 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.282507 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1628742}
I20250903 20:05:31.282955 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 941) took 1484 ms. Trace:
I20250903 20:05:31.283031 27687 rpcz_store.cc:276] 0903 20:05:29.798084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:29.798133 (+ 49us) service_pool.cc:225] Handling call
0903 20:05:31.282835 (+1484702us) ts_tablet_manager.cc:1145] Deleting tablet d1c96296a579440c887ba6c2a93f7204
0903 20:05:31.282843 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.282940 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1484459}
I20250903 20:05:31.283319 27352 tablet_service.cc:1515] Processing DeleteTablet for tablet f9709dfd18654c5bb071e39da7c8889f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:17 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:31.283490 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet f2629f46704e4a0fa7922533928ca2ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:31.283650 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 4af8c03c9d834405b4e25d35441db961 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:31.283776 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 726f4d920f3e47fea41b4e12c021c988 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:31.283802 27685 ts_tablet_manager.cc:1918] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:31.283924 27685 log.cc:1199] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/514e382479014fa7bf1f404efff11c86
I20250903 20:05:31.284075 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:31.284257 27685 ts_tablet_manager.cc:1939] T 514e382479014fa7bf1f404efff11c86 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:31.284250 27350 tablet_service.cc:1515] Processing DeleteTablet for tablet 512908bc4d6941ec81b145ee5795405f with delete_type TABLET_DATA_DELETED (replaced by tablet 1598824a733a4a93bed043e81b86abf3) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:31.284360 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet b7a6d5d8a97a4769a81ec7224d591f8c with delete_type TABLET_DATA_DELETED (replaced by tablet 79c0cc0c30ab42eab0be415e3db9589f) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:31.284413 27348 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f27f525fdab407280aa23df1ae71b43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:31.284541 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e2f98f760cf4df4bc2a10530e2157c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:31.284579 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 771) took 5163 ms. Trace:
I20250903 20:05:31.284689 27352 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bf69315f07a470bb2b897ad7c590729 with delete_type TABLET_DATA_DELETED (replaced by tablet 17514a2af39b402abbe8263d7c1abc7f) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:31.284655 27685 rpcz_store.cc:276] 0903 20:05:26.120688 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.120747 (+ 59us) service_pool.cc:225] Handling call
0903 20:05:31.123785 (+5003038us) ts_tablet_manager.cc:1145] Deleting tablet 514e382479014fa7bf1f404efff11c86
0903 20:05:31.123793 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.280358 (+156565us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.283660 (+ 3302us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.284561 (+ 901us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5002854}
I20250903 20:05:31.284817 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 59a1dd48d76241d7ba817070fbffd853 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:31.285053 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 777) took 5095 ms. Trace:
W20250903 20:05:31.285022 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 3e930dbaac0e40ee8f175ccf87592d31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e930dbaac0e40ee8f175ccf87592d31 already in progress: creating tablet
I20250903 20:05:31.285122 27685 rpcz_store.cc:276] 0903 20:05:26.189224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.189286 (+ 62us) service_pool.cc:225] Handling call
0903 20:05:31.284927 (+5095641us) ts_tablet_manager.cc:1145] Deleting tablet d93258c032524b71997389b81846a5be
0903 20:05:31.284936 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.285037 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5095440}
I20250903 20:05:31.285372 27685 tablet_replica.cc:331] T a4c582b441f6467c99ba0b090abcf4cc P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:31.285560 27685 raft_consensus.cc:2241] T a4c582b441f6467c99ba0b090abcf4cc P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:31.286820 27684 ts_tablet_manager.cc:1905] T 33fb6b357bda43e7abeaed5634183ef6 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:31.287063 28761 raft_consensus.cc:491] T 67ca1dcf6fc7435db555392235f9900f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:31.287194 28761 raft_consensus.cc:513] T 67ca1dcf6fc7435db555392235f9900f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:31.287748 28761 leader_election.cc:290] T 67ca1dcf6fc7435db555392235f9900f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:31.288007 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 944) took 1202 ms. Trace:
I20250903 20:05:31.288085 27687 rpcz_store.cc:276] 0903 20:05:30.085823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.086530 (+ 707us) service_pool.cc:225] Handling call
0903 20:05:31.287896 (+1201366us) ts_tablet_manager.cc:1145] Deleting tablet 1bad3c509460422396a17a3c4ae3d1e8
0903 20:05:31.287910 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.287987 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1196458}
I20250903 20:05:31.287010 27517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e98cd1d9bd246e182e2969d147e68bf" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:31.290860 27687 tablet_replica.cc:331] T 607c9a9cbc5540e5bd0fa436fcba564c P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:31.291366 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "961ade36f7d14eba91bb16d5eae2e254" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:31.291623 27249 raft_consensus.cc:2466] T 961ade36f7d14eba91bb16d5eae2e254 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:31.292447 27695 tablet_bootstrap.cc:654] T 541278ca9ffb4e41b341139c2faa337f P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:31.292871 27687 raft_consensus.cc:2241] T 607c9a9cbc5540e5bd0fa436fcba564c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:31.293272 27687 raft_consensus.cc:2270] T 607c9a9cbc5540e5bd0fa436fcba564c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:31.295230 27687 ts_tablet_manager.cc:1905] T 607c9a9cbc5540e5bd0fa436fcba564c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:31.297461 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet d93258c032524b71997389b81846a5be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d93258c032524b71997389b81846a5be already in progress: creating tablet
W20250903 20:05:31.297710 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet cc1bc205bf39447bb9c3c7863a0959cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc1bc205bf39447bb9c3c7863a0959cf already in progress: creating tablet
W20250903 20:05:31.297855 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet d1c96296a579440c887ba6c2a93f7204 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1c96296a579440c887ba6c2a93f7204 already in progress: creating tablet
W20250903 20:05:31.298000 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 1bad3c509460422396a17a3c4ae3d1e8 because the tablet was not found. No further retry: Not found: Tablet not found: 1bad3c509460422396a17a3c4ae3d1e8
I20250903 20:05:31.299917 27517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "541278ca9ffb4e41b341139c2faa337f" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:31.300204 27517 raft_consensus.cc:2466] T 541278ca9ffb4e41b341139c2faa337f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:31.300244 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22bc87d1a4804b6c80f77f9692500d63" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:31.300379 27516 raft_consensus.cc:2466] T 22bc87d1a4804b6c80f77f9692500d63 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:31.311640 27517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22bc87d1a4804b6c80f77f9692500d63" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:31.311944 27517 raft_consensus.cc:3058] T 22bc87d1a4804b6c80f77f9692500d63 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.313534 27462 leader_election.cc:304] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:31.314728 28596 raft_consensus.cc:2802] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250903 20:05:31.314808 28596 raft_consensus.cc:491] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 51578fb7bceb4e119b3ab1e58c4f5528)
I20250903 20:05:31.314873 28596 raft_consensus.cc:3058] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Advancing to term 2
I20250903 20:05:31.317476 27517 raft_consensus.cc:2466] T 22bc87d1a4804b6c80f77f9692500d63 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:31.317577 28596 raft_consensus.cc:513] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:31.318655 28596 leader_election.cc:290] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 2 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:31.319788 28870 consensus_queue.cc:1035] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:31.325973 27684 ts_tablet_manager.cc:1918] T 33fb6b357bda43e7abeaed5634183ef6 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:31.326112 27684 log.cc:1199] T 33fb6b357bda43e7abeaed5634183ef6 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/33fb6b357bda43e7abeaed5634183ef6
I20250903 20:05:31.328778 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "67ca1dcf6fc7435db555392235f9900f" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:31.328990 27249 raft_consensus.cc:2466] T 67ca1dcf6fc7435db555392235f9900f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:31.329381 27684 ts_tablet_manager.cc:1939] T 33fb6b357bda43e7abeaed5634183ef6 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:31.329991 27376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "67ca1dcf6fc7435db555392235f9900f" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:31.330171 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 881) took 3333 ms. Trace:
I20250903 20:05:31.330307 27684 rpcz_store.cc:276] 0903 20:05:27.997057 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.997121 (+ 64us) service_pool.cc:225] Handling call
0903 20:05:31.274410 (+3277289us) ts_tablet_manager.cc:1145] Deleting tablet 33fb6b357bda43e7abeaed5634183ef6
0903 20:05:31.274427 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.303873 (+ 29446us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.325850 (+ 21977us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.330149 (+ 4299us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3277056}
I20250903 20:05:31.330766 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 33fb6b357bda43e7abeaed5634183ef6 (table default.Table_7a6b693ddf594ff6821359a4a7fbee01 [id=fb19a730ea6343fbba98571261d7a894]) successfully deleted
I20250903 20:05:31.331668 27320 leader_election.cc:304] T 9e98cd1d9bd246e182e2969d147e68bf P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:31.332432 28770 raft_consensus.cc:3058] T 9e98cd1d9bd246e182e2969d147e68bf P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.333091 27684 tablet_replica.cc:331] T 6f8cfe53b91b41d7ba7f9074e1005af3 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:31.333787 28870 raft_consensus.cc:491] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:31.333976 28870 raft_consensus.cc:513] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.334678 28870 leader_election.cc:290] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:31.335289 27384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512908bc4d6941ec81b145ee5795405f" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:31.335477 27384 raft_consensus.cc:2466] T 512908bc4d6941ec81b145ee5795405f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:31.335511 28770 raft_consensus.cc:2747] T 9e98cd1d9bd246e182e2969d147e68bf P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:31.336200 27320 leader_election.cc:304] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:31.336798 28770 raft_consensus.cc:2802] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:31.336932 28770 raft_consensus.cc:491] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:31.337056 28770 raft_consensus.cc:3058] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.339960 27184 leader_election.cc:304] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:31.340335 28770 raft_consensus.cc:513] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.340737 28870 raft_consensus.cc:2802] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:31.340816 28870 raft_consensus.cc:491] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:31.340871 28870 raft_consensus.cc:3058] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.341980 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "541278ca9ffb4e41b341139c2faa337f" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
W20250903 20:05:31.342593 27321 leader_election.cc:343] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:31.342779 28770 leader_election.cc:290] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:31.343839 28870 raft_consensus.cc:513] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.344852 27384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512908bc4d6941ec81b145ee5795405f" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:31.345057 27384 raft_consensus.cc:3058] T 512908bc4d6941ec81b145ee5795405f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.347713 27384 raft_consensus.cc:2466] T 512908bc4d6941ec81b145ee5795405f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:31.348254 27184 leader_election.cc:304] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:31.348349 28710 raft_consensus.cc:491] T 512908bc4d6941ec81b145ee5795405f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:31.348495 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "961ade36f7d14eba91bb16d5eae2e254" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:31.348487 28710 raft_consensus.cc:513] T 512908bc4d6941ec81b145ee5795405f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.348649 27249 raft_consensus.cc:3058] T 961ade36f7d14eba91bb16d5eae2e254 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Advancing to term 2
I20250903 20:05:31.348692 28808 raft_consensus.cc:2802] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:31.348807 28808 raft_consensus.cc:695] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:31.349086 28710 leader_election.cc:290] T 512908bc4d6941ec81b145ee5795405f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:31.349032 28808 consensus_queue.cc:237] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.352594 28870 leader_election.cc:290] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:31.353348 27517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512908bc4d6941ec81b145ee5795405f" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:31.353899 27517 raft_consensus.cc:2466] T 512908bc4d6941ec81b145ee5795405f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:31.357092 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512908bc4d6941ec81b145ee5795405f" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:31.357767 27516 raft_consensus.cc:3058] T 512908bc4d6941ec81b145ee5795405f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.359059 27249 raft_consensus.cc:2466] T 961ade36f7d14eba91bb16d5eae2e254 P 89f8f0ee68b940db9ce250c0604f934a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 2.
I20250903 20:05:31.360755 28596 raft_consensus.cc:491] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:31.360925 28596 raft_consensus.cc:513] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.361534 28596 leader_election.cc:290] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:31.362702 27687 ts_tablet_manager.cc:1918] T 607c9a9cbc5540e5bd0fa436fcba564c P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:31.362969 27687 log.cc:1199] T 607c9a9cbc5540e5bd0fa436fcba564c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/607c9a9cbc5540e5bd0fa436fcba564c
I20250903 20:05:31.363426 27687 ts_tablet_manager.cc:1939] T 607c9a9cbc5540e5bd0fa436fcba564c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:31.364517 27516 raft_consensus.cc:2466] T 512908bc4d6941ec81b145ee5795405f P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:31.365458 27684 raft_consensus.cc:2241] T 6f8cfe53b91b41d7ba7f9074e1005af3 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:31.365653 27684 raft_consensus.cc:2270] T 6f8cfe53b91b41d7ba7f9074e1005af3 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:31.366251 27684 ts_tablet_manager.cc:1905] T 6f8cfe53b91b41d7ba7f9074e1005af3 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:31.374033 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_fc064d54000a4fe99c75c42de917231b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:31.374869 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc064d54000a4fe99c75c42de917231b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:31.377377 27462 leader_election.cc:304] T 22bc87d1a4804b6c80f77f9692500d63 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:31.377660 28596 raft_consensus.cc:2747] T 22bc87d1a4804b6c80f77f9692500d63 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:31.378036 27516 raft_consensus.cc:3058] T 2059ccbd6d884085886d7d564afe03aa P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:31.378151 27462 leader_election.cc:343] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 961ade36f7d14eba91bb16d5eae2e254
W20250903 20:05:31.378919 27462 leader_election.cc:343] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 961ade36f7d14eba91bb16d5eae2e254
I20250903 20:05:31.380141 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512908bc4d6941ec81b145ee5795405f" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:31.382858 27692 tablet_bootstrap.cc:492] T 567630e2a93645e58f9bf05ffff87b32 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:31.383015 27692 ts_tablet_manager.cc:1397] T 567630e2a93645e58f9bf05ffff87b32 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.133s user 0.020s sys 0.009s
I20250903 20:05:31.383819 27692 raft_consensus.cc:357] T 567630e2a93645e58f9bf05ffff87b32 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.384006 27692 raft_consensus.cc:383] T 567630e2a93645e58f9bf05ffff87b32 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:31.384068 27692 raft_consensus.cc:738] T 567630e2a93645e58f9bf05ffff87b32 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:31.384254 27692 consensus_queue.cc:260] T 567630e2a93645e58f9bf05ffff87b32 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.384716 27692 ts_tablet_manager.cc:1428] T 567630e2a93645e58f9bf05ffff87b32 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:31.384985 27692 tablet_bootstrap.cc:492] T 8fff36d5a80e45af8a256c3908555e08 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:31.386790 27692 tablet_bootstrap.cc:654] T 8fff36d5a80e45af8a256c3908555e08 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:31.388798 27059 rpcz_store.cc:276] 0903 20:05:30.029525 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.723275 (+693750us) service_pool.cc:225] Handling call
0903 20:05:30.723640 (+ 365us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0903 20:05:30.723980 (+ 340us) catalog_manager.cc:3683] Apply alter schema
0903 20:05:30.724267 (+ 287us) catalog_manager.cc:3715] Acquired catalog manager lock
0903 20:05:30.724383 (+ 116us) catalog_manager.cc:3873] Updating metadata on disk
0903 20:05:31.167588 (+443205us) catalog_manager.cc:3904] Committing alterations to in-memory state
0903 20:05:31.167999 (+ 411us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:31.149652 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:31.149786 (+ 134us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:31.149794 (+ 8us) write_op.cc:435] Acquired schema lock
0903 20:05:31.149796 (+ 2us) tablet.cc:660] Decoding operations
0903 20:05:31.149910 (+ 114us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:31.149923 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:31.149928 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0903 20:05:31.150015 (+ 87us) tablet.cc:699] Row locks acquired
0903 20:05:31.150019 (+ 4us) write_op.cc:260] PREPARE: finished
0903 20:05:31.150063 (+ 44us) write_op.cc:270] Start()
0903 20:05:31.150100 (+ 37us) write_op.cc:276] Timestamp: P: 1756929931150004 usec, L: 0
0903 20:05:31.150104 (+ 4us) op_driver.cc:348] REPLICATION: starting
0903 20:05:31.160447 (+ 10343us) log.cc:844] Serialized 405 byte log entry
0903 20:05:31.166834 (+ 6387us) op_driver.cc:464] REPLICATION: finished
0903 20:05:31.167027 (+ 193us) write_op.cc:301] APPLY: starting
0903 20:05:31.167051 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:31.167117 (+ 66us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:31.167155 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:31.167218 (+ 63us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:31.167257 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0903 20:05:31.167264 (+ 7us) write_op.cc:312] APPLY: finished
0903 20:05:31.167358 (+ 94us) log.cc:844] Serialized 23 byte log entry
0903 20:05:31.167420 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:31.167443 (+ 23us) write_op.cc:454] Released schema lock
0903 20:05:31.167451 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0903 20:05:31.237577 (+ 70126us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:31.237585 (+ 8us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":135,"apply.run_cpu_time_us":495,"apply.run_wall_time_us":495,"num_ops":1,"prepare.queue_time_us":424707,"prepare.run_cpu_time_us":1011,"prepare.run_wall_time_us":87904,"raft.queue_time_us":52,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":713,"replication_time_us":16690}]]}
I20250903 20:05:31.389503 27695 tablet_bootstrap.cc:492] T 541278ca9ffb4e41b341139c2faa337f P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:31.389734 27695 ts_tablet_manager.cc:1397] T 541278ca9ffb4e41b341139c2faa337f P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.114s user 0.013s sys 0.000s
I20250903 20:05:31.285712 27685 raft_consensus.cc:2270] T a4c582b441f6467c99ba0b090abcf4cc P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:31.390767 27685 ts_tablet_manager.cc:1905] T a4c582b441f6467c99ba0b090abcf4cc P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:31.390897 27695 raft_consensus.cc:357] T 541278ca9ffb4e41b341139c2faa337f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.391220 27695 raft_consensus.cc:383] T 541278ca9ffb4e41b341139c2faa337f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:31.391338 27695 raft_consensus.cc:738] T 541278ca9ffb4e41b341139c2faa337f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:31.391701 27695 consensus_queue.cc:260] T 541278ca9ffb4e41b341139c2faa337f P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.392321 27695 ts_tablet_manager.cc:1428] T 541278ca9ffb4e41b341139c2faa337f P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:31.392752 27695 tablet_bootstrap.cc:492] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:31.392813 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0f79831866b40c49c5b27ce2014c3f6" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:31.394513 27695 tablet_bootstrap.cc:654] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:31.400545 27693 raft_consensus.cc:738] T 67ca1dcf6fc7435db555392235f9900f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:31.403054 28596 consensus_queue.cc:1035] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:31.404665 27462 leader_election.cc:304] T 67ca1dcf6fc7435db555392235f9900f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:31.404861 28967 tablet.cc:1722] T ac1b0dc95fca49b4af6697c0a70e9730 P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:31.405483 28596 raft_consensus.cc:2802] T 67ca1dcf6fc7435db555392235f9900f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:31.405566 28596 raft_consensus.cc:491] T 67ca1dcf6fc7435db555392235f9900f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:31.405640 28596 raft_consensus.cc:3058] T 67ca1dcf6fc7435db555392235f9900f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.406386 27692 tablet_bootstrap.cc:492] T 8fff36d5a80e45af8a256c3908555e08 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:31.407142 27692 ts_tablet_manager.cc:1397] T 8fff36d5a80e45af8a256c3908555e08 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20250903 20:05:31.408212 27692 raft_consensus.cc:357] T 8fff36d5a80e45af8a256c3908555e08 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.403862 28808 raft_consensus.cc:491] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:31.412789 28870 consensus_queue.cc:1035] T 2059ccbd6d884085886d7d564afe03aa P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:31.414024 27692 raft_consensus.cc:383] T 8fff36d5a80e45af8a256c3908555e08 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:31.414777 28808 raft_consensus.cc:513] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.409057 28596 raft_consensus.cc:513] T 67ca1dcf6fc7435db555392235f9900f P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:31.409511 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "541278ca9ffb4e41b341139c2faa337f" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:31.416633 28808 leader_election.cc:290] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:31.392258 27060 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:45936 (request call id 1149) took 1321 ms. Trace:
I20250903 20:05:31.414121 27692 raft_consensus.cc:738] T 8fff36d5a80e45af8a256c3908555e08 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:31.417986 27692 consensus_queue.cc:260] T 8fff36d5a80e45af8a256c3908555e08 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.417786 27060 rpcz_store.cc:276] 0903 20:05:30.070402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.890308 (+819906us) service_pool.cc:225] Handling call
0903 20:05:30.895667 (+ 5359us) spinlock_profiling.cc:243] Waited 5.04 ms on lock 0x616000040f54. stack: 00007f4b9d6c2dc6 00007f4b9d6c2b80 00007f4b9c6aa3ad 0000000000972d94 0000000000972d5d 00007f4bac49014d 00007f4bac490243 00007f4ba914f941 00007f4bb0cdaa7d 00007f4bb0e63eb3 00007f4ba7c9b7a6 00007f4ba7c9b4d5 00007f4ba44730be 00007f4ba4471f9e 00007f4ba44781d5 00007f4ba4478aff
0903 20:05:30.896807 (+ 1140us) catalog_manager.cc:2076] Acquired catalog manager lock
0903 20:05:30.897396 (+ 589us) catalog_manager.cc:2122] Created new table and tablet info
0903 20:05:31.391962 (+494566us) catalog_manager.cc:2182] Wrote table and tablets to system table
0903 20:05:31.392097 (+ 135us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0903 20:05:31.392210 (+ 113us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:31.264227 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:31.264319 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:31.264323 (+ 4us) write_op.cc:435] Acquired schema lock
0903 20:05:31.264325 (+ 2us) tablet.cc:660] Decoding operations
0903 20:05:31.264398 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:31.264406 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:31.264410 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0903 20:05:31.264466 (+ 56us) tablet.cc:699] Row locks acquired
0903 20:05:31.264468 (+ 2us) write_op.cc:260] PREPARE: finished
0903 20:05:31.264497 (+ 29us) write_op.cc:270] Start()
0903 20:05:31.264525 (+ 28us) write_op.cc:276] Timestamp: P: 1756929931264475 usec, L: 0
0903 20:05:31.264530 (+ 5us) op_driver.cc:348] REPLICATION: starting
0903 20:05:31.264669 (+ 139us) log.cc:844] Serialized 433 byte log entry
0903 20:05:31.389686 (+125017us) op_driver.cc:464] REPLICATION: finished
0903 20:05:31.390711 (+ 1025us) write_op.cc:301] APPLY: starting
0903 20:05:31.390724 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:31.390757 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:31.390777 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:31.390901 (+ 124us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:31.390931 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0903 20:05:31.390939 (+ 8us) write_op.cc:312] APPLY: finished
0903 20:05:31.391000 (+ 61us) log.cc:844] Serialized 29 byte log entry
0903 20:05:31.391046 (+ 46us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:31.391073 (+ 27us) write_op.cc:454] Released schema lock
0903 20:05:31.391079 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0903 20:05:31.391143 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:31.391147 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":11593600,"child_traces":[["op",{"apply.queue_time_us":992,"apply.run_cpu_time_us":424,"apply.run_wall_time_us":424,"num_ops":2,"prepare.queue_time_us":366172,"prepare.run_cpu_time_us":491,"prepare.run_wall_time_us":490,"replication_time_us":125142}]]}
I20250903 20:05:31.418597 27692 ts_tablet_manager.cc:1428] T 8fff36d5a80e45af8a256c3908555e08 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250903 20:05:31.418908 27692 tablet_bootstrap.cc:492] T 976322efa9464ed7a2175d3e3de51dab P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:31.420423 27684 ts_tablet_manager.cc:1918] T 6f8cfe53b91b41d7ba7f9074e1005af3 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:31.420532 27684 log.cc:1199] T 6f8cfe53b91b41d7ba7f9074e1005af3 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/6f8cfe53b91b41d7ba7f9074e1005af3
I20250903 20:05:31.420781 27692 tablet_bootstrap.cc:654] T 976322efa9464ed7a2175d3e3de51dab P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:31.420959 27684 ts_tablet_manager.cc:1939] T 6f8cfe53b91b41d7ba7f9074e1005af3 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
W20250903 20:05:31.392606 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 931404d9b3844d908baaed3190d57b56 consensus state has no leader
I20250903 20:05:31.421932 27061 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:45936 (request call id 1150) took 1313 ms. Trace:
I20250903 20:05:31.395578 27058 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:45936 (request call id 1151) took 1284 ms. Trace:
I20250903 20:05:31.423159 27058 rpcz_store.cc:276] 0903 20:05:30.110648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.011838 (+901190us) service_pool.cc:225] Handling call
0903 20:05:31.031078 (+ 19240us) spinlock_profiling.cc:243] Waited 18.9 ms on lock 0x616000040f54. stack: 00007f4b9d6c2dc6 00007f4b9d6c2b80 00007f4b9c6aa3ad 0000000000972d94 0000000000972d5d 00007f4bac49014d 00007f4bac490243 00007f4ba914f941 00007f4bb0cdaa7d 00007f4bb0e63eb3 00007f4ba7c9b7a6 00007f4ba7c9b4d5 00007f4ba44730be 00007f4ba4471f9e 00007f4ba44781d5 00007f4ba4478aff
0903 20:05:31.032284 (+ 1206us) catalog_manager.cc:2076] Acquired catalog manager lock
0903 20:05:31.033030 (+ 746us) catalog_manager.cc:2122] Created new table and tablet info
0903 20:05:31.395308 (+362278us) catalog_manager.cc:2182] Wrote table and tablets to system table
0903 20:05:31.395426 (+ 118us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0903 20:05:31.395535 (+ 109us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:31.265254 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:31.265347 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:31.265352 (+ 5us) write_op.cc:435] Acquired schema lock
0903 20:05:31.265354 (+ 2us) tablet.cc:660] Decoding operations
0903 20:05:31.265419 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:31.265427 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:31.265431 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0903 20:05:31.265487 (+ 56us) tablet.cc:699] Row locks acquired
0903 20:05:31.265490 (+ 3us) write_op.cc:260] PREPARE: finished
0903 20:05:31.265512 (+ 22us) write_op.cc:270] Start()
0903 20:05:31.265533 (+ 21us) write_op.cc:276] Timestamp: P: 1756929931265492 usec, L: 0
0903 20:05:31.265536 (+ 3us) op_driver.cc:348] REPLICATION: starting
0903 20:05:31.265645 (+ 109us) log.cc:844] Serialized 433 byte log entry
0903 20:05:31.394096 (+128451us) op_driver.cc:464] REPLICATION: finished
0903 20:05:31.394308 (+ 212us) write_op.cc:301] APPLY: starting
0903 20:05:31.394322 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:31.394364 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:31.394388 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:31.394498 (+ 110us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:31.394530 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0903 20:05:31.394537 (+ 7us) write_op.cc:312] APPLY: finished
0903 20:05:31.394614 (+ 77us) log.cc:844] Serialized 29 byte log entry
0903 20:05:31.394678 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:31.394695 (+ 17us) write_op.cc:454] Released schema lock
0903 20:05:31.394701 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0903 20:05:31.394790 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:31.394794 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":43491072,"child_traces":[["op",{"apply.queue_time_us":159,"apply.run_cpu_time_us":473,"apply.run_wall_time_us":473,"num_ops":2,"prepare.queue_time_us":231546,"prepare.run_cpu_time_us":455,"prepare.run_wall_time_us":454,"replication_time_us":128517}]]}
I20250903 20:05:31.424464 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_94e9e32ef1134f07ade1cdf8da69ead6" } modify_external_catalogs: true
I20250903 20:05:31.424672 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_94e9e32ef1134f07ade1cdf8da69ead6" } modify_external_catalogs: true
I20250903 20:05:31.425688 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 885) took 3046 ms. Trace:
I20250903 20:05:31.425809 27684 rpcz_store.cc:276] 0903 20:05:28.379140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.379189 (+ 49us) service_pool.cc:225] Handling call
0903 20:05:31.332991 (+2953802us) ts_tablet_manager.cc:1145] Deleting tablet 6f8cfe53b91b41d7ba7f9074e1005af3
0903 20:05:31.333011 (+ 20us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.383513 (+ 50502us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.420284 (+ 36771us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.425661 (+ 5377us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2953577}
W20250903 20:05:31.395916 27057 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3739ac27204c4eb18747a0ae0d9254eb has no consensus state
I20250903 20:05:31.426191 27684 tablet_replica.cc:331] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:31.426275 27057 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:45936 (request call id 1152) took 1151 ms. Trace:
I20250903 20:05:31.426393 27684 raft_consensus.cc:2241] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:31.426560 27684 raft_consensus.cc:2270] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:31.426429 27057 rpcz_store.cc:276] 0903 20:05:30.274965 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.055205 (+780240us) service_pool.cc:225] Handling call
0903 20:05:31.088445 (+ 33240us) spinlock_profiling.cc:243] Waited 32.9 ms on lock 0x616000040f54. stack: 00007f4b9d6c2dc6 00007f4b9d6c2b80 00007f4b9c6aa3ad 0000000000972d94 0000000000972d5d 00007f4bac49014d 00007f4bac490243 00007f4ba914f941 00007f4bb0cdaa7d 00007f4bb0e66146 00007f4ba7c9f126 00007f4ba7c9ee55 00007f4ba44730be 00007f4ba4471f9e 00007f4ba44781d5 00007f4ba4478aff
0903 20:05:31.088756 (+ 311us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0903 20:05:31.088958 (+ 202us) catalog_manager.cc:3683] Apply alter schema
0903 20:05:31.089157 (+ 199us) catalog_manager.cc:3715] Acquired catalog manager lock
0903 20:05:31.089239 (+ 82us) catalog_manager.cc:3873] Updating metadata on disk
0903 20:05:31.395705 (+306466us) catalog_manager.cc:3904] Committing alterations to in-memory state
0903 20:05:31.426251 (+ 30546us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:31.265741 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:31.265826 (+ 85us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:31.265830 (+ 4us) write_op.cc:435] Acquired schema lock
0903 20:05:31.265832 (+ 2us) tablet.cc:660] Decoding operations
0903 20:05:31.265898 (+ 66us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:31.265906 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:31.265909 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0903 20:05:31.265950 (+ 41us) tablet.cc:699] Row locks acquired
0903 20:05:31.265952 (+ 2us) write_op.cc:260] PREPARE: finished
0903 20:05:31.265988 (+ 36us) write_op.cc:270] Start()
0903 20:05:31.266011 (+ 23us) write_op.cc:276] Timestamp: P: 1756929931265955 usec, L: 0
0903 20:05:31.266015 (+ 4us) op_driver.cc:348] REPLICATION: starting
0903 20:05:31.266141 (+ 126us) log.cc:844] Serialized 405 byte log entry
0903 20:05:31.394172 (+128031us) op_driver.cc:464] REPLICATION: finished
0903 20:05:31.394865 (+ 693us) write_op.cc:301] APPLY: starting
0903 20:05:31.394876 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:31.394906 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:31.394944 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:31.394982 (+ 38us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:31.395008 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0903 20:05:31.395015 (+ 7us) write_op.cc:312] APPLY: finished
0903 20:05:31.395084 (+ 69us) log.cc:844] Serialized 23 byte log entry
0903 20:05:31.395127 (+ 43us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:31.395139 (+ 12us) write_op.cc:454] Released schema lock
0903 20:05:31.395144 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0903 20:05:31.395208 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:31.395212 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":75736832,"child_traces":[["op",{"apply.queue_time_us":655,"apply.run_cpu_time_us":338,"apply.run_wall_time_us":337,"num_ops":1,"prepare.queue_time_us":176014,"prepare.run_cpu_time_us":448,"prepare.run_wall_time_us":447,"replication_time_us":128142}]]}
I20250903 20:05:31.427121 27684 ts_tablet_manager.cc:1905] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:31.427327 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_58569618f9d04fdfbd005b3d0e15c049" } modify_external_catalogs: true
I20250903 20:05:31.427524 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_58569618f9d04fdfbd005b3d0e15c049" } modify_external_catalogs: true
I20250903 20:05:31.428679 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 6f8cfe53b91b41d7ba7f9074e1005af3 (table default.Table_e5def563dd414101b9ee8fbaf7156e29 [id=7e67c0d1d2f34172a987a8ea785c5a6a]) successfully deleted
I20250903 20:05:31.422071 27061 rpcz_store.cc:276] 0903 20:05:30.108107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.907013 (+798906us) service_pool.cc:225] Handling call
0903 20:05:30.959883 (+ 52870us) spinlock_profiling.cc:243] Waited 52.5 ms on lock 0x616000040f54. stack: 00007f4b9d6c2dc6 00007f4b9d6c2b80 00007f4b9c6aa3ad 0000000000972d94 0000000000972d5d 00007f4bac49014d 00007f4bac490243 00007f4ba914f941 00007f4bb0cdaa7d 00007f4bb0e66146 00007f4ba7c9f126 00007f4ba7c9ee55 00007f4ba44730be 00007f4ba4471f9e 00007f4ba44781d5 00007f4ba4478aff
0903 20:05:30.960223 (+ 340us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0903 20:05:30.960462 (+ 239us) catalog_manager.cc:3683] Apply alter schema
0903 20:05:30.960663 (+ 201us) catalog_manager.cc:3715] Acquired catalog manager lock
0903 20:05:30.960781 (+ 118us) catalog_manager.cc:3873] Updating metadata on disk
0903 20:05:31.392387 (+431606us) catalog_manager.cc:3904] Committing alterations to in-memory state
0903 20:05:31.421897 (+ 29510us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:31.264751 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:31.264833 (+ 82us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:31.264837 (+ 4us) write_op.cc:435] Acquired schema lock
0903 20:05:31.264839 (+ 2us) tablet.cc:660] Decoding operations
0903 20:05:31.264913 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:31.264920 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:31.264924 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0903 20:05:31.264972 (+ 48us) tablet.cc:699] Row locks acquired
0903 20:05:31.264988 (+ 16us) write_op.cc:260] PREPARE: finished
0903 20:05:31.265012 (+ 24us) write_op.cc:270] Start()
0903 20:05:31.265035 (+ 23us) write_op.cc:276] Timestamp: P: 1756929931264991 usec, L: 0
0903 20:05:31.265038 (+ 3us) op_driver.cc:348] REPLICATION: starting
0903 20:05:31.265170 (+ 132us) log.cc:844] Serialized 405 byte log entry
0903 20:05:31.389738 (+124568us) op_driver.cc:464] REPLICATION: finished
0903 20:05:31.391219 (+ 1481us) write_op.cc:301] APPLY: starting
0903 20:05:31.391232 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:31.391264 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:31.391295 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:31.391330 (+ 35us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:31.391355 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0903 20:05:31.391366 (+ 11us) write_op.cc:312] APPLY: finished
0903 20:05:31.391416 (+ 50us) log.cc:844] Serialized 23 byte log entry
0903 20:05:31.391453 (+ 37us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:31.391464 (+ 11us) write_op.cc:454] Released schema lock
0903 20:05:31.391469 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":120857856,"child_traces":[["op",{"apply.queue_time_us":1441,"apply.run_cpu_time_us":304,"apply.run_wall_time_us":305,"num_ops":1,"prepare.queue_time_us":303443,"prepare.run_cpu_time_us":467,"prepare.run_wall_time_us":466,"replication_time_us":124685,"thread_start_us":4196,"threads_started":1}]]}
I20250903 20:05:31.430527 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_10c65228c77a4a9d8ccc535d2792f59f" } modify_external_catalogs: true
I20250903 20:05:31.430721 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_10c65228c77a4a9d8ccc535d2792f59f" } modify_external_catalogs: true
I20250903 20:05:31.409215 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512908bc4d6941ec81b145ee5795405f" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:31.409369 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0f79831866b40c49c5b27ce2014c3f6" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:31.431478 27381 raft_consensus.cc:2466] T a0f79831866b40c49c5b27ce2014c3f6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:31.431941 27385 raft_consensus.cc:2391] T 512908bc4d6941ec81b145ee5795405f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 89f8f0ee68b940db9ce250c0604f934a in this term.
I20250903 20:05:31.410724 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 945) took 1324 ms. Trace:
I20250903 20:05:31.416033 27384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63a1f26fac3b4b939ebb79c6a07d8f8e" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:31.432334 27384 raft_consensus.cc:2391] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89f8f0ee68b940db9ce250c0604f934a in current term 1: Already voted for candidate e72ef1dbcc5143cf9f04a808f101c80f in this term.
I20250903 20:05:31.432760 27687 rpcz_store.cc:276] 0903 20:05:30.085912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.086800 (+ 888us) service_pool.cc:225] Handling call
0903 20:05:31.290774 (+1203974us) ts_tablet_manager.cc:1145] Deleting tablet 607c9a9cbc5540e5bd0fa436fcba564c
0903 20:05:31.290797 (+ 23us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.299357 (+ 8560us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.333595 (+ 34238us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.410703 (+ 77108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1201830}
I20250903 20:05:31.433249 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 607c9a9cbc5540e5bd0fa436fcba564c (table default.Table_5b44669e91a8427c8e7d96b866fdb3b5 [id=96167b0d9f9b47168a9fc59635b7136d]) successfully deleted
I20250903 20:05:31.433696 27687 tablet_replica.cc:331] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:31.433930 27687 raft_consensus.cc:2241] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:31.434336 27687 raft_consensus.cc:2270] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:31.435197 27687 ts_tablet_manager.cc:1905] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:31.416446 28710 consensus_queue.cc:1035] T 9e98cd1d9bd246e182e2969d147e68bf P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20250903 20:05:31.416179 27518 raft_consensus.cc:3058] T 541278ca9ffb4e41b341139c2faa337f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.439404 27518 raft_consensus.cc:2466] T 541278ca9ffb4e41b341139c2faa337f P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:31.415911 28596 leader_election.cc:290] T 67ca1dcf6fc7435db555392235f9900f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:31.416949 27508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63a1f26fac3b4b939ebb79c6a07d8f8e" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:31.416540 27462 leader_election.cc:304] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
W20250903 20:05:31.440208 27462 raft_consensus.cc:2684] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250903 20:05:31.440351 27183 leader_election.cc:304] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:31.440713 27462 leader_election.cc:343] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:31.440754 28947 raft_consensus.cc:3058] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.443408 28947 raft_consensus.cc:2747] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:31.448127 27184 consensus_peers.cc:489] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0a9b7770432942da9373d5100c71bd6a. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:31.448717 27693 consensus_queue.cc:260] T 67ca1dcf6fc7435db555392235f9900f P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:31.450312 27376 raft_consensus.cc:2466] T 67ca1dcf6fc7435db555392235f9900f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:31.451399 27693 ts_tablet_manager.cc:1428] T 67ca1dcf6fc7435db555392235f9900f P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.182s user 0.022s sys 0.001s
I20250903 20:05:31.455013 27693 tablet_bootstrap.cc:492] T 976322efa9464ed7a2175d3e3de51dab P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:31.457047 27693 tablet_bootstrap.cc:654] T 976322efa9464ed7a2175d3e3de51dab P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:31.457755 28808 consensus_queue.cc:1035] T 2059ccbd6d884085886d7d564afe03aa P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:31.466449 27693 tablet_bootstrap.cc:492] T 976322efa9464ed7a2175d3e3de51dab P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:31.467258 27693 ts_tablet_manager.cc:1397] T 976322efa9464ed7a2175d3e3de51dab P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.002s
I20250903 20:05:31.469375 27693 raft_consensus.cc:357] T 976322efa9464ed7a2175d3e3de51dab P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.471673 27684 ts_tablet_manager.cc:1918] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:31.472077 27693 raft_consensus.cc:383] T 976322efa9464ed7a2175d3e3de51dab P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:31.472383 27693 raft_consensus.cc:738] T 976322efa9464ed7a2175d3e3de51dab P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:31.460477 27687 ts_tablet_manager.cc:1918] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:31.472815 27687 log.cc:1199] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/90baf83357494a0aae325908c118b471
I20250903 20:05:31.473388 27687 ts_tablet_manager.cc:1939] T 90baf83357494a0aae325908c118b471 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:31.474332 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 946) took 1367 ms. Trace:
I20250903 20:05:31.474866 27687 rpcz_store.cc:276] 0903 20:05:30.106828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.106978 (+ 150us) service_pool.cc:225] Handling call
0903 20:05:31.433586 (+1326608us) ts_tablet_manager.cc:1145] Deleting tablet 90baf83357494a0aae325908c118b471
0903 20:05:31.433600 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.438320 (+ 4720us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.460361 (+ 22041us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.474307 (+ 13946us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1326343}
I20250903 20:05:31.475641 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 90baf83357494a0aae325908c118b471 (table default.Table_75f638a7c40e48e584a02949e6f864fd [id=7e9a08baa86e406c85192816b4a5d26d]) successfully deleted
I20250903 20:05:31.474660 27693 consensus_queue.cc:260] T 976322efa9464ed7a2175d3e3de51dab P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.477450 27693 ts_tablet_manager.cc:1428] T 976322efa9464ed7a2175d3e3de51dab P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
W20250903 20:05:31.478137 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 167b359047ac457c9d165ec199cd9163 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 167b359047ac457c9d165ec199cd9163 already in progress: creating tablet
W20250903 20:05:31.479063 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 07439db2c3b8470eb2b797cc68376aa7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07439db2c3b8470eb2b797cc68376aa7 already in progress: creating tablet
I20250903 20:05:31.479691 27693 tablet_bootstrap.cc:492] T a253e3399622409e92299069b19d3698 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:31.479951 27684 log.cc:1199] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/961ade36f7d14eba91bb16d5eae2e254
I20250903 20:05:31.481094 27684 ts_tablet_manager.cc:1939] T 961ade36f7d14eba91bb16d5eae2e254 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:31.482599 27693 tablet_bootstrap.cc:654] T a253e3399622409e92299069b19d3698 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:31.483013 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 1f2124e9baae4c819aeb51cef79c9ff3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f2124e9baae4c819aeb51cef79c9ff3 already in progress: creating tablet
I20250903 20:05:31.484678 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 886) took 3105 ms. Trace:
I20250903 20:05:31.483387 27687 tablet_replica.cc:331] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:31.487126 27687 raft_consensus.cc:2241] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:31.487510 27687 raft_consensus.cc:2270] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:31.488128 27687 ts_tablet_manager.cc:1905] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:31.485775 27684 rpcz_store.cc:276] 0903 20:05:28.379271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.379325 (+ 54us) service_pool.cc:225] Handling call
0903 20:05:31.426135 (+3046810us) ts_tablet_manager.cc:1145] Deleting tablet 961ade36f7d14eba91bb16d5eae2e254
0903 20:05:31.426148 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.461665 (+ 35517us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.471547 (+ 9882us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.484655 (+ 13108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3046594}
I20250903 20:05:31.489926 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 961ade36f7d14eba91bb16d5eae2e254 (table default.Table_459903503b5148d09d3cb8f2f5519fab [id=3c69612aaf164dc899dcc252d85636f6]) successfully deleted
I20250903 20:05:31.490566 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 897) took 2753 ms. Trace:
I20250903 20:05:31.491055 27684 rpcz_store.cc:276] 0903 20:05:28.736687 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.736836 (+ 149us) service_pool.cc:225] Handling call
0903 20:05:31.490405 (+2753569us) ts_tablet_manager.cc:1145] Deleting tablet 4739f10128d64e2c986257b1d9deb90f
0903 20:05:31.490419 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.490545 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2753366}
I20250903 20:05:31.494045 27695 tablet_bootstrap.cc:492] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:31.494412 27695 ts_tablet_manager.cc:1397] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.102s user 0.008s sys 0.007s
I20250903 20:05:31.495292 27695 raft_consensus.cc:357] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.495544 27695 raft_consensus.cc:383] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:31.495637 27695 raft_consensus.cc:738] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:31.495931 27695 consensus_queue.cc:260] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.496521 27695 ts_tablet_manager.cc:1428] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:31.496866 27695 tablet_bootstrap.cc:492] T a0f79831866b40c49c5b27ce2014c3f6 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:31.497280 27062 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:45936 (request call id 1156) took 1059 ms. Trace:
I20250903 20:05:31.497449 27062 rpcz_store.cc:276] 0903 20:05:30.437959 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.277722 (+839763us) service_pool.cc:225] Handling call
0903 20:05:31.373970 (+ 96248us) spinlock_profiling.cc:243] Waited 95.9 ms on lock 0x616000040f54. stack: 00007f4b9d6c2dc6 00007f4b9d6c2b80 00007f4b9c6aa3ad 0000000000972d94 0000000000972d5d 00007f4bac49014d 00007f4bac490243 00007f4ba914f941 00007f4bb0cdaa7d 00007f4bb0e63eb3 00007f4ba7c9b7a6 00007f4ba7c9b4d5 00007f4ba44730be 00007f4ba4471f9e 00007f4ba44781d5 00007f4ba4478aff
0903 20:05:31.374988 (+ 1018us) catalog_manager.cc:2076] Acquired catalog manager lock
0903 20:05:31.375525 (+ 537us) catalog_manager.cc:2122] Created new table and tablet info
0903 20:05:31.497007 (+121482us) catalog_manager.cc:2182] Wrote table and tablets to system table
0903 20:05:31.497134 (+ 127us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0903 20:05:31.497241 (+ 107us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:31.381580 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:31.381739 (+ 159us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:31.381746 (+ 7us) write_op.cc:435] Acquired schema lock
0903 20:05:31.381749 (+ 3us) tablet.cc:660] Decoding operations
0903 20:05:31.381868 (+ 119us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:31.381883 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:31.381888 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0903 20:05:31.382012 (+ 124us) tablet.cc:699] Row locks acquired
0903 20:05:31.382016 (+ 4us) write_op.cc:260] PREPARE: finished
0903 20:05:31.382060 (+ 44us) write_op.cc:270] Start()
0903 20:05:31.382100 (+ 40us) write_op.cc:276] Timestamp: P: 1756929931382025 usec, L: 0
0903 20:05:31.382103 (+ 3us) op_driver.cc:348] REPLICATION: starting
0903 20:05:31.382369 (+ 266us) log.cc:844] Serialized 433 byte log entry
0903 20:05:31.494268 (+111899us) op_driver.cc:464] REPLICATION: finished
0903 20:05:31.495498 (+ 1230us) write_op.cc:301] APPLY: starting
0903 20:05:31.495520 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:31.495570 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:31.495595 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:31.495762 (+ 167us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:31.495799 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0903 20:05:31.495806 (+ 7us) write_op.cc:312] APPLY: finished
0903 20:05:31.495892 (+ 86us) log.cc:844] Serialized 29 byte log entry
0903 20:05:31.495968 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:31.495992 (+ 24us) write_op.cc:454] Released schema lock
0903 20:05:31.496000 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0903 20:05:31.496078 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:31.496084 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":220472064,"child_traces":[["op",{"apply.queue_time_us":1164,"apply.run_cpu_time_us":573,"apply.run_wall_time_us":572,"num_ops":2,"prepare.queue_time_us":2883,"prepare.run_cpu_time_us":881,"prepare.run_wall_time_us":881,"replication_time_us":112079}]]}
I20250903 20:05:31.498358 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e6fcf7607d01415abdf6d459997b3ede"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:31.498729 27695 tablet_bootstrap.cc:654] T a0f79831866b40c49c5b27ce2014c3f6 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:31.498991 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6fcf7607d01415abdf6d459997b3ede in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:31.496997 27692 tablet_bootstrap.cc:492] T 976322efa9464ed7a2175d3e3de51dab P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:31.500065 27692 ts_tablet_manager.cc:1397] T 976322efa9464ed7a2175d3e3de51dab P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.081s user 0.014s sys 0.015s
I20250903 20:05:31.500803 27692 raft_consensus.cc:357] T 976322efa9464ed7a2175d3e3de51dab P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.500982 27692 raft_consensus.cc:383] T 976322efa9464ed7a2175d3e3de51dab P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:31.501030 27692 raft_consensus.cc:738] T 976322efa9464ed7a2175d3e3de51dab P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:31.501261 27692 consensus_queue.cc:260] T 976322efa9464ed7a2175d3e3de51dab P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.501818 27692 ts_tablet_manager.cc:1428] T 976322efa9464ed7a2175d3e3de51dab P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:31.502331 27692 tablet_bootstrap.cc:492] T 120ca61b3da742dc9a2efa483ead7e97 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
W20250903 20:05:31.502385 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 4739f10128d64e2c986257b1d9deb90f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4739f10128d64e2c986257b1d9deb90f already in progress: creating tablet
I20250903 20:05:31.504053 27692 tablet_bootstrap.cc:654] T 120ca61b3da742dc9a2efa483ead7e97 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:31.505165 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 898) took 2768 ms. Trace:
I20250903 20:05:31.505257 27684 rpcz_store.cc:276] 0903 20:05:28.736794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.737062 (+ 268us) service_pool.cc:225] Handling call
0903 20:05:31.505032 (+2767970us) ts_tablet_manager.cc:1145] Deleting tablet b3bb55085af044ec8bb2368c845e7404
0903 20:05:31.505042 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.505145 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2767850}
I20250903 20:05:31.505616 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 899) took 2768 ms. Trace:
I20250903 20:05:31.505703 27684 rpcz_store.cc:276] 0903 20:05:28.737112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.737196 (+ 84us) service_pool.cc:225] Handling call
0903 20:05:31.505513 (+2768317us) ts_tablet_manager.cc:1145] Deleting tablet de20d18d7f264811a0631aa1144aef2c
0903 20:05:31.505522 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.505601 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2768222}
I20250903 20:05:31.506016 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 900) took 2725 ms. Trace:
I20250903 20:05:31.506093 27684 rpcz_store.cc:276] 0903 20:05:28.780549 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.781008 (+ 459us) service_pool.cc:225] Handling call
0903 20:05:31.505911 (+2724903us) ts_tablet_manager.cc:1145] Deleting tablet 5962685d79044081a75532e19bfa2bce
0903 20:05:31.505919 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.506000 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2724627}
I20250903 20:05:31.506407 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 901) took 2693 ms. Trace:
I20250903 20:05:31.506485 27684 rpcz_store.cc:276] 0903 20:05:28.813091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.813153 (+ 62us) service_pool.cc:225] Handling call
0903 20:05:31.506317 (+2693164us) ts_tablet_manager.cc:1145] Deleting tablet 13496efe34944b7280e7a684f180e6e0
0903 20:05:31.506325 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.506393 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2692967}
I20250903 20:05:31.506754 27684 tablet_replica.cc:331] T 53881ad30bc843839008ace4012a3f70 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:31.506948 27684 raft_consensus.cc:2241] T 53881ad30bc843839008ace4012a3f70 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:31.507171 27684 raft_consensus.cc:2270] T 53881ad30bc843839008ace4012a3f70 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:31.507195 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet b3bb55085af044ec8bb2368c845e7404 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3bb55085af044ec8bb2368c845e7404 already in progress: creating tablet
W20250903 20:05:31.507478 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet de20d18d7f264811a0631aa1144aef2c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de20d18d7f264811a0631aa1144aef2c already in progress: creating tablet
W20250903 20:05:31.507714 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 5962685d79044081a75532e19bfa2bce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5962685d79044081a75532e19bfa2bce already in progress: creating tablet
I20250903 20:05:31.507784 27684 ts_tablet_manager.cc:1905] T 53881ad30bc843839008ace4012a3f70 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:31.507972 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 13496efe34944b7280e7a684f180e6e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13496efe34944b7280e7a684f180e6e0 already in progress: creating tablet
I20250903 20:05:31.508821 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "67ca1dcf6fc7435db555392235f9900f" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:31.509050 27248 raft_consensus.cc:3058] T 67ca1dcf6fc7435db555392235f9900f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.511232 27693 tablet_bootstrap.cc:492] T a253e3399622409e92299069b19d3698 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:31.511384 27693 ts_tablet_manager.cc:1397] T a253e3399622409e92299069b19d3698 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.000s
I20250903 20:05:31.512189 27064 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet a9f43781299e47a8aecc37a0b73d7a64 (table default.Table_73a4ac2afbad4654ac623cd581a45eb2 [id=f70d7ea93e154e6199f6766316353862]). Expected version 3 got 0
I20250903 20:05:31.512152 27693 raft_consensus.cc:357] T a253e3399622409e92299069b19d3698 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.512323 27693 raft_consensus.cc:383] T a253e3399622409e92299069b19d3698 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:31.512302 27248 raft_consensus.cc:2466] T 67ca1dcf6fc7435db555392235f9900f P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:31.512377 27693 raft_consensus.cc:738] T a253e3399622409e92299069b19d3698 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:31.512530 27064 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 2059ccbd6d884085886d7d564afe03aa (table default.Table_ea5cb66a574f4466bf7e7f635f22b426 [id=91abc1158ea047849f398926e3695927]). Expected version 1 got 0
I20250903 20:05:31.512593 27693 consensus_queue.cc:260] T a253e3399622409e92299069b19d3698 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.512898 27064 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 9e98cd1d9bd246e182e2969d147e68bf (table default.Table_28a829a9988442049d50e6ae254e6274 [id=b27f08d56c3c43e6a09a465fdda4d3db]). Expected version 1 got 0
I20250903 20:05:31.513139 27693 ts_tablet_manager.cc:1428] T a253e3399622409e92299069b19d3698 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:31.513468 27693 tablet_bootstrap.cc:492] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:31.517184 27349 tablet_service.cc:1468] Processing CreateTablet for tablet f27e2aa2d0bf4de48429a030146df3d8 (DEFAULT_TABLE table=default.Table_06e3b212bc67490fbe6c44740bbaf704 [id=bf131ce7b3cb45f4ae32d02ca15d2b3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.517616 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f27e2aa2d0bf4de48429a030146df3d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.517783 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "67ca1dcf6fc7435db555392235f9900f" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:31.518026 27385 raft_consensus.cc:3058] T 67ca1dcf6fc7435db555392235f9900f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.520840 27060 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 67ca1dcf6fc7435db555392235f9900f (table default.Table_d3945d5a36d84041b35394333dc704f9 [id=5568f788d1414c83b4359b5adf28632a]). Expected version 2 got 0
I20250903 20:05:31.521102 27060 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 2059ccbd6d884085886d7d564afe03aa (table default.Table_ea5cb66a574f4466bf7e7f635f22b426 [id=91abc1158ea047849f398926e3695927]). Expected version 1 got 0
I20250903 20:05:31.521252 27385 raft_consensus.cc:2466] T 67ca1dcf6fc7435db555392235f9900f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:31.521893 27060 catalog_manager.cc:5582] T 9e98cd1d9bd246e182e2969d147e68bf P e72ef1dbcc5143cf9f04a808f101c80f reported cstate change: term changed from 0 to 1, leader changed from <none> to e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67). New cstate: current_term: 1 leader_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: HEALTHY } } }
I20250903 20:05:31.523785 27507 tablet_service.cc:1468] Processing CreateTablet for tablet f27e2aa2d0bf4de48429a030146df3d8 (DEFAULT_TABLE table=default.Table_06e3b212bc67490fbe6c44740bbaf704 [id=bf131ce7b3cb45f4ae32d02ca15d2b3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.524293 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f27e2aa2d0bf4de48429a030146df3d8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:31.525610 27183 consensus_peers.cc:489] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f8cfe53b91b41d7ba7f9074e1005af3. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:31.527853 27349 tablet_service.cc:1468] Processing CreateTablet for tablet f2309a6166f24503ae1040dc2bdea54b (DEFAULT_TABLE table=default.Table_14b6895698244e7097617536094efbd5 [id=a99afee0f2904ff99cbfc593cc350345]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.528306 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2309a6166f24503ae1040dc2bdea54b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.528251 27227 tablet_service.cc:1468] Processing CreateTablet for tablet f27e2aa2d0bf4de48429a030146df3d8 (DEFAULT_TABLE table=default.Table_06e3b212bc67490fbe6c44740bbaf704 [id=bf131ce7b3cb45f4ae32d02ca15d2b3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.529444 27352 tablet_service.cc:1468] Processing CreateTablet for tablet 70d1b4eed08c4165b23acd9dd920ba0e (DEFAULT_TABLE table=default.Table_6cbf23397cb948a9981807cb7036eb12 [id=994240f9b2ff42b2819915aab6b78941]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.529757 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70d1b4eed08c4165b23acd9dd920ba0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.529750 27215 tablet_service.cc:1468] Processing CreateTablet for tablet f2309a6166f24503ae1040dc2bdea54b (DEFAULT_TABLE table=default.Table_14b6895698244e7097617536094efbd5 [id=a99afee0f2904ff99cbfc593cc350345]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.530298 27228 tablet_service.cc:1468] Processing CreateTablet for tablet 70d1b4eed08c4165b23acd9dd920ba0e (DEFAULT_TABLE table=default.Table_6cbf23397cb948a9981807cb7036eb12 [id=994240f9b2ff42b2819915aab6b78941]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.530345 27494 tablet_service.cc:1468] Processing CreateTablet for tablet f2309a6166f24503ae1040dc2bdea54b (DEFAULT_TABLE table=default.Table_14b6895698244e7097617536094efbd5 [id=a99afee0f2904ff99cbfc593cc350345]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.530689 27494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2309a6166f24503ae1040dc2bdea54b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.530799 27499 tablet_service.cc:1468] Processing CreateTablet for tablet 70d1b4eed08c4165b23acd9dd920ba0e (DEFAULT_TABLE table=default.Table_6cbf23397cb948a9981807cb7036eb12 [id=994240f9b2ff42b2819915aab6b78941]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.531123 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70d1b4eed08c4165b23acd9dd920ba0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.534092 27353 tablet_service.cc:1468] Processing CreateTablet for tablet a5698b385d9c4cc6ab67354c457d2a18 (DEFAULT_TABLE table=default.Table_e6135c51a2164b27bf944da7e525f349 [id=bb11002823cb41a2967c902f980aca1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.534507 27353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5698b385d9c4cc6ab67354c457d2a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.534490 27504 tablet_service.cc:1468] Processing CreateTablet for tablet a5698b385d9c4cc6ab67354c457d2a18 (DEFAULT_TABLE table=default.Table_e6135c51a2164b27bf944da7e525f349 [id=bb11002823cb41a2967c902f980aca1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.534852 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5698b385d9c4cc6ab67354c457d2a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.535838 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 8c04adef0af14dfeb4305d730c8a78fd (DEFAULT_TABLE table=default.Table_b418b651df1e4007a0cb50170fc85a46 [id=fd05981fe6694adea959d14c9b790098]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.536160 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c04adef0af14dfeb4305d730c8a78fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.538560 27213 tablet_service.cc:1468] Processing CreateTablet for tablet a5698b385d9c4cc6ab67354c457d2a18 (DEFAULT_TABLE table=default.Table_e6135c51a2164b27bf944da7e525f349 [id=bb11002823cb41a2967c902f980aca1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.539207 27348 tablet_service.cc:1468] Processing CreateTablet for tablet 8c04adef0af14dfeb4305d730c8a78fd (DEFAULT_TABLE table=default.Table_b418b651df1e4007a0cb50170fc85a46 [id=fd05981fe6694adea959d14c9b790098]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.539585 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c04adef0af14dfeb4305d730c8a78fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.542539 27229 tablet_service.cc:1468] Processing CreateTablet for tablet 8c04adef0af14dfeb4305d730c8a78fd (DEFAULT_TABLE table=default.Table_b418b651df1e4007a0cb50170fc85a46 [id=fd05981fe6694adea959d14c9b790098]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.542811 27494 tablet_service.cc:1468] Processing CreateTablet for tablet e35ae50252734a099ba5ce2d04e49e1f (DEFAULT_TABLE table=default.Table_c9acca1de7954498ab2dcc3ee1ff2f45 [id=8e4ff95b880a4539b0958e34bfd08ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.543184 27494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e35ae50252734a099ba5ce2d04e49e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.543187 27359 tablet_service.cc:1468] Processing CreateTablet for tablet e35ae50252734a099ba5ce2d04e49e1f (DEFAULT_TABLE table=default.Table_c9acca1de7954498ab2dcc3ee1ff2f45 [id=8e4ff95b880a4539b0958e34bfd08ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.543537 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e35ae50252734a099ba5ce2d04e49e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.546545 27221 tablet_service.cc:1468] Processing CreateTablet for tablet e35ae50252734a099ba5ce2d04e49e1f (DEFAULT_TABLE table=default.Table_c9acca1de7954498ab2dcc3ee1ff2f45 [id=8e4ff95b880a4539b0958e34bfd08ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.547055 27222 tablet_service.cc:1468] Processing CreateTablet for tablet 65f93bcf7b674be1b464b68501238f93 (DEFAULT_TABLE table=default.Table_f6fb411b2d7949b4a424655ba7ff4921 [id=78d8c3e9d1fd41b7b119ef6ad6268b6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.547123 27219 tablet_service.cc:1468] Processing CreateTablet for tablet 7dc5214512af449bbc537f111a3d8dbc (DEFAULT_TABLE table=default.Table_8b2d83116755450387b770b591314010 [id=96837f9a9bae4d4b830036d3f309e153]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.547626 27350 tablet_service.cc:1468] Processing CreateTablet for tablet a2358eac86914f3ba51bfc6c299838c3 (DEFAULT_TABLE table=default.Table_77f24f93527744b5b55d7c2243ef8774 [id=a52bbf692b2f4a6a986a547c8db00eab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.547662 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 65f93bcf7b674be1b464b68501238f93 (DEFAULT_TABLE table=default.Table_f6fb411b2d7949b4a424655ba7ff4921 [id=78d8c3e9d1fd41b7b119ef6ad6268b6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.547966 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2358eac86914f3ba51bfc6c299838c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.548069 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65f93bcf7b674be1b464b68501238f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.548142 27488 tablet_service.cc:1468] Processing CreateTablet for tablet a2358eac86914f3ba51bfc6c299838c3 (DEFAULT_TABLE table=default.Table_77f24f93527744b5b55d7c2243ef8774 [id=a52bbf692b2f4a6a986a547c8db00eab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.548415 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2358eac86914f3ba51bfc6c299838c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.552103 28874 consensus_queue.cc:1035] T 2fed2821d2644f75aff6e0ceb5944a37 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:31.557492 27214 tablet_service.cc:1468] Processing CreateTablet for tablet a2358eac86914f3ba51bfc6c299838c3 (DEFAULT_TABLE table=default.Table_77f24f93527744b5b55d7c2243ef8774 [id=a52bbf692b2f4a6a986a547c8db00eab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.561669 27353 tablet_service.cc:1468] Processing CreateTablet for tablet 7dc5214512af449bbc537f111a3d8dbc (DEFAULT_TABLE table=default.Table_8b2d83116755450387b770b591314010 [id=96837f9a9bae4d4b830036d3f309e153]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.561753 27352 tablet_service.cc:1468] Processing CreateTablet for tablet 65f93bcf7b674be1b464b68501238f93 (DEFAULT_TABLE table=default.Table_f6fb411b2d7949b4a424655ba7ff4921 [id=78d8c3e9d1fd41b7b119ef6ad6268b6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.564385 27353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dc5214512af449bbc537f111a3d8dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.568451 27320 leader_election.cc:304] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:31.569043 28916 raft_consensus.cc:2802] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:31.569167 28916 raft_consensus.cc:695] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:31.569460 28916 consensus_queue.cc:237] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.571233 27488 tablet_service.cc:1468] Processing CreateTablet for tablet fb5b3e7d6af7447c9fe1c3c27df34535 (DEFAULT_TABLE table=default.Table_7b59b780063640dd8ef72c72e5e5cf7a [id=c39e2e1b8f694dc5bf907dc80d050e7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.571776 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb5b3e7d6af7447c9fe1c3c27df34535. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.576535 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65f93bcf7b674be1b464b68501238f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.577807 27360 tablet_service.cc:1468] Processing CreateTablet for tablet fb5b3e7d6af7447c9fe1c3c27df34535 (DEFAULT_TABLE table=default.Table_7b59b780063640dd8ef72c72e5e5cf7a [id=c39e2e1b8f694dc5bf907dc80d050e7b]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:31.578061 27184 consensus_peers.cc:489] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 607c9a9cbc5540e5bd0fa436fcba564c. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:31.578233 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb5b3e7d6af7447c9fe1c3c27df34535. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.576637 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet e64c4a9179b6491ba92df75e0cc1bc50
I20250903 20:05:31.576849 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 3f303233e6c347289650c6cf83f8127f
I20250903 20:05:31.587196 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet e64c4a9179b6491ba92df75e0cc1bc50 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:31.577055 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 17514a2af39b402abbe8263d7c1abc7f
I20250903 20:05:31.587839 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet 3f303233e6c347289650c6cf83f8127f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:31.588043 27063 catalog_manager.cc:6652] ReplaceTablet: tablet 17514a2af39b402abbe8263d7c1abc7f deleted and replaced by tablet 38b912e24bf244759be8a6dc70cf9f52
I20250903 20:05:31.588167 27063 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:45936 (request call id 1155) took 1243 ms. Trace:
I20250903 20:05:31.588267 27063 rpcz_store.cc:276] 0903 20:05:30.344870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.140395 (+795525us) service_pool.cc:225] Handling call
0903 20:05:31.588136 (+447741us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:31.513768 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:31.513929 (+ 161us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:31.513939 (+ 10us) write_op.cc:435] Acquired schema lock
0903 20:05:31.513943 (+ 4us) tablet.cc:660] Decoding operations
0903 20:05:31.514106 (+ 163us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:31.514122 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:31.514127 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0903 20:05:31.514272 (+ 145us) tablet.cc:699] Row locks acquired
0903 20:05:31.514276 (+ 4us) write_op.cc:260] PREPARE: finished
0903 20:05:31.514335 (+ 59us) write_op.cc:270] Start()
0903 20:05:31.514376 (+ 41us) write_op.cc:276] Timestamp: P: 1756929931514282 usec, L: 0
0903 20:05:31.514380 (+ 4us) op_driver.cc:348] REPLICATION: starting
0903 20:05:31.514584 (+ 204us) log.cc:844] Serialized 625 byte log entry
0903 20:05:31.573968 (+ 59384us) op_driver.cc:464] REPLICATION: finished
0903 20:05:31.574944 (+ 976us) write_op.cc:301] APPLY: starting
0903 20:05:31.574955 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:31.574990 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:31.575011 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:31.575116 (+ 105us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:31.575161 (+ 45us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0903 20:05:31.575168 (+ 7us) write_op.cc:312] APPLY: finished
0903 20:05:31.575222 (+ 54us) log.cc:844] Serialized 29 byte log entry
0903 20:05:31.575261 (+ 39us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:31.575274 (+ 13us) write_op.cc:454] Released schema lock
0903 20:05:31.575279 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0903 20:05:31.575335 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:31.575338 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":945,"apply.run_cpu_time_us":385,"apply.run_wall_time_us":385,"num_ops":2,"prepare.queue_time_us":3920,"prepare.run_cpu_time_us":902,"prepare.run_wall_time_us":901,"replication_time_us":59571}]]}
I20250903 20:05:31.588798 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet 17514a2af39b402abbe8263d7c1abc7f with delete_type TABLET_DATA_DELETED (replaced by tablet 38b912e24bf244759be8a6dc70cf9f52) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:31.590224 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 17514a2af39b402abbe8263d7c1abc7f with delete_type TABLET_DATA_DELETED (replaced by tablet 38b912e24bf244759be8a6dc70cf9f52) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:31.591141 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet 17514a2af39b402abbe8263d7c1abc7f with delete_type TABLET_DATA_DELETED (replaced by tablet 38b912e24bf244759be8a6dc70cf9f52) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:31.592595 28808 consensus_queue.cc:1035] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:31.592825 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet 3f303233e6c347289650c6cf83f8127f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:31.586273 27060 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 9e98cd1d9bd246e182e2969d147e68bf (table default.Table_28a829a9988442049d50e6ae254e6274 [id=b27f08d56c3c43e6a09a465fdda4d3db]). Expected version 1 got 0
I20250903 20:05:31.595259 27060 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet a9f43781299e47a8aecc37a0b73d7a64 (table default.Table_73a4ac2afbad4654ac623cd581a45eb2 [id=f70d7ea93e154e6199f6766316353862]). Expected version 3 got 0
I20250903 20:05:31.596810 27350 tablet_service.cc:1515] Processing DeleteTablet for tablet e64c4a9179b6491ba92df75e0cc1bc50 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:31.601421 27385 raft_consensus.cc:3058] T 2fed2821d2644f75aff6e0ceb5944a37 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.601007 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet e64c4a9179b6491ba92df75e0cc1bc50 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:31.602648 27462 leader_election.cc:304] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:31.603004 28596 raft_consensus.cc:2802] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:31.603163 27462 leader_election.cc:304] T 512908bc4d6941ec81b145ee5795405f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:31.603235 28596 raft_consensus.cc:491] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:31.603374 28596 raft_consensus.cc:3058] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.604182 27348 tablet_service.cc:1515] Processing DeleteTablet for tablet 3f303233e6c347289650c6cf83f8127f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:31.606366 28710 raft_consensus.cc:2747] T 512908bc4d6941ec81b145ee5795405f P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:31.606648 28710 consensus_queue.cc:1035] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:31.606403 28761 consensus_queue.cc:1035] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250903 20:05:31.607919 27226 tablet_service.cc:1468] Processing CreateTablet for tablet fb5b3e7d6af7447c9fe1c3c27df34535 (DEFAULT_TABLE table=default.Table_7b59b780063640dd8ef72c72e5e5cf7a [id=c39e2e1b8f694dc5bf907dc80d050e7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.611322 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 40a7d9544350404d8811f734915e73fc (DEFAULT_TABLE table=default.Table_f9124043f4d84586858e6bdeb679df55 [id=8813121580cb489ea380347c230f5cd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.612555 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 7dc5214512af449bbc537f111a3d8dbc (DEFAULT_TABLE table=default.Table_8b2d83116755450387b770b591314010 [id=96837f9a9bae4d4b830036d3f309e153]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.613054 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dc5214512af449bbc537f111a3d8dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.613962 28596 raft_consensus.cc:513] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:31.615700 27184 consensus_peers.cc:489] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a72b85dfb34047528b2b28ff0d40f828. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:31.615760 28596 leader_election.cc:290] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:31.619304 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 40a7d9544350404d8811f734915e73fc (DEFAULT_TABLE table=default.Table_f9124043f4d84586858e6bdeb679df55 [id=8813121580cb489ea380347c230f5cd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.620461 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40a7d9544350404d8811f734915e73fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.621434 28596 raft_consensus.cc:491] T 726f4d920f3e47fea41b4e12c021c988 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:31.621021 27348 tablet_service.cc:1468] Processing CreateTablet for tablet 40a7d9544350404d8811f734915e73fc (DEFAULT_TABLE table=default.Table_f9124043f4d84586858e6bdeb679df55 [id=8813121580cb489ea380347c230f5cd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.624547 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40a7d9544350404d8811f734915e73fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.621749 28596 raft_consensus.cc:513] T 726f4d920f3e47fea41b4e12c021c988 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.626334 28596 leader_election.cc:290] T 726f4d920f3e47fea41b4e12c021c988 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:31.632692 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 04094bb7dd324b53a865980c53d4a17a (DEFAULT_TABLE table=default.Table_d7af80d1b68748c99d85f6914f57b251 [id=ae9ad46294b14f1a8e38f6ed9b2e8018]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.634260 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 4978cd831361454d86b23c1a52ab1c3b (DEFAULT_TABLE table=default.Table_e997a321ad0c45dc9258399326f0830d [id=ebe67c98862d476da14013880fd849e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.634622 27495 tablet_service.cc:1468] Processing CreateTablet for tablet e9b5ab312f8744af99d557c0733bf71d (DEFAULT_TABLE table=default.Table_10a7df6626f24353ae6661cbbdf3efb0 [id=416ff29f66d14564826a6734dde1c604]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.635078 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9b5ab312f8744af99d557c0733bf71d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.635368 27224 tablet_service.cc:1468] Processing CreateTablet for tablet 3739ac27204c4eb18747a0ae0d9254eb (DEFAULT_TABLE table=default.Table_27ad52e9c3794a6f8b8959abe991f5f0 [id=941315b442724e98a077abc1f93bcd46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.635540 27216 tablet_service.cc:1468] Processing CreateTablet for tablet e9b5ab312f8744af99d557c0733bf71d (DEFAULT_TABLE table=default.Table_10a7df6626f24353ae6661cbbdf3efb0 [id=416ff29f66d14564826a6734dde1c604]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.635921 27499 tablet_service.cc:1468] Processing CreateTablet for tablet 3739ac27204c4eb18747a0ae0d9254eb (DEFAULT_TABLE table=default.Table_27ad52e9c3794a6f8b8959abe991f5f0 [id=941315b442724e98a077abc1f93bcd46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.638542 27212 tablet_service.cc:1468] Processing CreateTablet for tablet b4f37d3bfafc45a3877e52f7e760254b (DEFAULT_TABLE table=default.Table_490a5f122961450f90ac24f19af39dac [id=6f3f88926f944dafade7c0e0a2f93f51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.638806 27365 tablet_service.cc:1468] Processing CreateTablet for tablet e9b5ab312f8744af99d557c0733bf71d (DEFAULT_TABLE table=default.Table_10a7df6626f24353ae6661cbbdf3efb0 [id=416ff29f66d14564826a6734dde1c604]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.634622 27489 tablet_service.cc:1468] Processing CreateTablet for tablet b4f37d3bfafc45a3877e52f7e760254b (DEFAULT_TABLE table=default.Table_490a5f122961450f90ac24f19af39dac [id=6f3f88926f944dafade7c0e0a2f93f51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.639204 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9b5ab312f8744af99d557c0733bf71d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.639662 27350 tablet_service.cc:1468] Processing CreateTablet for tablet 3739ac27204c4eb18747a0ae0d9254eb (DEFAULT_TABLE table=default.Table_27ad52e9c3794a6f8b8959abe991f5f0 [id=941315b442724e98a077abc1f93bcd46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.640940 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3739ac27204c4eb18747a0ae0d9254eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.642616 27347 tablet_service.cc:1468] Processing CreateTablet for tablet b4f37d3bfafc45a3877e52f7e760254b (DEFAULT_TABLE table=default.Table_490a5f122961450f90ac24f19af39dac [id=6f3f88926f944dafade7c0e0a2f93f51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.643064 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4f37d3bfafc45a3877e52f7e760254b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.648118 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 4978cd831361454d86b23c1a52ab1c3b (DEFAULT_TABLE table=default.Table_e997a321ad0c45dc9258399326f0830d [id=ebe67c98862d476da14013880fd849e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.650817 27352 tablet_service.cc:1468] Processing CreateTablet for tablet 04094bb7dd324b53a865980c53d4a17a (DEFAULT_TABLE table=default.Table_d7af80d1b68748c99d85f6914f57b251 [id=ae9ad46294b14f1a8e38f6ed9b2e8018]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.652148 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4978cd831361454d86b23c1a52ab1c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.668043 27494 tablet_service.cc:1468] Processing CreateTablet for tablet 04094bb7dd324b53a865980c53d4a17a (DEFAULT_TABLE table=default.Table_d7af80d1b68748c99d85f6914f57b251 [id=ae9ad46294b14f1a8e38f6ed9b2e8018]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.671875 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 7fa6e21e891741df85497235d6eef27b (DEFAULT_TABLE table=default.Table_8ca1972ab99a4f4db9225a77b8d5ebc4 [id=f68a6d7dc2a3468aacaa1aa51e3acefd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.672044 27493 tablet_service.cc:1468] Processing CreateTablet for tablet 7fa6e21e891741df85497235d6eef27b (DEFAULT_TABLE table=default.Table_8ca1972ab99a4f4db9225a77b8d5ebc4 [id=f68a6d7dc2a3468aacaa1aa51e3acefd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.672742 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f27e2aa2d0bf4de48429a030146df3d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.673070 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3739ac27204c4eb18747a0ae0d9254eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.673262 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04094bb7dd324b53a865980c53d4a17a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.673506 27212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4f37d3bfafc45a3877e52f7e760254b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.673965 27223 tablet_service.cc:1468] Processing CreateTablet for tablet 7fa6e21e891741df85497235d6eef27b (DEFAULT_TABLE table=default.Table_8ca1972ab99a4f4db9225a77b8d5ebc4 [id=f68a6d7dc2a3468aacaa1aa51e3acefd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.674423 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5698b385d9c4cc6ab67354c457d2a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.674669 27223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fa6e21e891741df85497235d6eef27b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.675012 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb5b3e7d6af7447c9fe1c3c27df34535. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.677899 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4978cd831361454d86b23c1a52ab1c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.678686 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40a7d9544350404d8811f734915e73fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.681650 27216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9b5ab312f8744af99d557c0733bf71d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.681911 27229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c04adef0af14dfeb4305d730c8a78fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.682543 27228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70d1b4eed08c4165b23acd9dd920ba0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.685729 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2309a6166f24503ae1040dc2bdea54b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.689483 27219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dc5214512af449bbc537f111a3d8dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.691198 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e35ae50252734a099ba5ce2d04e49e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.692587 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65f93bcf7b674be1b464b68501238f93. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:31.693727 27184 consensus_peers.cc:489] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a223f75045345ae833f52bcc8f239f9. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:31.693873 28808 raft_consensus.cc:491] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:31.694010 28808 raft_consensus.cc:513] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.694691 28808 leader_election.cc:290] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:31.695523 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "567630e2a93645e58f9bf05ffff87b32" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:31.696316 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2358eac86914f3ba51bfc6c299838c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.695930 27518 raft_consensus.cc:2466] T 567630e2a93645e58f9bf05ffff87b32 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:31.700574 27183 leader_election.cc:304] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:31.700985 28808 raft_consensus.cc:2802] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:31.701172 28808 raft_consensus.cc:491] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:31.701892 28808 raft_consensus.cc:3058] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.703563 28870 consensus_queue.cc:1035] T 2fed2821d2644f75aff6e0ceb5944a37 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:31.704849 28808 raft_consensus.cc:513] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.706051 28808 leader_election.cc:290] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:31.706776 27508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "567630e2a93645e58f9bf05ffff87b32" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:31.707085 27508 raft_consensus.cc:3058] T 567630e2a93645e58f9bf05ffff87b32 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.709029 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3739ac27204c4eb18747a0ae0d9254eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.710134 27508 raft_consensus.cc:2466] T 567630e2a93645e58f9bf05ffff87b32 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:31.710449 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fa6e21e891741df85497235d6eef27b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.711278 27183 leader_election.cc:304] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:31.712091 28870 raft_consensus.cc:2802] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:31.712294 28870 raft_consensus.cc:695] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:31.712546 28870 consensus_queue.cc:237] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.715114 27489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4f37d3bfafc45a3877e52f7e760254b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.720129 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "567630e2a93645e58f9bf05ffff87b32" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:31.720402 27381 raft_consensus.cc:3058] T 567630e2a93645e58f9bf05ffff87b32 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.723569 27381 raft_consensus.cc:2466] T 567630e2a93645e58f9bf05ffff87b32 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:31.724236 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 4978cd831361454d86b23c1a52ab1c3b (DEFAULT_TABLE table=default.Table_e997a321ad0c45dc9258399326f0830d [id=ebe67c98862d476da14013880fd849e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:31.724978 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4978cd831361454d86b23c1a52ab1c3b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:31.726974 27184 consensus_peers.cc:489] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 466e501e424e46bc98e8bbc64f97005e. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:31.731525 27494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04094bb7dd324b53a865980c53d4a17a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.726119 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "567630e2a93645e58f9bf05ffff87b32" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:31.740471 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04094bb7dd324b53a865980c53d4a17a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.740471 27385 raft_consensus.cc:2374] T 567630e2a93645e58f9bf05ffff87b32 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1. Re-sending same reply.
I20250903 20:05:31.743325 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fa6e21e891741df85497235d6eef27b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:31.745579 27695 tablet_bootstrap.cc:492] T a0f79831866b40c49c5b27ce2014c3f6 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:31.746662 27693 tablet_bootstrap.cc:654] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:31.751470 27695 ts_tablet_manager.cc:1397] T a0f79831866b40c49c5b27ce2014c3f6 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.255s user 0.003s sys 0.002s
I20250903 20:05:31.752472 27695 raft_consensus.cc:357] T a0f79831866b40c49c5b27ce2014c3f6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.752777 27695 raft_consensus.cc:383] T a0f79831866b40c49c5b27ce2014c3f6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:31.752871 28874 raft_consensus.cc:491] T b7a6d5d8a97a4769a81ec7224d591f8c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:31.753005 28874 raft_consensus.cc:513] T b7a6d5d8a97a4769a81ec7224d591f8c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.753602 28874 leader_election.cc:290] T b7a6d5d8a97a4769a81ec7224d591f8c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:31.757140 27684 ts_tablet_manager.cc:1918] T 53881ad30bc843839008ace4012a3f70 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:31.762924 27384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b7a6d5d8a97a4769a81ec7224d591f8c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:31.766507 27384 raft_consensus.cc:2391] T b7a6d5d8a97a4769a81ec7224d591f8c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89f8f0ee68b940db9ce250c0604f934a in current term 1: Already voted for candidate e72ef1dbcc5143cf9f04a808f101c80f in this term.
I20250903 20:05:31.766938 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_edc3f0ba095c4b65b0bb1f12d3f231c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:31.767648 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_edc3f0ba095c4b65b0bb1f12d3f231c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:31.769006 27687 ts_tablet_manager.cc:1918] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250903 20:05:31.769230 27687 log.cc:1199] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/e8bc99de4ffe43d58351d170b7c61d8a
I20250903 20:05:31.769687 27687 ts_tablet_manager.cc:1939] T e8bc99de4ffe43d58351d170b7c61d8a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:31.769744 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_dbeec832140d416aa6667b1958c1abf2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:31.770356 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbeec832140d416aa6667b1958c1abf2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:31.770587 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 965) took 1041 ms. Trace:
I20250903 20:05:31.770664 27687 rpcz_store.cc:276] 0903 20:05:30.729054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.729110 (+ 56us) service_pool.cc:225] Handling call
0903 20:05:31.483339 (+754229us) ts_tablet_manager.cc:1145] Deleting tablet e8bc99de4ffe43d58351d170b7c61d8a
0903 20:05:31.483349 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.515207 (+ 31858us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.768893 (+253686us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.770566 (+ 1673us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":754017}
I20250903 20:05:31.770692 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_2f3c5f77e6bd4a0ebf0ac41e63589e79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:31.771153 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f3c5f77e6bd4a0ebf0ac41e63589e79 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:31.771329 27687 tablet_replica.cc:331] T f2629f46704e4a0fa7922533928ca2ed P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:31.771272 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_db15e7b90ba74a349a32034316f18378"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:31.771502 27687 raft_consensus.cc:2241] T f2629f46704e4a0fa7922533928ca2ed P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:31.771642 27687 raft_consensus.cc:2270] T f2629f46704e4a0fa7922533928ca2ed P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:31.771733 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db15e7b90ba74a349a32034316f18378 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:31.772140 27687 ts_tablet_manager.cc:1905] T f2629f46704e4a0fa7922533928ca2ed P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:31.778023 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet e8bc99de4ffe43d58351d170b7c61d8a (table default.Table_10a7df6626f24353ae6661cbbdf3efb0 [id=416ff29f66d14564826a6734dde1c604]) successfully deleted
W20250903 20:05:31.778370 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 90776f6369744cec8ce380b7695c44f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90776f6369744cec8ce380b7695c44f1 already in progress: creating tablet
W20250903 20:05:31.778559 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 96c9875830b246b3a9117536254cbc4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96c9875830b246b3a9117536254cbc4e already in progress: creating tablet
W20250903 20:05:31.778723 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet f9709dfd18654c5bb071e39da7c8889f because the tablet was not found. No further retry: Not found: Tablet not found: f9709dfd18654c5bb071e39da7c8889f
I20250903 20:05:31.752873 27695 raft_consensus.cc:738] T a0f79831866b40c49c5b27ce2014c3f6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:31.783325 27695 consensus_queue.cc:260] T a0f79831866b40c49c5b27ce2014c3f6 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.784057 27695 ts_tablet_manager.cc:1428] T a0f79831866b40c49c5b27ce2014c3f6 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.032s user 0.001s sys 0.001s
I20250903 20:05:31.795609 27693 tablet_bootstrap.cc:492] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:31.795776 27693 ts_tablet_manager.cc:1397] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.282s user 0.003s sys 0.001s
I20250903 20:05:31.796533 27693 raft_consensus.cc:357] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.796702 27693 raft_consensus.cc:383] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:31.796752 27693 raft_consensus.cc:738] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:31.796954 27693 consensus_queue.cc:260] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.761426 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b7a6d5d8a97a4769a81ec7224d591f8c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:31.763548 27462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 159865us
I20250903 20:05:31.798009 28710 consensus_queue.cc:1035] T 9e98cd1d9bd246e182e2969d147e68bf P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:31.798169 27183 leader_election.cc:304] T b7a6d5d8a97a4769a81ec7224d591f8c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:31.798626 28947 raft_consensus.cc:3058] T b7a6d5d8a97a4769a81ec7224d591f8c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.798841 27462 leader_election.cc:304] T 67ca1dcf6fc7435db555392235f9900f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:31.799189 28710 raft_consensus.cc:2802] T 67ca1dcf6fc7435db555392235f9900f P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:31.799331 28710 raft_consensus.cc:695] T 67ca1dcf6fc7435db555392235f9900f P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:31.799567 28710 consensus_queue.cc:237] T 67ca1dcf6fc7435db555392235f9900f P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:31.801257 28947 raft_consensus.cc:2747] T b7a6d5d8a97a4769a81ec7224d591f8c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:31.802345 27684 log.cc:1199] T 53881ad30bc843839008ace4012a3f70 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/53881ad30bc843839008ace4012a3f70
I20250903 20:05:31.802615 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0f79831866b40c49c5b27ce2014c3f6" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:31.802793 27239 raft_consensus.cc:3058] T a0f79831866b40c49c5b27ce2014c3f6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.802871 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "726f4d920f3e47fea41b4e12c021c988" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:31.803046 27684 ts_tablet_manager.cc:1939] T 53881ad30bc843839008ace4012a3f70 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:31.803109 27381 raft_consensus.cc:2391] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 89f8f0ee68b940db9ce250c0604f934a in this term.
I20250903 20:05:31.803853 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 902) took 2962 ms. Trace:
I20250903 20:05:31.803969 27684 rpcz_store.cc:276] 0903 20:05:28.841232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.841647 (+ 415us) service_pool.cc:225] Handling call
0903 20:05:31.506710 (+2665063us) ts_tablet_manager.cc:1145] Deleting tablet 53881ad30bc843839008ace4012a3f70
0903 20:05:31.506718 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.753712 (+246994us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.756931 (+ 3219us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.803817 (+ 46886us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2664874}
I20250903 20:05:31.804466 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 53881ad30bc843839008ace4012a3f70 (table default.Table_2b6469f261a04e5d9cb69bf836d97734 [id=cedbbfab7d0e41369cafd9ce516f2dba]) successfully deleted
I20250903 20:05:31.805344 27239 raft_consensus.cc:2466] T a0f79831866b40c49c5b27ce2014c3f6 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:31.805840 27684 tablet_replica.cc:331] T 1bad3c509460422396a17a3c4ae3d1e8 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:31.806070 27684 raft_consensus.cc:2241] T 1bad3c509460422396a17a3c4ae3d1e8 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:31.806334 27684 raft_consensus.cc:2270] T 1bad3c509460422396a17a3c4ae3d1e8 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:31.806991 27684 ts_tablet_manager.cc:1905] T 1bad3c509460422396a17a3c4ae3d1e8 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:31.807539 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "726f4d920f3e47fea41b4e12c021c988" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:31.819590 27462 leader_election.cc:304] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:31.820672 27695 tablet_bootstrap.cc:492] T 02b3e262dd7841dfbb0520377eb94743 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:31.821497 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0f79831866b40c49c5b27ce2014c3f6" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:31.821851 27385 raft_consensus.cc:3058] T a0f79831866b40c49c5b27ce2014c3f6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.822744 27695 tablet_bootstrap.cc:654] T 02b3e262dd7841dfbb0520377eb94743 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:31.824916 28761 raft_consensus.cc:2802] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:31.826370 29035 consensus_queue.cc:1035] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:31.835330 27462 leader_election.cc:304] T 726f4d920f3e47fea41b4e12c021c988 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:31.836115 29045 raft_consensus.cc:3058] T 726f4d920f3e47fea41b4e12c021c988 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.838752 29045 raft_consensus.cc:2747] T 726f4d920f3e47fea41b4e12c021c988 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:31.841403 27692 tablet_bootstrap.cc:492] T 120ca61b3da742dc9a2efa483ead7e97 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:31.841605 27692 ts_tablet_manager.cc:1397] T 120ca61b3da742dc9a2efa483ead7e97 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.339s user 0.006s sys 0.003s
I20250903 20:05:31.841817 27385 raft_consensus.cc:2466] T a0f79831866b40c49c5b27ce2014c3f6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:31.843638 27063 master_service.cc:948] ReplaceTablet: received request to replace tablet 2478fe89414b4845b0ba377c1cde98b7 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:31.847800 27692 raft_consensus.cc:357] T 120ca61b3da742dc9a2efa483ead7e97 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.848058 27692 raft_consensus.cc:383] T 120ca61b3da742dc9a2efa483ead7e97 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:31.848131 27692 raft_consensus.cc:738] T 120ca61b3da742dc9a2efa483ead7e97 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:31.848382 27692 consensus_queue.cc:260] T 120ca61b3da742dc9a2efa483ead7e97 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.848960 27692 ts_tablet_manager.cc:1428] T 120ca61b3da742dc9a2efa483ead7e97 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250903 20:05:31.849273 27692 tablet_bootstrap.cc:492] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:31.850114 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_d6bdbaa18dba41bc997a529a1d2d322d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:31.850818 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6bdbaa18dba41bc997a529a1d2d322d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:31.850996 27692 tablet_bootstrap.cc:654] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:31.852543 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_f573c7cc8d7240c0bf60622441b33a69"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:31.853113 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f573c7cc8d7240c0bf60622441b33a69 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:31.854472 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e1654a60dac84f5f8f1d80d5f7bd5b82" } modify_external_catalogs: true
I20250903 20:05:31.854712 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e1654a60dac84f5f8f1d80d5f7bd5b82" } modify_external_catalogs: true
I20250903 20:05:31.858916 29045 consensus_queue.cc:1035] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:31.859117 27692 tablet_bootstrap.cc:492] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:31.859261 27692 ts_tablet_manager.cc:1397] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20250903 20:05:31.860399 27687 ts_tablet_manager.cc:1918] T f2629f46704e4a0fa7922533928ca2ed P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:31.862176 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_fa60556c9a9b433a9eff6f68ee824909" } new_table_name: "default.Table_b0d0c6220ab44b91b2334baa4908574a" modify_external_catalogs: true
I20250903 20:05:31.862322 27687 log.cc:1199] T f2629f46704e4a0fa7922533928ca2ed P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/f2629f46704e4a0fa7922533928ca2ed
I20250903 20:05:31.862958 27687 ts_tablet_manager.cc:1939] T f2629f46704e4a0fa7922533928ca2ed P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:31.871654 27684 ts_tablet_manager.cc:1918] T 1bad3c509460422396a17a3c4ae3d1e8 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:31.872500 27684 log.cc:1199] T 1bad3c509460422396a17a3c4ae3d1e8 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/1bad3c509460422396a17a3c4ae3d1e8
I20250903 20:05:31.873145 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet b5bff419ed31453e80ceec23ea05caad
I20250903 20:05:31.874022 28770 raft_consensus.cc:491] T a253e3399622409e92299069b19d3698 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:31.874154 28567 raft_consensus.cc:491] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:31.874325 28567 raft_consensus.cc:513] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.874660 27350 tablet_service.cc:1515] Processing DeleteTablet for tablet b5bff419ed31453e80ceec23ea05caad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:31.874954 28567 leader_election.cc:290] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:31.874156 28770 raft_consensus.cc:513] T a253e3399622409e92299069b19d3698 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.877130 28770 leader_election.cc:290] T a253e3399622409e92299069b19d3698 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:31.880990 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_397211088a974a09bd7f031d5c7ca1cc" } modify_external_catalogs: true
I20250903 20:05:31.881204 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_397211088a974a09bd7f031d5c7ca1cc" } modify_external_catalogs: true
I20250903 20:05:31.883491 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_331a93e34699477294aedde756d58a8f" } new_table_name: "default.Table_741f03359c5343268a5ebe2c70552ed2" modify_external_catalogs: true
W20250903 20:05:31.885596 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ea2d1d9e7e724aa08dfec812dad9589e consensus state has no leader
I20250903 20:05:31.886310 27062 master_service.cc:948] ReplaceTablet: received request to replace tablet 6788b1683f984949816edfc5b68de8db from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:31.887591 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_0dbf85eafa35406cba9418cfa377f180"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:31.888482 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0dbf85eafa35406cba9418cfa377f180 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:31.889540 28947 raft_consensus.cc:491] T 541278ca9ffb4e41b341139c2faa337f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:31.889745 28947 raft_consensus.cc:513] T 541278ca9ffb4e41b341139c2faa337f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.890012 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a253e3399622409e92299069b19d3698" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:31.890863 28947 leader_election.cc:290] T 541278ca9ffb4e41b341139c2faa337f P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
W20250903 20:05:31.891820 27321 leader_election.cc:343] T a253e3399622409e92299069b19d3698 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:31.892155 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8fff36d5a80e45af8a256c3908555e08" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:31.892627 27321 leader_election.cc:343] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:31.898801 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet c8a293c6fa034e9fb29682439cb3bcc2
I20250903 20:05:31.899749 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet c8a293c6fa034e9fb29682439cb3bcc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:31.900708 27693 ts_tablet_manager.cc:1428] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.105s user 0.000s sys 0.002s
I20250903 20:05:31.901060 27693 tablet_bootstrap.cc:492] T 02b3e262dd7841dfbb0520377eb94743 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:31.902989 27057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_53e3669346c9467e8e27c3a33c6e1660" } new_table_name: "default.Table_f632fa586b124749a0763c5c17dfb9b3" modify_external_catalogs: true
I20250903 20:05:31.903719 27693 tablet_bootstrap.cc:654] T 02b3e262dd7841dfbb0520377eb94743 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:31.904839 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet f2629f46704e4a0fa7922533928ca2ed (table default.Table_ae457e422e0c4314aeaecd7755413366 [id=2051ecb140f24088911aad60fdb9082b]) successfully deleted
I20250903 20:05:31.905241 27687 tablet_replica.cc:331] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:31.905444 27687 raft_consensus.cc:2241] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:31.905766 27687 raft_consensus.cc:2270] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:31.906468 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cab87148f7b84222915bf8eefc78a5f1 consensus state has no leader
I20250903 20:05:31.906956 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_74cf241e5c694a219f63258fb4b582c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:31.907714 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74cf241e5c694a219f63258fb4b582c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:31.907866 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "541278ca9ffb4e41b341139c2faa337f" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:31.910527 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f1f4f57d96f04f0caff06d9a642f5c46" } new_table_name: "default.Table_b44b0ef404e14710b769f7df4cd92ddc" modify_external_catalogs: true
I20250903 20:05:31.750094 27685 ts_tablet_manager.cc:1918] T a4c582b441f6467c99ba0b090abcf4cc P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:31.911809 27685 log.cc:1199] T a4c582b441f6467c99ba0b090abcf4cc P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/a4c582b441f6467c99ba0b090abcf4cc
I20250903 20:05:31.912541 27687 ts_tablet_manager.cc:1905] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:31.912672 27685 ts_tablet_manager.cc:1939] T a4c582b441f6467c99ba0b090abcf4cc P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:31.912993 28947 consensus_queue.cc:1035] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:31.914026 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet c8a293c6fa034e9fb29682439cb3bcc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:31.914712 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet b5bff419ed31453e80ceec23ea05caad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:31.916433 27050 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 4aea507c5e214a57a80bd4de1a6ccd67 (table default.Table_af252d2e4c2f40469f1e0b5cd3fa20ff [id=cc99c3326ca04692af63ce855bbb1caf], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4aea507c5e214a57a80bd4de1a6ccd67 consensus state has no leader
W20250903 20:05:31.918972 27320 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 72547.7us
W20250903 20:05:31.921207 27184 consensus_peers.cc:489] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e2f98f760cf4df4bc2a10530e2157c6. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:31.875626 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet b5bff419ed31453e80ceec23ea05caad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:31.923619 27684 ts_tablet_manager.cc:1939] T 1bad3c509460422396a17a3c4ae3d1e8 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:31.924502 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 903) took 3083 ms. Trace:
I20250903 20:05:31.924665 27684 rpcz_store.cc:276] 0903 20:05:28.841354 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.841868 (+ 514us) service_pool.cc:225] Handling call
0903 20:05:31.805744 (+2963876us) ts_tablet_manager.cc:1145] Deleting tablet 1bad3c509460422396a17a3c4ae3d1e8
0903 20:05:31.805755 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.830345 (+ 24590us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.871487 (+ 41142us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.924447 (+ 52960us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2963720}
I20250903 20:05:31.924880 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 783) took 5717 ms. Trace:
I20250903 20:05:31.925135 27684 tablet_replica.cc:331] T 1fbe54dc40dd47daa109ceeb07546b0a P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:31.925309 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 1bad3c509460422396a17a3c4ae3d1e8 (table default.Table_1d4af1f6213b4b4db3bfaee9e1ede2d0 [id=41f9673d6e604f45a8def10948ce21ac]) successfully deleted
I20250903 20:05:31.925364 27684 raft_consensus.cc:2241] T 1fbe54dc40dd47daa109ceeb07546b0a P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:31.925546 27684 raft_consensus.cc:2270] T 1fbe54dc40dd47daa109ceeb07546b0a P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:31.925653 27685 rpcz_store.cc:276] 0903 20:05:26.207802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.207867 (+ 65us) service_pool.cc:225] Handling call
0903 20:05:31.285327 (+5077460us) ts_tablet_manager.cc:1145] Deleting tablet a4c582b441f6467c99ba0b090abcf4cc
0903 20:05:31.285335 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.506383 (+221048us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.749970 (+243587us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.924856 (+174886us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5077253}
I20250903 20:05:31.897662 27518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "541278ca9ffb4e41b341139c2faa337f" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:31.901233 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet c8a293c6fa034e9fb29682439cb3bcc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:31.926627 27518 raft_consensus.cc:2391] T 541278ca9ffb4e41b341139c2faa337f P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89f8f0ee68b940db9ce250c0604f934a in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
I20250903 20:05:31.926931 27684 ts_tablet_manager.cc:1905] T 1fbe54dc40dd47daa109ceeb07546b0a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:31.931627 27183 leader_election.cc:304] T 541278ca9ffb4e41b341139c2faa337f P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:31.932190 28870 raft_consensus.cc:3058] T 541278ca9ffb4e41b341139c2faa337f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.934157 27685 tablet_replica.cc:331] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:31.934412 27685 raft_consensus.cc:2241] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:31.934777 27685 raft_consensus.cc:2270] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:31.935194 28870 raft_consensus.cc:2747] T 541278ca9ffb4e41b341139c2faa337f P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:31.906489 27692 raft_consensus.cc:357] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.936905 27692 raft_consensus.cc:383] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:31.937312 27692 raft_consensus.cc:738] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:31.937821 27692 consensus_queue.cc:260] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.938305 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b8435bf217c34217bbea1bad9cb30edb" } new_table_name: "default.Table_c088a192c5f74cd6a9635db0dfa8227c" modify_external_catalogs: true
W20250903 20:05:31.938865 27057 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a785adcd0bd943fc97b201b6d29073f4 consensus state has no leader
I20250903 20:05:31.938939 27692 ts_tablet_manager.cc:1428] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.080s user 0.001s sys 0.001s
I20250903 20:05:31.939256 27692 tablet_bootstrap.cc:492] T 7345305820be4d32a3ceb227f045f009 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:31.939249 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_05b4f4b3f80c45df89e4f38ce76f42a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:31.939880 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_05b4f4b3f80c45df89e4f38ce76f42a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:31.940965 27692 tablet_bootstrap.cc:654] T 7345305820be4d32a3ceb227f045f009 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:31.921114 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8fff36d5a80e45af8a256c3908555e08" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:31.921552 27517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a253e3399622409e92299069b19d3698" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:31.941139 27516 raft_consensus.cc:2466] T 8fff36d5a80e45af8a256c3908555e08 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:31.919364 28761 raft_consensus.cc:695] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:31.941614 28761 consensus_queue.cc:237] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.943281 28874 raft_consensus.cc:491] T 33fb6b357bda43e7abeaed5634183ef6 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51578fb7bceb4e119b3ab1e58c4f5528)
I20250903 20:05:31.943420 28874 raft_consensus.cc:513] T 33fb6b357bda43e7abeaed5634183ef6 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:31.943912 27695 tablet_bootstrap.cc:492] T 02b3e262dd7841dfbb0520377eb94743 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:31.943969 28874 leader_election.cc:290] T 33fb6b357bda43e7abeaed5634183ef6 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:31.944136 27695 ts_tablet_manager.cc:1397] T 02b3e262dd7841dfbb0520377eb94743 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.124s user 0.002s sys 0.011s
I20250903 20:05:31.944919 27695 raft_consensus.cc:357] T 02b3e262dd7841dfbb0520377eb94743 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.945526 27695 raft_consensus.cc:383] T 02b3e262dd7841dfbb0520377eb94743 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:31.945641 27695 raft_consensus.cc:738] T 02b3e262dd7841dfbb0520377eb94743 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
W20250903 20:05:31.945683 27183 leader_election.cc:343] T 33fb6b357bda43e7abeaed5634183ef6 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 33fb6b357bda43e7abeaed5634183ef6
I20250903 20:05:31.945931 27695 consensus_queue.cc:260] T 02b3e262dd7841dfbb0520377eb94743 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.946605 27695 ts_tablet_manager.cc:1428] T 02b3e262dd7841dfbb0520377eb94743 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:31.948055 29055 tablet.cc:1722] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250903 20:05:31.949019 27695 tablet_bootstrap.cc:492] T fc634116a78f45dd833034ddda90e9bf P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:31.949481 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_bbc905e2f2804b9ead3a69b83cf50fde"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:31.950125 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbc905e2f2804b9ead3a69b83cf50fde in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:31.954783 27687 ts_tablet_manager.cc:1918] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250903 20:05:31.955055 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e15da14bcbdc4c80820795f46ef89b75 consensus state has no leader
I20250903 20:05:31.955482 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a6683ce4dd5d4470a5b08bddd66ee17f" } modify_external_catalogs: true
I20250903 20:05:31.955706 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a6683ce4dd5d4470a5b08bddd66ee17f" } modify_external_catalogs: true
I20250903 20:05:31.959707 29035 consensus_queue.cc:1035] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:31.947250 29056 tablet.cc:1722] T a9f43781299e47a8aecc37a0b73d7a64 P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250903 20:05:31.966576 27687 log.cc:1199] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/4af8c03c9d834405b4e25d35441db961
I20250903 20:05:31.967516 27687 ts_tablet_manager.cc:1939] T 4af8c03c9d834405b4e25d35441db961 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
W20250903 20:05:31.968158 27320 leader_election.cc:343] T a253e3399622409e92299069b19d3698 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:31.968325 27320 leader_election.cc:304] T a253e3399622409e92299069b19d3698 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:31.968622 28916 raft_consensus.cc:2747] T a253e3399622409e92299069b19d3698 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:31.968825 27320 leader_election.cc:304] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:31.969193 28916 raft_consensus.cc:2802] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:31.969334 28916 raft_consensus.cc:491] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:31.969497 28916 raft_consensus.cc:3058] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.969633 29044 raft_consensus.cc:491] T 976322efa9464ed7a2175d3e3de51dab P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:31.969815 29044 raft_consensus.cc:513] T 976322efa9464ed7a2175d3e3de51dab P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.970570 29044 leader_election.cc:290] T 976322efa9464ed7a2175d3e3de51dab P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:31.970906 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f2505e03caaa40c4b17d2e60b0196843" } modify_external_catalogs: true
I20250903 20:05:31.971159 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f2505e03caaa40c4b17d2e60b0196843" } modify_external_catalogs: true
I20250903 20:05:31.972741 28916 raft_consensus.cc:513] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:31.974166 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8fff36d5a80e45af8a256c3908555e08" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
W20250903 20:05:31.974709 27321 leader_election.cc:343] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:31.975006 28916 leader_election.cc:290] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:31.976475 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 4af8c03c9d834405b4e25d35441db961 (table default.Table_9eac2fd82a524aae943ebd76428f95eb [id=058a64daaf104dfdace7d05d52388b79]) successfully deleted
I20250903 20:05:31.978904 27687 tablet_replica.cc:331] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:31.980027 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "976322efa9464ed7a2175d3e3de51dab" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:31.983327 27376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "976322efa9464ed7a2175d3e3de51dab" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:31.983661 27376 raft_consensus.cc:2466] T 976322efa9464ed7a2175d3e3de51dab P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:31.984867 27685 ts_tablet_manager.cc:1905] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:31.988508 27184 leader_election.cc:343] T 33fb6b357bda43e7abeaed5634183ef6 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 33fb6b357bda43e7abeaed5634183ef6
I20250903 20:05:31.988744 27184 leader_election.cc:304] T 33fb6b357bda43e7abeaed5634183ef6 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:31.990454 27462 leader_election.cc:304] T 976322efa9464ed7a2175d3e3de51dab P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:31.990847 28870 raft_consensus.cc:2747] T 33fb6b357bda43e7abeaed5634183ef6 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:31.991179 27517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8fff36d5a80e45af8a256c3908555e08" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:31.991392 27517 raft_consensus.cc:3058] T 8fff36d5a80e45af8a256c3908555e08 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:31.994145 27517 raft_consensus.cc:2466] T 8fff36d5a80e45af8a256c3908555e08 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:31.995501 27684 ts_tablet_manager.cc:1918] T 1fbe54dc40dd47daa109ceeb07546b0a P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:31.999058 27684 log.cc:1199] T 1fbe54dc40dd47daa109ceeb07546b0a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/1fbe54dc40dd47daa109ceeb07546b0a
I20250903 20:05:31.999045 27057 master_service.cc:948] ReplaceTablet: received request to replace tablet 6d21f38759d743af972635f8b5b8d8d2 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:32.000089 27684 ts_tablet_manager.cc:1939] T 1fbe54dc40dd47daa109ceeb07546b0a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:32.009460 29046 raft_consensus.cc:2802] T 976322efa9464ed7a2175d3e3de51dab P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250903 20:05:32.009598 27462 leader_election.cc:343] T 976322efa9464ed7a2175d3e3de51dab P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:32.009613 29046 raft_consensus.cc:491] T 976322efa9464ed7a2175d3e3de51dab P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:32.009920 29046 raft_consensus.cc:3058] T 976322efa9464ed7a2175d3e3de51dab P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:32.010143 27320 leader_election.cc:304] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:32.010605 28916 raft_consensus.cc:2802] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:32.010931 27695 tablet_bootstrap.cc:654] T fc634116a78f45dd833034ddda90e9bf P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:32.011032 28916 raft_consensus.cc:695] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:32.011308 28916 consensus_queue.cc:237] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.012101 29062 tablet.cc:1722] T a9f43781299e47a8aecc37a0b73d7a64 P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250903 20:05:32.013085 29046 raft_consensus.cc:513] T 976322efa9464ed7a2175d3e3de51dab P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.013816 29046 leader_election.cc:290] T 976322efa9464ed7a2175d3e3de51dab P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:32.015164 27687 raft_consensus.cc:2241] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:32.015548 27687 pending_rounds.cc:70] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528: Trying to abort 1 pending ops.
I20250903 20:05:32.015664 27687 pending_rounds.cc:77] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196384996841996288 op_type: NO_OP noop_request { }
I20250903 20:05:32.015898 27687 raft_consensus.cc:2887] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:32.016067 27687 raft_consensus.cc:2270] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.017050 27687 ts_tablet_manager.cc:1905] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:32.019177 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "976322efa9464ed7a2175d3e3de51dab" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:32.020093 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "976322efa9464ed7a2175d3e3de51dab" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:32.020654 27381 raft_consensus.cc:3058] T 976322efa9464ed7a2175d3e3de51dab P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:32.020747 27184 consensus_peers.cc:489] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20250903 20:05:32.021472 27462 leader_election.cc:343] T 976322efa9464ed7a2175d3e3de51dab P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:32.027845 26885 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskH7eH49/build/asan/bin/kudu pid 28184: Timed out: CheckLeaks RPC to 127.26.65.125:44059 timed out after 1.000s (SENT)
I20250903 20:05:32.028169 26885 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskH7eH49/build/asan/bin/kudu with pid 28184
I20250903 20:05:32.029306 27693 tablet_bootstrap.cc:492] T 02b3e262dd7841dfbb0520377eb94743 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:32.029464 27693 ts_tablet_manager.cc:1397] T 02b3e262dd7841dfbb0520377eb94743 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.129s user 0.000s sys 0.004s
I20250903 20:05:32.029927 27692 tablet_bootstrap.cc:492] T 7345305820be4d32a3ceb227f045f009 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:32.030129 27692 ts_tablet_manager.cc:1397] T 7345305820be4d32a3ceb227f045f009 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.091s user 0.001s sys 0.002s
I20250903 20:05:32.034053 27693 raft_consensus.cc:357] T 02b3e262dd7841dfbb0520377eb94743 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.034327 27693 raft_consensus.cc:383] T 02b3e262dd7841dfbb0520377eb94743 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:32.034415 27693 raft_consensus.cc:738] T 02b3e262dd7841dfbb0520377eb94743 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:32.034701 27693 consensus_queue.cc:260] T 02b3e262dd7841dfbb0520377eb94743 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.036824 27381 raft_consensus.cc:2466] T 976322efa9464ed7a2175d3e3de51dab P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:32.034117 27692 raft_consensus.cc:357] T 7345305820be4d32a3ceb227f045f009 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.037288 27692 raft_consensus.cc:383] T 7345305820be4d32a3ceb227f045f009 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:32.039156 27695 tablet_bootstrap.cc:492] T fc634116a78f45dd833034ddda90e9bf P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:32.039310 27695 ts_tablet_manager.cc:1397] T fc634116a78f45dd833034ddda90e9bf P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.090s user 0.004s sys 0.002s
I20250903 20:05:32.040071 27695 raft_consensus.cc:357] T fc634116a78f45dd833034ddda90e9bf P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:32.040220 27695 raft_consensus.cc:383] T fc634116a78f45dd833034ddda90e9bf P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:32.040263 27695 raft_consensus.cc:738] T fc634116a78f45dd833034ddda90e9bf P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:32.040457 27695 consensus_queue.cc:260] T fc634116a78f45dd833034ddda90e9bf P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:32.040951 27695 ts_tablet_manager.cc:1428] T fc634116a78f45dd833034ddda90e9bf P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:32.041235 27695 tablet_bootstrap.cc:492] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:32.043395 27695 tablet_bootstrap.cc:654] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:32.045524 27692 raft_consensus.cc:738] T 7345305820be4d32a3ceb227f045f009 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:32.045995 27692 consensus_queue.cc:260] T 7345305820be4d32a3ceb227f045f009 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.047793 29056 tablet.cc:1722] T 912eee5b48ef467bbd6bbe34f8aa390c P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250903 20:05:32.048907 27692 ts_tablet_manager.cc:1428] T 7345305820be4d32a3ceb227f045f009 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20250903 20:05:32.049280 27692 tablet_bootstrap.cc:492] T d1c96296a579440c887ba6c2a93f7204 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:32.050805 28567 raft_consensus.cc:491] T 02b3e262dd7841dfbb0520377eb94743 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:32.050989 28567 raft_consensus.cc:513] T 02b3e262dd7841dfbb0520377eb94743 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.051355 28874 consensus_queue.cc:1035] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:32.051671 28567 leader_election.cc:290] T 02b3e262dd7841dfbb0520377eb94743 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:32.037602 27693 ts_tablet_manager.cc:1428] T 02b3e262dd7841dfbb0520377eb94743 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250903 20:05:32.039353 29053 consensus_queue.cc:1035] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:32.047530 29055 tablet.cc:1722] T 912eee5b48ef467bbd6bbe34f8aa390c P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250903 20:05:32.052352 27693 tablet_bootstrap.cc:492] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:32.053938 27693 tablet_bootstrap.cc:654] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:32.056216 27692 tablet_bootstrap.cc:654] T d1c96296a579440c887ba6c2a93f7204 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:32.057181 27059 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 67ca1dcf6fc7435db555392235f9900f (table default.Table_d3945d5a36d84041b35394333dc704f9 [id=5568f788d1414c83b4359b5adf28632a]). Expected version 2 got 0
I20250903 20:05:32.057502 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 2478fe89414b4845b0ba377c1cde98b7
I20250903 20:05:32.057601 27059 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 9e98cd1d9bd246e182e2969d147e68bf (table default.Table_28a829a9988442049d50e6ae254e6274 [id=b27f08d56c3c43e6a09a465fdda4d3db]). Expected version 1 got 0
I20250903 20:05:32.057977 27063 catalog_manager.cc:6652] ReplaceTablet: tablet 2478fe89414b4845b0ba377c1cde98b7 deleted and replaced by tablet 361156ca43a444b1b07e1d1fab48c234
I20250903 20:05:32.058095 27059 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet a9f43781299e47a8aecc37a0b73d7a64 (table default.Table_73a4ac2afbad4654ac623cd581a45eb2 [id=f70d7ea93e154e6199f6766316353862]). Expected version 3 got 0
I20250903 20:05:32.058668 27059 catalog_manager.cc:5582] T 2fed2821d2644f75aff6e0ceb5944a37 P 89f8f0ee68b940db9ce250c0604f934a reported cstate change: term changed from 0 to 1, leader changed from <none> to 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65). New cstate: current_term: 1 leader_uuid: "89f8f0ee68b940db9ce250c0604f934a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: HEALTHY } } }
I20250903 20:05:32.058916 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 2478fe89414b4845b0ba377c1cde98b7 with delete_type TABLET_DATA_DELETED (replaced by tablet 361156ca43a444b1b07e1d1fab48c234) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.057240 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 4e34d6d5d68e4395aed78821ea077c1b
I20250903 20:05:32.059839 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 1ad2ca3e72434b69ad48e533da72e8fd
I20250903 20:05:32.060364 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_58b22c5102fc41568b4619bd19092925"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:32.060966 27687 ts_tablet_manager.cc:1918] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:32.058808 27685 ts_tablet_manager.cc:1918] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:32.060312 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02b3e262dd7841dfbb0520377eb94743" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:32.062002 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e34d6d5d68e4395aed78821ea077c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:32.062000 27685 log.cc:1199] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/a0293a69f1c246d08d414e95729b4ab4
I20250903 20:05:32.062175 27240 raft_consensus.cc:2466] T 02b3e262dd7841dfbb0520377eb94743 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:32.062251 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet 1ad2ca3e72434b69ad48e533da72e8fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:32.062569 27685 ts_tablet_manager.cc:1939] T a0293a69f1c246d08d414e95729b4ab4 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:32.059367 27059 catalog_manager.cc:5582] T 2059ccbd6d884085886d7d564afe03aa P 89f8f0ee68b940db9ce250c0604f934a reported cstate change: term changed from 0 to 1, leader changed from <none> to 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65). New cstate: current_term: 1 leader_uuid: "89f8f0ee68b940db9ce250c0604f934a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: HEALTHY } } }
I20250903 20:05:32.061998 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet 2478fe89414b4845b0ba377c1cde98b7 with delete_type TABLET_DATA_DELETED (replaced by tablet 361156ca43a444b1b07e1d1fab48c234) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:32.062913 27321 leader_election.cc:304] T 02b3e262dd7841dfbb0520377eb94743 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:32.063004 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 784) took 5852 ms. Trace:
I20250903 20:05:32.062852 27059 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 2059ccbd6d884085886d7d564afe03aa (table default.Table_ea5cb66a574f4466bf7e7f635f22b426 [id=91abc1158ea047849f398926e3695927]). Expected version 1 got 0
I20250903 20:05:32.063072 27685 rpcz_store.cc:276] 0903 20:05:26.210262 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.210345 (+ 83us) service_pool.cc:225] Handling call
0903 20:05:31.926040 (+5715695us) ts_tablet_manager.cc:1145] Deleting tablet a0293a69f1c246d08d414e95729b4ab4
0903 20:05:31.934098 (+ 8058us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.031177 (+ 97079us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.058707 (+ 27530us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.062989 (+ 4282us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5715500}
I20250903 20:05:32.063230 29053 raft_consensus.cc:2802] T 02b3e262dd7841dfbb0520377eb94743 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:32.063294 29053 raft_consensus.cc:491] T 02b3e262dd7841dfbb0520377eb94743 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:32.063362 29053 raft_consensus.cc:3058] T 02b3e262dd7841dfbb0520377eb94743 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:32.058558 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 2478fe89414b4845b0ba377c1cde98b7 with delete_type TABLET_DATA_DELETED (replaced by tablet 361156ca43a444b1b07e1d1fab48c234) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:32.061118 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e34d6d5d68e4395aed78821ea077c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:32.061231 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 1ad2ca3e72434b69ad48e533da72e8fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:32.066111 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 786) took 5851 ms. Trace:
I20250903 20:05:32.066248 27685 rpcz_store.cc:276] 0903 20:05:26.214235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.214306 (+ 71us) service_pool.cc:225] Handling call
0903 20:05:32.065991 (+5851685us) ts_tablet_manager.cc:1145] Deleting tablet 121a8ba102454cb4807483fac16e3781
0903 20:05:32.066000 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.066095 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5851447}
I20250903 20:05:32.063321 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_9ae576ba1bb8422b9b57aa5a78deddf5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:32.068343 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ae576ba1bb8422b9b57aa5a78deddf5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:32.069450 27462 leader_election.cc:304] T 976322efa9464ed7a2175d3e3de51dab P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:32.069489 27517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02b3e262dd7841dfbb0520377eb94743" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:32.069696 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 6d21f38759d743af972635f8b5b8d8d2
I20250903 20:05:32.070132 27057 catalog_manager.cc:6652] ReplaceTablet: tablet 6d21f38759d743af972635f8b5b8d8d2 deleted and replaced by tablet fa2014bccb5e4f0a91cbc9845df4634d
I20250903 20:05:32.070075 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 792) took 5669 ms. Trace:
I20250903 20:05:32.070186 27685 rpcz_store.cc:276] 0903 20:05:26.400465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.413936 (+ 13471us) service_pool.cc:225] Handling call
0903 20:05:32.069867 (+5655931us) ts_tablet_manager.cc:1145] Deleting tablet 55b2cdee20264ae6b64ebe3607fe1ca6
0903 20:05:32.069877 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.070041 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5646988}
I20250903 20:05:32.070497 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_867d45b3bc9248559147d26a7af1e231"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:32.071030 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_867d45b3bc9248559147d26a7af1e231 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:32.072212 27685 tablet_replica.cc:331] T c9e90abef9814338b70a3550ab5686e0 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:32.072408 27685 raft_consensus.cc:2241] T c9e90abef9814338b70a3550ab5686e0 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:32.072580 27685 raft_consensus.cc:2270] T c9e90abef9814338b70a3550ab5686e0 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.073148 27685 ts_tablet_manager.cc:1905] T c9e90abef9814338b70a3550ab5686e0 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:32.073552 29044 raft_consensus.cc:2802] T 976322efa9464ed7a2175d3e3de51dab P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:32.073729 29044 raft_consensus.cc:695] T 976322efa9464ed7a2175d3e3de51dab P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:32.074091 29044 consensus_queue.cc:237] T 976322efa9464ed7a2175d3e3de51dab P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.074295 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d21f38759d743af972635f8b5b8d8d2 with delete_type TABLET_DATA_DELETED (replaced by tablet fa2014bccb5e4f0a91cbc9845df4634d) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:32.060305 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_0570092e479a4896841a51ffcf2a88d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:32.063539 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet a0293a69f1c246d08d414e95729b4ab4 (table default.Table_61bb115ae97549f7ab9c44e52b75acf0 [id=4ce7195efb6e432e9b232b8a4749179f]) successfully deleted
W20250903 20:05:32.075552 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0570092e479a4896841a51ffcf2a88d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:32.075724 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 55b2cdee20264ae6b64ebe3607fe1ca6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55b2cdee20264ae6b64ebe3607fe1ca6 already in progress: creating tablet
I20250903 20:05:32.076260 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d21f38759d743af972635f8b5b8d8d2 with delete_type TABLET_DATA_DELETED (replaced by tablet fa2014bccb5e4f0a91cbc9845df4634d) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:32.065282 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58b22c5102fc41568b4619bd19092925 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:32.079497 27064 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_6e9cf5ead5a24831a6a14d131dd8397e [id=524a27cbe1ad4c10aa2fd4f4074105c6] alter complete (version 3)
I20250903 20:05:32.080689 27066 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_dc24dd98fc4146eda8660576fd53a29d [id=7add4b636f1b4bffae663fdf432de05f] alter complete (version 4)
I20250903 20:05:32.081394 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 904) took 3239 ms. Trace:
I20250903 20:05:32.081516 27684 rpcz_store.cc:276] 0903 20:05:28.841432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.842058 (+ 626us) service_pool.cc:225] Handling call
0903 20:05:31.925047 (+3082989us) ts_tablet_manager.cc:1145] Deleting tablet 1fbe54dc40dd47daa109ceeb07546b0a
0903 20:05:31.925059 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:31.956885 (+ 31826us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:31.995285 (+ 38400us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.080664 (+ 85379us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":75735,"tablet-delete.queue_time_us":3082865}
I20250903 20:05:32.082147 27684 tablet_replica.cc:331] T f0efbb7eefb84ab6bee5b9625b91e298 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:32.082406 27684 raft_consensus.cc:2241] T f0efbb7eefb84ab6bee5b9625b91e298 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:32.082563 27684 raft_consensus.cc:2270] T f0efbb7eefb84ab6bee5b9625b91e298 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.083182 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet 726f4d920f3e47fea41b4e12c021c988 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:32.083456 27066 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.26.65.65:44885 (request call id 162) took 1668 ms (client timeout 1000 ms). Trace:
I20250903 20:05:32.083499 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet 120ca61b3da742dc9a2efa483ead7e97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:32.083770 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 1fbe54dc40dd47daa109ceeb07546b0a (table default.Table_975e53d37e054938a5b63957252ff50d [id=f261e84232234244824d0232efbb4399]) successfully deleted
I20250903 20:05:32.084194 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:32.085391 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet 607c9a9cbc5540e5bd0fa436fcba564c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:32.085521 27065 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.26.65.67:58355 (request call id 150) took 1749 ms (client timeout 1000 ms). Trace:
I20250903 20:05:32.085604 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:32.085618 27065 rpcz_store.cc:269] 0903 20:05:30.335701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.513112 (+177411us) service_pool.cc:225] Handling call
0903 20:05:30.550763 (+ 37651us) spinlock_profiling.cc:243] Waited 36.5 ms on lock 0x616000040f54. stack: 00007f4b9d6c2dc6 00007f4b9d6c2b80 00007f4b9c6aa3ad 0000000000972d94 0000000000972d5d 00007f4bac49014d 00007f4bac490243 00007f4ba914f941 00007f4bb0cdaa7d 00007f4bb0e5e6f8 00007f4ba7c98c86 00007f4ba7c989b5 00007f4ba44730be 00007f4ba4471f9e 00007f4ba44781d5 00007f4ba4478aff
0903 20:05:32.085462 (+1534699us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":16,"spinlock_wait_cycles":84002176}
I20250903 20:05:32.085764 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet 6788b1683f984949816edfc5b68de8db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:32.085853 27489 tablet_service.cc:1515] Processing DeleteTablet for tablet 4af8c03c9d834405b4e25d35441db961 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:32.086066 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 53881ad30bc843839008ace4012a3f70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:32.086309 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 4af8c03c9d834405b4e25d35441db961 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:32.086514 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet 20c25c84c6fb4609a1964a4ff3c526da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:32.086544 27695 tablet_bootstrap.cc:492] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:32.086702 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet a0293a69f1c246d08d414e95729b4ab4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:12 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:32.086895 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 63a1f26fac3b4b939ebb79c6a07d8f8e with delete_type TABLET_DATA_DELETED (replaced by tablet 4789bd72aa454b639df76bd5ad379850) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:32.086905 27695 ts_tablet_manager.cc:1397] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.046s user 0.001s sys 0.010s
I20250903 20:05:32.086130 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_36f5d67acb72437493d261988d5ccb28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:32.087568 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36f5d67acb72437493d261988d5ccb28 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:32.088438 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet f9709dfd18654c5bb071e39da7c8889f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:17 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:32.088608 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet 0a9b7770432942da9373d5100c71bd6a with delete_type TABLET_DATA_DELETED (replaced by tablet 6800e0b6269045f29682704fd847d5d1) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:32.083585 27066 rpcz_store.cc:269] 0903 20:05:30.414912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.515487 (+100575us) service_pool.cc:225] Handling call
0903 20:05:30.543594 (+ 28107us) spinlock_profiling.cc:243] Waited 27 ms on lock 0x616000040f54. stack: 00007f4b9d6c2dc6 00007f4b9d6c2b80 00007f4b9c6aa3ad 0000000000972d94 0000000000972d5d 00007f4bac49014d 00007f4bac490243 00007f4ba914f941 00007f4bb0cdaa7d 00007f4bb0e5e6f8 00007f4ba7c98c86 00007f4ba7c989b5 00007f4ba44730be 00007f4ba4471f9e 00007f4ba44781d5 00007f4ba4478aff
0903 20:05:32.083403 (+1539809us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:31.263665 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:31.263778 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:31.263783 (+ 5us) write_op.cc:435] Acquired schema lock
0903 20:05:31.263785 (+ 2us) tablet.cc:660] Decoding operations
0903 20:05:31.263867 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:31.263876 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:31.263880 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0903 20:05:31.263929 (+ 49us) tablet.cc:699] Row locks acquired
0903 20:05:31.263931 (+ 2us) write_op.cc:260] PREPARE: finished
0903 20:05:31.263960 (+ 29us) write_op.cc:270] Start()
0903 20:05:31.264000 (+ 40us) write_op.cc:276] Timestamp: P: 1756929931263937 usec, L: 0
0903 20:05:31.264005 (+ 5us) op_driver.cc:348] REPLICATION: starting
0903 20:05:31.264138 (+ 133us) log.cc:844] Serialized 356 byte log entry
0903 20:05:31.389616 (+125478us) op_driver.cc:464] REPLICATION: finished
0903 20:05:31.390047 (+ 431us) write_op.cc:301] APPLY: starting
0903 20:05:31.390084 (+ 37us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:31.390138 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:31.390179 (+ 41us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:31.390284 (+ 105us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:31.390323 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0903 20:05:31.390331 (+ 8us) write_op.cc:312] APPLY: finished
0903 20:05:31.390448 (+ 117us) log.cc:844] Serialized 23 byte log entry
0903 20:05:31.390525 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:31.390547 (+ 22us) write_op.cc:454] Released schema lock
0903 20:05:31.390557 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0903 20:05:31.390630 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:31.390633 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":19,"spinlock_wait_cycles":62080256,"child_traces":[["op",{"apply.queue_time_us":371,"apply.run_cpu_time_us":567,"apply.run_wall_time_us":578,"num_ops":1,"prepare.queue_time_us":475946,"prepare.run_cpu_time_us":524,"prepare.run_wall_time_us":523,"replication_time_us":125558}]]}
I20250903 20:05:32.090579 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_4e5f622e7f16447aa8286aca4aa3c983" } modify_external_catalogs: true
I20250903 20:05:32.090790 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_4e5f622e7f16447aa8286aca4aa3c983" } modify_external_catalogs: true
I20250903 20:05:32.095881 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet b0d79c78e0024b7ba3bb3356be7c8ceb
I20250903 20:05:32.096565 27058 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:45936 (request call id 1157) took 1371 ms. Trace:
I20250903 20:05:32.096695 27058 rpcz_store.cc:276] 0903 20:05:30.725432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.424218 (+698786us) service_pool.cc:225] Handling call
0903 20:05:31.424766 (+ 548us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0903 20:05:31.424935 (+ 169us) catalog_manager.cc:2790] Modifying in-memory table state
0903 20:05:31.425015 (+ 80us) catalog_manager.cc:2799] Locking tablets
0903 20:05:31.425167 (+ 152us) catalog_manager.cc:2813] Removing table and tablets from system table
0903 20:05:31.503803 (+ 78636us) catalog_manager.cc:2831] Removing table from by-name map
0903 20:05:31.503897 (+ 94us) catalog_manager.cc:2847] Committing in-memory state
0903 20:05:32.095841 (+591944us) catalog_manager.cc:2851] Aborting table tasks
0903 20:05:32.096483 (+ 642us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:31.434737 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:31.434928 (+ 191us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:31.434939 (+ 11us) write_op.cc:435] Acquired schema lock
0903 20:05:31.434943 (+ 4us) tablet.cc:660] Decoding operations
0903 20:05:31.435118 (+ 175us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:31.435134 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:31.435143 (+ 9us) tablet.cc:683] Acquiring locks for 2 operations
0903 20:05:31.435319 (+ 176us) tablet.cc:699] Row locks acquired
0903 20:05:31.435323 (+ 4us) write_op.cc:260] PREPARE: finished
0903 20:05:31.435377 (+ 54us) write_op.cc:270] Start()
0903 20:05:31.435426 (+ 49us) write_op.cc:276] Timestamp: P: 1756929931435333 usec, L: 0
0903 20:05:31.435433 (+ 7us) op_driver.cc:348] REPLICATION: starting
0903 20:05:31.435701 (+ 268us) log.cc:844] Serialized 793 byte log entry
0903 20:05:31.494763 (+ 59062us) op_driver.cc:464] REPLICATION: finished
0903 20:05:31.496166 (+ 1403us) write_op.cc:301] APPLY: starting
0903 20:05:31.496180 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:31.496221 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:31.496244 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:31.496323 (+ 79us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:31.496359 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0903 20:05:31.496365 (+ 6us) write_op.cc:312] APPLY: finished
0903 20:05:31.496424 (+ 59us) log.cc:844] Serialized 29 byte log entry
0903 20:05:31.496488 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:31.496504 (+ 16us) write_op.cc:454] Released schema lock
0903 20:05:31.496510 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0903 20:05:31.496575 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:31.496579 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1357,"apply.run_cpu_time_us":404,"apply.run_wall_time_us":404,"num_ops":2,"prepare.queue_time_us":8614,"prepare.run_cpu_time_us":1068,"prepare.run_wall_time_us":1068,"replication_time_us":59299}]]}
I20250903 20:05:32.100224 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet b0d79c78e0024b7ba3bb3356be7c8ceb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:32.101472 27499 tablet_service.cc:1468] Processing CreateTablet for tablet 9836eb8ce82046949413445cdbd2e12f (DEFAULT_TABLE table=default.Table_e6fcf7607d01415abdf6d459997b3ede [id=b586317499424b2aa4ea1a1b942985a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.101976 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9836eb8ce82046949413445cdbd2e12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.103044 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet b0d79c78e0024b7ba3bb3356be7c8ceb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:32.104034 27222 tablet_service.cc:1468] Processing CreateTablet for tablet 9836eb8ce82046949413445cdbd2e12f (DEFAULT_TABLE table=default.Table_e6fcf7607d01415abdf6d459997b3ede [id=b586317499424b2aa4ea1a1b942985a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.104450 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9836eb8ce82046949413445cdbd2e12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.106475 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 7ed23e706f0b48818184067b9d58d97e (DEFAULT_TABLE table=default.Table_182ace2fdd7f4619b244f455e93d4df9 [id=b2099d269700452b9dac60029486bc0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.107494 29062 tablet.cc:1722] T 912eee5b48ef467bbd6bbe34f8aa390c P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250903 20:05:32.108024 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ed23e706f0b48818184067b9d58d97e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.110500 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 7ed23e706f0b48818184067b9d58d97e (DEFAULT_TABLE table=default.Table_182ace2fdd7f4619b244f455e93d4df9 [id=b2099d269700452b9dac60029486bc0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.110908 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ed23e706f0b48818184067b9d58d97e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.111735 29044 raft_consensus.cc:491] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:32.111862 29044 raft_consensus.cc:513] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.112488 29044 leader_election.cc:290] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:32.113179 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02b3e262dd7841dfbb0520377eb94743" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:32.113401 27240 raft_consensus.cc:2466] T 02b3e262dd7841dfbb0520377eb94743 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:32.114288 27462 leader_election.cc:304] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:32.114533 29044 raft_consensus.cc:2802] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:32.114599 29044 raft_consensus.cc:491] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:32.114665 29044 raft_consensus.cc:3058] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:32.114634 27695 raft_consensus.cc:357] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.114786 27695 raft_consensus.cc:383] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:32.114840 27695 raft_consensus.cc:738] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:32.114766 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_93cd83ac3d974349b325ce4fa2fc27ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:32.115370 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93cd83ac3d974349b325ce4fa2fc27ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:32.115252 27695 consensus_queue.cc:260] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.115974 27695 ts_tablet_manager.cc:1428] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.029s user 0.000s sys 0.001s
I20250903 20:05:32.116313 27695 tablet_bootstrap.cc:492] T 8fff36d5a80e45af8a256c3908555e08 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:32.117841 29044 raft_consensus.cc:513] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.118495 27684 ts_tablet_manager.cc:1905] T f0efbb7eefb84ab6bee5b9625b91e298 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:32.119922 29044 leader_election.cc:290] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:32.120473 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02b3e262dd7841dfbb0520377eb94743" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:32.120666 27240 raft_consensus.cc:3058] T 02b3e262dd7841dfbb0520377eb94743 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:32.124224 27240 raft_consensus.cc:2466] T 02b3e262dd7841dfbb0520377eb94743 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:32.125164 27462 leader_election.cc:304] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:32.125617 29044 raft_consensus.cc:2802] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:32.125849 29044 raft_consensus.cc:695] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:32.126310 29044 consensus_queue.cc:237] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.133280 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02b3e262dd7841dfbb0520377eb94743" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:32.061069 27687 log.cc:1199] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/726f4d920f3e47fea41b4e12c021c988
I20250903 20:05:32.134114 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e34d6d5d68e4395aed78821ea077c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.118345 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02b3e262dd7841dfbb0520377eb94743" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:32.134548 27348 tablet_service.cc:1515] Processing DeleteTablet for tablet b0d79c78e0024b7ba3bb3356be7c8ceb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.134548 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet 1ad2ca3e72434b69ad48e533da72e8fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.136013 27687 ts_tablet_manager.cc:1939] T 726f4d920f3e47fea41b4e12c021c988 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:32.137737 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 9836eb8ce82046949413445cdbd2e12f (DEFAULT_TABLE table=default.Table_e6fcf7607d01415abdf6d459997b3ede [id=b586317499424b2aa4ea1a1b942985a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.138182 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9836eb8ce82046949413445cdbd2e12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.138325 27350 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d21f38759d743af972635f8b5b8d8d2 with delete_type TABLET_DATA_DELETED (replaced by tablet fa2014bccb5e4f0a91cbc9845df4634d) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.138422 28874 consensus_queue.cc:1035] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:32.139811 27687 tablet_replica.cc:331] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:32.140614 27687 raft_consensus.cc:2241] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:32.140659 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 726f4d920f3e47fea41b4e12c021c988 (table default.Table_a6608fc0162446a480d8e8ecb07db28f [id=66b42d6c99de4096998cba421c5b4663]) successfully deleted
I20250903 20:05:32.137737 27353 tablet_service.cc:1468] Processing CreateTablet for tablet fc17b136c3314bd3a3e68d1f4c856ff3 (DEFAULT_TABLE table=default.Table_c623ccc6c3ab480caca560c89b2444ae [id=cf05fbc4782e4f448d51618e9ff5311f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.140954 27687 raft_consensus.cc:2270] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.141242 27353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc17b136c3314bd3a3e68d1f4c856ff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.141609 27348 tablet_service.cc:1468] Processing CreateTablet for tablet 7ed23e706f0b48818184067b9d58d97e (DEFAULT_TABLE table=default.Table_182ace2fdd7f4619b244f455e93d4df9 [id=b2099d269700452b9dac60029486bc0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.141806 27687 ts_tablet_manager.cc:1905] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:32.141937 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ed23e706f0b48818184067b9d58d97e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.143513 29053 raft_consensus.cc:513] T 02b3e262dd7841dfbb0520377eb94743 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.144027 27385 raft_consensus.cc:2391] T 02b3e262dd7841dfbb0520377eb94743 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
I20250903 20:05:32.144619 29053 leader_election.cc:290] T 02b3e262dd7841dfbb0520377eb94743 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:32.145156 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02b3e262dd7841dfbb0520377eb94743" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:32.145264 27695 tablet_bootstrap.cc:654] T 8fff36d5a80e45af8a256c3908555e08 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:32.145452 27240 raft_consensus.cc:2391] T 02b3e262dd7841dfbb0520377eb94743 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 51578fb7bceb4e119b3ab1e58c4f5528 in current term 1: Already voted for candidate e72ef1dbcc5143cf9f04a808f101c80f in this term.
I20250903 20:05:32.145778 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02b3e262dd7841dfbb0520377eb94743" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:32.146732 27320 leader_election.cc:304] T 02b3e262dd7841dfbb0520377eb94743 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:32.147276 29053 raft_consensus.cc:2747] T 02b3e262dd7841dfbb0520377eb94743 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250903 20:05:32.159162 27506 tablet_service.cc:1468] Processing CreateTablet for tablet fc17b136c3314bd3a3e68d1f4c856ff3 (DEFAULT_TABLE table=default.Table_c623ccc6c3ab480caca560c89b2444ae [id=cf05fbc4782e4f448d51618e9ff5311f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.159770 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc17b136c3314bd3a3e68d1f4c856ff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.164814 27695 tablet_bootstrap.cc:492] T 8fff36d5a80e45af8a256c3908555e08 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:32.164963 27695 ts_tablet_manager.cc:1397] T 8fff36d5a80e45af8a256c3908555e08 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.000s
I20250903 20:05:32.173688 27222 tablet_service.cc:1468] Processing CreateTablet for tablet fc17b136c3314bd3a3e68d1f4c856ff3 (DEFAULT_TABLE table=default.Table_c623ccc6c3ab480caca560c89b2444ae [id=cf05fbc4782e4f448d51618e9ff5311f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.176199 27685 ts_tablet_manager.cc:1918] T c9e90abef9814338b70a3550ab5686e0 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:32.176419 27685 log.cc:1199] T c9e90abef9814338b70a3550ab5686e0 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/c9e90abef9814338b70a3550ab5686e0
I20250903 20:05:32.176911 27685 ts_tablet_manager.cc:1939] T c9e90abef9814338b70a3550ab5686e0 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:32.176945 27348 tablet_service.cc:1468] Processing CreateTablet for tablet d356e5f4d96f46689720b1c00e882f9c (DEFAULT_TABLE table=default.Table_e672f826c7de4a0eae169f4c4002f34a [id=172b5cbe5c0a4cb4a91101f40e79f977]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.177389 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 793) took 5721 ms. Trace:
I20250903 20:05:32.179237 27685 rpcz_store.cc:276] 0903 20:05:26.455863 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.460363 (+ 4500us) service_pool.cc:225] Handling call
0903 20:05:32.072155 (+5611792us) ts_tablet_manager.cc:1145] Deleting tablet c9e90abef9814338b70a3550ab5686e0
0903 20:05:32.072166 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.166492 (+ 94326us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.176078 (+ 9586us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.177370 (+ 1292us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5611556}
I20250903 20:05:32.179909 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 796) took 5710 ms. Trace:
I20250903 20:05:32.180086 27685 rpcz_store.cc:276] 0903 20:05:26.469042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.469125 (+ 83us) service_pool.cc:225] Handling call
0903 20:05:32.179750 (+5710625us) ts_tablet_manager.cc:1145] Deleting tablet ef02f91c0964446daf62ac1dde0330ff
0903 20:05:32.179765 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.179894 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5710400}
I20250903 20:05:32.180513 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 795) took 5711 ms. Trace:
I20250903 20:05:32.180653 27685 rpcz_store.cc:276] 0903 20:05:26.468909 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.469389 (+ 480us) service_pool.cc:225] Handling call
0903 20:05:32.180401 (+5711012us) ts_tablet_manager.cc:1145] Deleting tablet ab64604c56914379943c03129ccc2e7c
0903 20:05:32.180411 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.180498 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5710900}
I20250903 20:05:32.177858 27695 raft_consensus.cc:357] T 8fff36d5a80e45af8a256c3908555e08 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.181026 27695 raft_consensus.cc:383] T 8fff36d5a80e45af8a256c3908555e08 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:32.181072 27685 tablet_replica.cc:331] T 17e0864a32d5452d84ac470b55fbaaf4 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:32.181111 27695 raft_consensus.cc:738] T 8fff36d5a80e45af8a256c3908555e08 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:32.181429 27685 raft_consensus.cc:2241] T 17e0864a32d5452d84ac470b55fbaaf4 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:32.159351 27684 ts_tablet_manager.cc:1918] T f0efbb7eefb84ab6bee5b9625b91e298 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:32.181172 28947 consensus_queue.cc:1035] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:32.182090 27217 tablet_service.cc:1468] Processing CreateTablet for tablet d356e5f4d96f46689720b1c00e882f9c (DEFAULT_TABLE table=default.Table_e672f826c7de4a0eae169f4c4002f34a [id=172b5cbe5c0a4cb4a91101f40e79f977]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.181762 27684 log.cc:1199] T f0efbb7eefb84ab6bee5b9625b91e298 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/f0efbb7eefb84ab6bee5b9625b91e298
I20250903 20:05:32.182348 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet c9e90abef9814338b70a3550ab5686e0 (table default.Table_5b44669e91a8427c8e7d96b866fdb3b5 [id=96167b0d9f9b47168a9fc59635b7136d]) successfully deleted
W20250903 20:05:32.182646 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet ef02f91c0964446daf62ac1dde0330ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef02f91c0964446daf62ac1dde0330ff already in progress: creating tablet
I20250903 20:05:32.182682 27695 consensus_queue.cc:260] T 8fff36d5a80e45af8a256c3908555e08 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:32.182901 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet ab64604c56914379943c03129ccc2e7c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab64604c56914379943c03129ccc2e7c already in progress: creating tablet
I20250903 20:05:32.184762 27684 ts_tablet_manager.cc:1939] T f0efbb7eefb84ab6bee5b9625b91e298 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:32.185153 27226 tablet_service.cc:1468] Processing CreateTablet for tablet 38b912e24bf244759be8a6dc70cf9f52 (DEFAULT_TABLE table=default.Table_5fb98b65f51b4d628de6bed5d36b53d1 [id=b82c8f8f0f6b415580c2b590f05c0f7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.185889 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 905) took 3344 ms. Trace:
I20250903 20:05:32.186087 27684 rpcz_store.cc:276] 0903 20:05:28.841500 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.842199 (+ 699us) service_pool.cc:225] Handling call
0903 20:05:32.082076 (+3239877us) ts_tablet_manager.cc:1145] Deleting tablet f0efbb7eefb84ab6bee5b9625b91e298
0903 20:05:32.082093 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.131675 (+ 49582us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.159215 (+ 27540us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.185864 (+ 26649us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3239718}
I20250903 20:05:32.186290 27352 tablet_service.cc:1468] Processing CreateTablet for tablet 38b912e24bf244759be8a6dc70cf9f52 (DEFAULT_TABLE table=default.Table_5fb98b65f51b4d628de6bed5d36b53d1 [id=b82c8f8f0f6b415580c2b590f05c0f7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.186806 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet f0efbb7eefb84ab6bee5b9625b91e298 (table default.Table_781d39d6b06f49c28fa93c3692c7b5f7 [id=8d664216ea8e47fdaba51b2d6ca9d0a5]) successfully deleted
I20250903 20:05:32.187211 27684 tablet_replica.cc:331] T 59a1dd48d76241d7ba817070fbffd853 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:32.187672 27684 raft_consensus.cc:2241] T 59a1dd48d76241d7ba817070fbffd853 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:32.188227 29053 consensus_queue.cc:1035] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:32.188591 27684 raft_consensus.cc:2270] T 59a1dd48d76241d7ba817070fbffd853 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.189461 27684 ts_tablet_manager.cc:1905] T 59a1dd48d76241d7ba817070fbffd853 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:32.203300 28947 consensus_queue.cc:1035] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:32.206853 27684 ts_tablet_manager.cc:1918] T 59a1dd48d76241d7ba817070fbffd853 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:32.208403 27684 log.cc:1199] T 59a1dd48d76241d7ba817070fbffd853 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/59a1dd48d76241d7ba817070fbffd853
I20250903 20:05:32.208972 27684 ts_tablet_manager.cc:1939] T 59a1dd48d76241d7ba817070fbffd853 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:32.210060 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 906) took 3368 ms. Trace:
I20250903 20:05:32.212715 27695 ts_tablet_manager.cc:1428] T 8fff36d5a80e45af8a256c3908555e08 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.048s user 0.003s sys 0.000s
I20250903 20:05:32.213137 27695 tablet_bootstrap.cc:492] T 5002b7e8118c49b6a247cb8dc4ca1aea P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:32.213198 27350 tablet_service.cc:1468] Processing CreateTablet for tablet bd1dd609b75444c8a86d7df49745e1fa (DEFAULT_TABLE table=default.Table_fc064d54000a4fe99c75c42de917231b [id=8e5c4c866bd149b98d0247279cfcecac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.212642 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 38b912e24bf244759be8a6dc70cf9f52 (DEFAULT_TABLE table=default.Table_5fb98b65f51b4d628de6bed5d36b53d1 [id=b82c8f8f0f6b415580c2b590f05c0f7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.214088 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38b912e24bf244759be8a6dc70cf9f52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.215341 27695 tablet_bootstrap.cc:654] T 5002b7e8118c49b6a247cb8dc4ca1aea P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:32.214859 27499 tablet_service.cc:1468] Processing CreateTablet for tablet d356e5f4d96f46689720b1c00e882f9c (DEFAULT_TABLE table=default.Table_e672f826c7de4a0eae169f4c4002f34a [id=172b5cbe5c0a4cb4a91101f40e79f977]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.218850 27214 tablet_service.cc:1468] Processing CreateTablet for tablet bd1dd609b75444c8a86d7df49745e1fa (DEFAULT_TABLE table=default.Table_fc064d54000a4fe99c75c42de917231b [id=8e5c4c866bd149b98d0247279cfcecac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.218880 27684 rpcz_store.cc:276] 0903 20:05:28.841587 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.842376 (+ 789us) service_pool.cc:225] Handling call
0903 20:05:32.187154 (+3344778us) ts_tablet_manager.cc:1145] Deleting tablet 59a1dd48d76241d7ba817070fbffd853
0903 20:05:32.187165 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.198177 (+ 11012us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.206683 (+ 8506us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.210033 (+ 3350us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3344650}
I20250903 20:05:32.219072 27489 tablet_service.cc:1468] Processing CreateTablet for tablet bd1dd609b75444c8a86d7df49745e1fa (DEFAULT_TABLE table=default.Table_fc064d54000a4fe99c75c42de917231b [id=8e5c4c866bd149b98d0247279cfcecac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.219404 27489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd1dd609b75444c8a86d7df49745e1fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.219497 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 59a1dd48d76241d7ba817070fbffd853 (table default.Table_d342b3c06e1c4b98bc7f07e88844cdc9 [id=34db41bcdc604ee08da11d0e559b4d35]) successfully deleted
I20250903 20:05:32.219988 27684 tablet_replica.cc:331] T a4f3230f50ec4c95b528fead8c191224 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:32.221618 27684 raft_consensus.cc:2241] T a4f3230f50ec4c95b528fead8c191224 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:32.221915 27684 raft_consensus.cc:2270] T a4f3230f50ec4c95b528fead8c191224 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.223057 27684 ts_tablet_manager.cc:1905] T a4f3230f50ec4c95b528fead8c191224 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:32.225320 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 355e02e2a021411e918493b8a63468c8 (DEFAULT_TABLE table=default.Table_c642bf5deebb4382b01bc6ceba8b9daa [id=99fc43093b674fc0bc5ac06ba7a35e87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.226980 28947 raft_consensus.cc:491] T 1bad3c509460422396a17a3c4ae3d1e8 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51578fb7bceb4e119b3ab1e58c4f5528)
I20250903 20:05:32.227119 28947 raft_consensus.cc:513] T 1bad3c509460422396a17a3c4ae3d1e8 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.227717 28947 leader_election.cc:290] T 1bad3c509460422396a17a3c4ae3d1e8 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
W20250903 20:05:32.229580 27183 leader_election.cc:343] T 1bad3c509460422396a17a3c4ae3d1e8 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 1bad3c509460422396a17a3c4ae3d1e8
I20250903 20:05:32.230772 28947 consensus_queue.cc:1035] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:32.231354 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d356e5f4d96f46689720b1c00e882f9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.231503 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 355e02e2a021411e918493b8a63468c8 (DEFAULT_TABLE table=default.Table_c642bf5deebb4382b01bc6ceba8b9daa [id=99fc43093b674fc0bc5ac06ba7a35e87]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:32.231599 27184 leader_election.cc:343] T 1bad3c509460422396a17a3c4ae3d1e8 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 1bad3c509460422396a17a3c4ae3d1e8
I20250903 20:05:32.231858 27184 leader_election.cc:304] T 1bad3c509460422396a17a3c4ae3d1e8 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:32.233373 27685 raft_consensus.cc:2270] T 17e0864a32d5452d84ac470b55fbaaf4 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.233433 28947 raft_consensus.cc:2747] T 1bad3c509460422396a17a3c4ae3d1e8 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:32.233973 27685 ts_tablet_manager.cc:1905] T 17e0864a32d5452d84ac470b55fbaaf4 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:32.235899 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 355e02e2a021411e918493b8a63468c8 (DEFAULT_TABLE table=default.Table_c642bf5deebb4382b01bc6ceba8b9daa [id=99fc43093b674fc0bc5ac06ba7a35e87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.252010 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 355e02e2a021411e918493b8a63468c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.258147 27695 tablet_bootstrap.cc:492] T 5002b7e8118c49b6a247cb8dc4ca1aea P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:32.261699 27695 ts_tablet_manager.cc:1397] T 5002b7e8118c49b6a247cb8dc4ca1aea P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.048s user 0.006s sys 0.000s
I20250903 20:05:32.267778 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_53bee4ebf7974f119a4a5b5bd223d51f" } modify_external_catalogs: true
I20250903 20:05:32.268015 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_53bee4ebf7974f119a4a5b5bd223d51f" } modify_external_catalogs: true
I20250903 20:05:32.269524 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_5cbc8c64bdd34f028ab716135624ab49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:32.269843 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5d167e8474a043f09801a985626f69e4" } new_table_name: "default.Table_5176efec4bc940be953d2bfcc1cbd226" modify_external_catalogs: true
W20250903 20:05:32.270200 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5cbc8c64bdd34f028ab716135624ab49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:32.271641 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b880e180787d4987961ba1ef18c3ca8f" } new_table_name: "default.Table_adaa11515bc04403b4f1a584b21c0549" modify_external_catalogs: true
I20250903 20:05:32.274693 27684 ts_tablet_manager.cc:1918] T a4f3230f50ec4c95b528fead8c191224 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:32.277115 27684 log.cc:1199] T a4f3230f50ec4c95b528fead8c191224 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/a4f3230f50ec4c95b528fead8c191224
I20250903 20:05:32.284111 27684 ts_tablet_manager.cc:1939] T a4f3230f50ec4c95b528fead8c191224 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:32.291828 27064 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_9b20469406c94986ab496ea3e678b586 [id=6d6f5e8cadf9445d8695c075cfabb2ea] alter complete (version 1)
I20250903 20:05:32.292430 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 907) took 3431 ms. Trace:
I20250903 20:05:32.293617 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet f9709dfd18654c5bb071e39da7c8889f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:17 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.293810 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet 120ca61b3da742dc9a2efa483ead7e97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.294303 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 90baf83357494a0aae325908c118b471 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.294477 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet f2629f46704e4a0fa7922533928ca2ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.293031 27684 rpcz_store.cc:276] 0903 20:05:28.861322 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.861412 (+ 90us) service_pool.cc:225] Handling call
0903 20:05:32.219902 (+3358490us) ts_tablet_manager.cc:1145] Deleting tablet a4f3230f50ec4c95b528fead8c191224
0903 20:05:32.219915 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.232382 (+ 12467us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.273226 (+ 40844us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.292398 (+ 19172us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3358120}
I20250903 20:05:32.294706 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet 0a9b7770432942da9373d5100c71bd6a with delete_type TABLET_DATA_DELETED (replaced by tablet 6800e0b6269045f29682704fd847d5d1) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.295063 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet a4f3230f50ec4c95b528fead8c191224 (table default.Table_be001b7aec594b93a1b45e9313b772ca [id=7a045d80e0eb42cb95fdca51d339da0d]) successfully deleted
I20250903 20:05:32.295895 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet 4af8c03c9d834405b4e25d35441db961 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.296092 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 8fff36d5a80e45af8a256c3908555e08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.296248 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 567630e2a93645e58f9bf05ffff87b32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.296324 27684 tablet_replica.cc:331] T 7e2f98f760cf4df4bc2a10530e2157c6 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:32.296555 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 541278ca9ffb4e41b341139c2faa337f with delete_type TABLET_DATA_DELETED (replaced by tablet e9126e4a18c546c28927f22c9f16426e) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.296875 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 63a1f26fac3b4b939ebb79c6a07d8f8e with delete_type TABLET_DATA_DELETED (replaced by tablet 4789bd72aa454b639df76bd5ad379850) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.297214 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 22bc87d1a4804b6c80f77f9692500d63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.297519 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 726f4d920f3e47fea41b4e12c021c988 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.298617 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bf69315f07a470bb2b897ad7c590729 with delete_type TABLET_DATA_DELETED (replaced by tablet 17514a2af39b402abbe8263d7c1abc7f) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:32.298728 27488 tablet_service.cc:1515] Processing DeleteTablet for tablet b7a6d5d8a97a4769a81ec7224d591f8c with delete_type TABLET_DATA_DELETED (replaced by tablet 79c0cc0c30ab42eab0be415e3db9589f) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:32.299155 27488 tablet_service.cc:1515] Processing DeleteTablet for tablet 541278ca9ffb4e41b341139c2faa337f with delete_type TABLET_DATA_DELETED (replaced by tablet e9126e4a18c546c28927f22c9f16426e) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:32.299154 27489 tablet_service.cc:1515] Processing DeleteTablet for tablet 512908bc4d6941ec81b145ee5795405f with delete_type TABLET_DATA_DELETED (replaced by tablet 1598824a733a4a93bed043e81b86abf3) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:32.299597 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 726f4d920f3e47fea41b4e12c021c988 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:32.300165 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet a0f79831866b40c49c5b27ce2014c3f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:32.301330 27489 tablet_service.cc:1515] Processing DeleteTablet for tablet a4f3230f50ec4c95b528fead8c191224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:32.301378 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 22bc87d1a4804b6c80f77f9692500d63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:32.301834 27060 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.26.65.67:58355 (request call id 151) took 896 ms (client timeout 1000 ms). Trace:
I20250903 20:05:32.300688 27684 raft_consensus.cc:2241] T 7e2f98f760cf4df4bc2a10530e2157c6 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
W20250903 20:05:32.301986 27060 rpcz_store.cc:269] 0903 20:05:31.405408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.418491 (+ 13083us) service_pool.cc:225] Handling call
0903 20:05:32.301749 (+883258us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:31.773505 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:31.780712 (+ 7207us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:31.780730 (+ 18us) write_op.cc:435] Acquired schema lock
0903 20:05:31.780733 (+ 3us) tablet.cc:660] Decoding operations
0903 20:05:31.780863 (+ 130us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:31.780876 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:31.780882 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0903 20:05:31.780948 (+ 66us) tablet.cc:699] Row locks acquired
0903 20:05:31.780950 (+ 2us) write_op.cc:260] PREPARE: finished
0903 20:05:31.780992 (+ 42us) write_op.cc:270] Start()
0903 20:05:31.781027 (+ 35us) write_op.cc:276] Timestamp: P: 1756929931780972 usec, L: 0
0903 20:05:31.781031 (+ 4us) op_driver.cc:348] REPLICATION: starting
0903 20:05:31.781227 (+ 196us) log.cc:844] Serialized 530 byte log entry
0903 20:05:31.787992 (+ 6765us) op_driver.cc:464] REPLICATION: finished
0903 20:05:31.788079 (+ 87us) write_op.cc:301] APPLY: starting
0903 20:05:31.788091 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:31.788129 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:31.788156 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:31.788198 (+ 42us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:31.788224 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0903 20:05:31.788229 (+ 5us) write_op.cc:312] APPLY: finished
0903 20:05:31.788284 (+ 55us) log.cc:844] Serialized 23 byte log entry
0903 20:05:31.788335 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:31.788347 (+ 12us) write_op.cc:454] Released schema lock
0903 20:05:31.788352 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":35,"child_traces":[["op",{"apply.queue_time_us":33,"apply.run_cpu_time_us":340,"apply.run_wall_time_us":339,"num_ops":1,"prepare.queue_time_us":175767,"prepare.run_cpu_time_us":825,"prepare.run_wall_time_us":9128,"raft.queue_time_us":35,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":242,"replication_time_us":6938}]]}
I20250903 20:05:32.303462 27684 raft_consensus.cc:2270] T 7e2f98f760cf4df4bc2a10530e2157c6 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:32.304868 27064 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.26.65.66:53547 (request call id 130) took 1155 ms (client timeout 1000 ms). Trace:
W20250903 20:05:32.305085 27064 rpcz_store.cc:269] 0903 20:05:31.149226 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.273587 (+124361us) service_pool.cc:225] Handling call
0903 20:05:31.376169 (+102582us) spinlock_profiling.cc:243] Waited 76.9 ms on lock 0x616000040f54. stack: 00007f4b9d6c2dc6 00007f4b9d6c2b80 00007f4b9c6aa3ad 0000000000972d94 0000000000972d5d 00007f4bac49014d 00007f4bac490243 00007f4ba914f941 00007f4bb0cdaa7d 00007f4bb0e5e6f8 00007f4ba7c98c86 00007f4ba7c989b5 00007f4ba44730be 00007f4ba4471f9e 00007f4ba44781d5 00007f4ba4478aff
0903 20:05:32.304788 (+928619us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:31.593144 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:31.593305 (+ 161us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:31.593321 (+ 16us) write_op.cc:435] Acquired schema lock
0903 20:05:31.593324 (+ 3us) tablet.cc:660] Decoding operations
0903 20:05:31.593461 (+ 137us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:31.593475 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:31.593481 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0903 20:05:31.593554 (+ 73us) tablet.cc:699] Row locks acquired
0903 20:05:31.593557 (+ 3us) write_op.cc:260] PREPARE: finished
0903 20:05:31.593603 (+ 46us) write_op.cc:270] Start()
0903 20:05:31.593706 (+ 103us) write_op.cc:276] Timestamp: P: 1756929931593548 usec, L: 0
0903 20:05:31.593712 (+ 6us) op_driver.cc:348] REPLICATION: starting
0903 20:05:31.769217 (+175505us) log.cc:844] Serialized 356 byte log entry
0903 20:05:31.772439 (+ 3222us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x616000040f54. stack: 00007f4b9d6c2dc6 00007f4b9d6c2b80 00007f4b9c6aa3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f4ba9121d62 00007f4baa4ca2d5 00007f4baa4c8df9 00007f4baa4d1220 00007f4baa4d102d 0000000000a774dc 00007f4b9d70914f 00007f4b9d70c41b 00007f4b9d70c298 0000000000a774dc
0903 20:05:31.784966 (+ 12527us) op_driver.cc:464] REPLICATION: finished
0903 20:05:31.787346 (+ 2380us) write_op.cc:301] APPLY: starting
0903 20:05:31.787362 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:31.787399 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:31.787403 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:31.787456 (+ 53us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:31.787490 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0903 20:05:31.787497 (+ 7us) write_op.cc:312] APPLY: finished
0903 20:05:31.787620 (+ 123us) log.cc:844] Serialized 23 byte log entry
0903 20:05:31.787690 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:31.787708 (+ 18us) write_op.cc:454] Released schema lock
0903 20:05:31.787713 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0903 20:05:32.112165 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:32.112251 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:32.112256 (+ 5us) write_op.cc:435] Acquired schema lock
0903 20:05:32.112258 (+ 2us) tablet.cc:660] Decoding operations
0903 20:05:32.112344 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:32.112352 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:32.112356 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0903 20:05:32.112399 (+ 43us) tablet.cc:699] Row locks acquired
0903 20:05:32.112401 (+ 2us) write_op.cc:260] PREPARE: finished
0903 20:05:32.112425 (+ 24us) write_op.cc:270] Start()
0903 20:05:32.112447 (+ 22us) write_op.cc:276] Timestamp: P: 1756929932112381 usec, L: 0
0903 20:05:32.112451 (+ 4us) op_driver.cc:348] REPLICATION: starting
0903 20:05:32.112570 (+ 119us) log.cc:844] Serialized 356 byte log entry
0903 20:05:32.289757 (+177187us) op_driver.cc:464] REPLICATION: finished
0903 20:05:32.291196 (+ 1439us) write_op.cc:301] APPLY: starting
0903 20:05:32.291215 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:32.291266 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:32.291296 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:32.291364 (+ 68us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:32.291410 (+ 46us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0903 20:05:32.291422 (+ 12us) write_op.cc:312] APPLY: finished
0903 20:05:32.291525 (+ 103us) log.cc:844] Serialized 23 byte log entry
0903 20:05:32.291596 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:32.291620 (+ 24us) write_op.cc:454] Released schema lock
0903 20:05:32.291629 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":27,"spinlock_wait_cycles":176914688,"child_traces":[["op",{"apply.queue_time_us":2313,"apply.run_cpu_time_us":437,"apply.run_wall_time_us":437,"num_ops":1,"prepare.queue_time_us":9002,"prepare.run_cpu_time_us":2538,"prepare.run_wall_time_us":180253,"raft.queue_time_us":88,"raft.run_cpu_time_us":282,"raft.run_wall_time_us":283,"replication_time_us":191224,"spinlock_wait_cycles":402984832}],["op",{"apply.queue_time_us":624,"apply.run_cpu_time_us":524,"apply.run_wall_time_us":524,"num_ops":1,"prepare.queue_time_us":12845,"prepare.run_cpu_time_us":446,"prepare.run_wall_time_us":446,"replication_time_us":176246}]]}
I20250903 20:05:32.307235 27684 ts_tablet_manager.cc:1905] T 7e2f98f760cf4df4bc2a10530e2157c6 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:32.297909 27695 raft_consensus.cc:357] T 5002b7e8118c49b6a247cb8dc4ca1aea P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.310719 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_a7795c6eb7c64c4ba9f47fbff01490ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:32.309877 27695 raft_consensus.cc:383] T 5002b7e8118c49b6a247cb8dc4ca1aea P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250903 20:05:32.311630 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7795c6eb7c64c4ba9f47fbff01490ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:32.311919 27695 raft_consensus.cc:738] T 5002b7e8118c49b6a247cb8dc4ca1aea P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:32.313303 27693 tablet_bootstrap.cc:492] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:32.315708 27516 raft_consensus.cc:3058] T 120ca61b3da742dc9a2efa483ead7e97 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:32.317636 27183 consensus_peers.cc:489] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59a1dd48d76241d7ba817070fbffd853. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:32.317517 27695 consensus_queue.cc:260] T 5002b7e8118c49b6a247cb8dc4ca1aea P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.313936 27693 ts_tablet_manager.cc:1397] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.262s user 0.005s sys 0.000s
W20250903 20:05:32.354465 27183 consensus_peers.cc:489] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53881ad30bc843839008ace4012a3f70. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:32.356848 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d356e5f4d96f46689720b1c00e882f9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.357532 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38b912e24bf244759be8a6dc70cf9f52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.357945 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd1dd609b75444c8a86d7df49745e1fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.358399 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 355e02e2a021411e918493b8a63468c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.359690 27693 raft_consensus.cc:357] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:32.359966 27693 raft_consensus.cc:383] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:32.360080 27693 raft_consensus.cc:738] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:32.362195 28567 consensus_queue.cc:1035] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:32.364197 27692 tablet_bootstrap.cc:492] T d1c96296a579440c887ba6c2a93f7204 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:32.364346 27692 ts_tablet_manager.cc:1397] T d1c96296a579440c887ba6c2a93f7204 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.315s user 0.015s sys 0.011s
I20250903 20:05:32.365149 27692 raft_consensus.cc:357] T d1c96296a579440c887ba6c2a93f7204 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.365309 27692 raft_consensus.cc:383] T d1c96296a579440c887ba6c2a93f7204 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:32.365360 27692 raft_consensus.cc:738] T d1c96296a579440c887ba6c2a93f7204 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:32.369256 27692 consensus_queue.cc:260] T d1c96296a579440c887ba6c2a93f7204 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.377676 27692 ts_tablet_manager.cc:1428] T d1c96296a579440c887ba6c2a93f7204 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.013s user 0.002s sys 0.001s
I20250903 20:05:32.379717 27684 ts_tablet_manager.cc:1918] T 7e2f98f760cf4df4bc2a10530e2157c6 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:32.380499 27684 log.cc:1199] T 7e2f98f760cf4df4bc2a10530e2157c6 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/7e2f98f760cf4df4bc2a10530e2157c6
I20250903 20:05:32.382041 27692 tablet_bootstrap.cc:492] T 5002b7e8118c49b6a247cb8dc4ca1aea P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:32.383463 27684 ts_tablet_manager.cc:1939] T 7e2f98f760cf4df4bc2a10530e2157c6 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:32.361553 27693 consensus_queue.cc:260] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:32.389994 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc17b136c3314bd3a3e68d1f4c856ff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.390304 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd1dd609b75444c8a86d7df49745e1fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.390508 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38b912e24bf244759be8a6dc70cf9f52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.390847 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 355e02e2a021411e918493b8a63468c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.396453 27685 ts_tablet_manager.cc:1918] T 17e0864a32d5452d84ac470b55fbaaf4 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:32.396595 27685 log.cc:1199] T 17e0864a32d5452d84ac470b55fbaaf4 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/17e0864a32d5452d84ac470b55fbaaf4
I20250903 20:05:32.396963 27685 ts_tablet_manager.cc:1939] T 17e0864a32d5452d84ac470b55fbaaf4 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:32.400940 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 908) took 3538 ms. Trace:
I20250903 20:05:32.401170 27684 rpcz_store.cc:276] 0903 20:05:28.862486 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.862537 (+ 51us) service_pool.cc:225] Handling call
0903 20:05:32.295927 (+3433390us) ts_tablet_manager.cc:1145] Deleting tablet 7e2f98f760cf4df4bc2a10530e2157c6
0903 20:05:32.296108 (+ 181us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.350824 (+ 54716us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.378687 (+ 27863us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.400906 (+ 22219us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3432934}
I20250903 20:05:32.401906 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 794) took 5936 ms. Trace:
I20250903 20:05:32.402654 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 7e2f98f760cf4df4bc2a10530e2157c6 (table default.Table_3185c25ec3784288bd52ccf75830589d [id=4d1a27b6dfb74dd69ba27d0f1e767dac]) successfully deleted
I20250903 20:05:32.402419 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d356e5f4d96f46689720b1c00e882f9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.409417 27684 tablet_replica.cc:331] T e0347eab6c8642beabdc2d7e7bbfdc9b P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:32.409734 27684 raft_consensus.cc:2241] T e0347eab6c8642beabdc2d7e7bbfdc9b P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:32.409868 27685 rpcz_store.cc:276] 0903 20:05:26.465203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.471294 (+ 6091us) service_pool.cc:225] Handling call
0903 20:05:32.180927 (+5709633us) ts_tablet_manager.cc:1145] Deleting tablet 17e0864a32d5452d84ac470b55fbaaf4
0903 20:05:32.180935 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.393467 (+212532us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.396348 (+ 2881us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.401887 (+ 5539us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5709205}
I20250903 20:05:32.410372 27684 raft_consensus.cc:2270] T e0347eab6c8642beabdc2d7e7bbfdc9b P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.410799 27685 tablet_replica.cc:331] T 961ade36f7d14eba91bb16d5eae2e254 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:32.412114 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 17e0864a32d5452d84ac470b55fbaaf4 (table default.Table_c21ce98311d847f2aaae2ea2c0549856 [id=263d1af690fb4868a4bd92ae090dfd0a]) successfully deleted
I20250903 20:05:32.412998 27685 raft_consensus.cc:2241] T 961ade36f7d14eba91bb16d5eae2e254 P 89f8f0ee68b940db9ce250c0604f934a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:32.413563 27684 ts_tablet_manager.cc:1905] T e0347eab6c8642beabdc2d7e7bbfdc9b P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:32.413847 27685 raft_consensus.cc:2270] T 961ade36f7d14eba91bb16d5eae2e254 P 89f8f0ee68b940db9ce250c0604f934a [term 2 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:32.421021 27184 consensus_peers.cc:489] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:32.413882 27692 tablet_bootstrap.cc:654] T 5002b7e8118c49b6a247cb8dc4ca1aea P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:32.430472 27685 ts_tablet_manager.cc:1905] T 961ade36f7d14eba91bb16d5eae2e254 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:32.432255 28947 raft_consensus.cc:491] T 8fff36d5a80e45af8a256c3908555e08 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:32.432410 28947 raft_consensus.cc:513] T 8fff36d5a80e45af8a256c3908555e08 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.432934 27684 ts_tablet_manager.cc:1918] T e0347eab6c8642beabdc2d7e7bbfdc9b P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:32.433147 28947 leader_election.cc:290] T 8fff36d5a80e45af8a256c3908555e08 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:32.433223 27684 log.cc:1199] T e0347eab6c8642beabdc2d7e7bbfdc9b P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/e0347eab6c8642beabdc2d7e7bbfdc9b
I20250903 20:05:32.434144 27684 ts_tablet_manager.cc:1939] T e0347eab6c8642beabdc2d7e7bbfdc9b P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:32.437124 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 909) took 3470 ms. Trace:
I20250903 20:05:32.437777 27684 rpcz_store.cc:276] 0903 20:05:28.966097 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.966155 (+ 58us) service_pool.cc:225] Handling call
0903 20:05:32.403011 (+3436856us) ts_tablet_manager.cc:1145] Deleting tablet e0347eab6c8642beabdc2d7e7bbfdc9b
0903 20:05:32.409344 (+ 6333us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.426749 (+ 17405us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.432686 (+ 5937us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.437004 (+ 4318us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3436620}
I20250903 20:05:32.439062 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet e0347eab6c8642beabdc2d7e7bbfdc9b (table default.Table_187e391a6d6d4d0d96d4bb8a0581face [id=54ebade293b7466aa39f00ec876f27eb]) successfully deleted
I20250903 20:05:32.440392 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 910) took 3447 ms. Trace:
I20250903 20:05:32.440596 27684 rpcz_store.cc:276] 0903 20:05:28.992662 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.992728 (+ 66us) service_pool.cc:225] Handling call
0903 20:05:32.439724 (+3446996us) ts_tablet_manager.cc:1145] Deleting tablet 86e33bd235c14042a578a4a1c70f11b7
0903 20:05:32.440169 (+ 445us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.440371 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3446057}
W20250903 20:05:32.441771 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 86e33bd235c14042a578a4a1c70f11b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86e33bd235c14042a578a4a1c70f11b7 already in progress: creating tablet
I20250903 20:05:32.442935 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 911) took 3418 ms. Trace:
I20250903 20:05:32.386883 27693 ts_tablet_manager.cc:1428] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.048s user 0.003s sys 0.001s
I20250903 20:05:32.433846 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8fff36d5a80e45af8a256c3908555e08" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:32.442009 28567 consensus_queue.cc:1035] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:32.443353 27684 rpcz_store.cc:276] 0903 20:05:29.024310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:29.024377 (+ 67us) service_pool.cc:225] Handling call
0903 20:05:32.442411 (+3418034us) ts_tablet_manager.cc:1145] Deleting tablet 9766cea9d5bb407a9900c7e3bb4d2641
0903 20:05:32.442424 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.442914 (+ 490us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3417708}
W20250903 20:05:32.444687 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 9766cea9d5bb407a9900c7e3bb4d2641 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9766cea9d5bb407a9900c7e3bb4d2641 already in progress: creating tablet
I20250903 20:05:32.446295 27508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8fff36d5a80e45af8a256c3908555e08" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:32.446558 27508 raft_consensus.cc:2391] T 8fff36d5a80e45af8a256c3908555e08 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89f8f0ee68b940db9ce250c0604f934a in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
I20250903 20:05:32.446993 27183 leader_election.cc:304] T 8fff36d5a80e45af8a256c3908555e08 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:32.447741 28947 raft_consensus.cc:3058] T 8fff36d5a80e45af8a256c3908555e08 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:32.448704 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 912) took 3392 ms. Trace:
I20250903 20:05:32.448791 27684 rpcz_store.cc:276] 0903 20:05:29.056468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:29.056901 (+ 433us) service_pool.cc:225] Handling call
0903 20:05:32.445239 (+3388338us) ts_tablet_manager.cc:1145] Deleting tablet bf459db3327640f88574fca7b5f2ca98
0903 20:05:32.448541 (+ 3302us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.448682 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3388008}
I20250903 20:05:32.449131 27684 tablet_replica.cc:331] T 8a223f75045345ae833f52bcc8f239f9 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:32.449350 27684 raft_consensus.cc:2241] T 8a223f75045345ae833f52bcc8f239f9 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
W20250903 20:05:32.449373 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet bf459db3327640f88574fca7b5f2ca98 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf459db3327640f88574fca7b5f2ca98 already in progress: creating tablet
I20250903 20:05:32.449535 27684 raft_consensus.cc:2270] T 8a223f75045345ae833f52bcc8f239f9 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.450198 27684 ts_tablet_manager.cc:1905] T 8a223f75045345ae833f52bcc8f239f9 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:32.450953 28947 raft_consensus.cc:2747] T 8fff36d5a80e45af8a256c3908555e08 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:32.456352 27687 ts_tablet_manager.cc:1918] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:32.456590 27687 log.cc:1199] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/1f6cd4d475f147fe8dfcc7e3a73dedf8
I20250903 20:05:32.457072 27687 ts_tablet_manager.cc:1939] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:32.459195 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 977) took 1175 ms. Trace:
I20250903 20:05:32.459542 27687 rpcz_store.cc:276] 0903 20:05:31.283824 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.284013 (+ 189us) service_pool.cc:225] Handling call
0903 20:05:32.139760 (+855747us) ts_tablet_manager.cc:1145] Deleting tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8
0903 20:05:32.139770 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.350015 (+210245us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.456222 (+106207us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.459130 (+ 2908us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":855590}
I20250903 20:05:32.461230 27693 tablet_bootstrap.cc:492] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:32.463526 27693 tablet_bootstrap.cc:654] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:32.462852 27695 ts_tablet_manager.cc:1428] T 5002b7e8118c49b6a247cb8dc4ca1aea P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.197s user 0.005s sys 0.002s
I20250903 20:05:32.467171 27692 tablet_bootstrap.cc:492] T 5002b7e8118c49b6a247cb8dc4ca1aea P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:32.467430 27692 ts_tablet_manager.cc:1397] T 5002b7e8118c49b6a247cb8dc4ca1aea P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.086s user 0.006s sys 0.000s
I20250903 20:05:32.468681 27059 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_cb85e96534984e88a664c7183a732fe4 [id=6952319d78524829b3bc6c3de6425a67] alter complete (version 2)
I20250903 20:05:32.468776 27692 raft_consensus.cc:357] T 5002b7e8118c49b6a247cb8dc4ca1aea P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.469090 27692 raft_consensus.cc:383] T 5002b7e8118c49b6a247cb8dc4ca1aea P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:32.469173 27692 raft_consensus.cc:738] T 5002b7e8118c49b6a247cb8dc4ca1aea P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:32.469506 27692 consensus_queue.cc:260] T 5002b7e8118c49b6a247cb8dc4ca1aea P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.470392 27692 ts_tablet_manager.cc:1428] T 5002b7e8118c49b6a247cb8dc4ca1aea P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250903 20:05:32.471295 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 22bc87d1a4804b6c80f77f9692500d63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:32.471560 27059 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.26.65.65:44885 (request call id 163) took 1039 ms (client timeout 1000 ms). Trace:
I20250903 20:05:32.472219 27221 tablet_service.cc:1515] Processing DeleteTablet for tablet b7a6d5d8a97a4769a81ec7224d591f8c with delete_type TABLET_DATA_DELETED (replaced by tablet 79c0cc0c30ab42eab0be415e3db9589f) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:32.471661 27059 rpcz_store.cc:269] 0903 20:05:31.432197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.443232 (+ 11035us) service_pool.cc:225] Handling call
0903 20:05:32.471491 (+1028259us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:32.112644 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:32.112726 (+ 82us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:32.112731 (+ 5us) write_op.cc:435] Acquired schema lock
0903 20:05:32.112733 (+ 2us) tablet.cc:660] Decoding operations
0903 20:05:32.112826 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:32.112834 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:32.112838 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0903 20:05:32.112914 (+ 76us) tablet.cc:699] Row locks acquired
0903 20:05:32.112917 (+ 3us) write_op.cc:260] PREPARE: finished
0903 20:05:32.112942 (+ 25us) write_op.cc:270] Start()
0903 20:05:32.112964 (+ 22us) write_op.cc:276] Timestamp: P: 1756929932112898 usec, L: 0
0903 20:05:32.112968 (+ 4us) op_driver.cc:348] REPLICATION: starting
0903 20:05:32.113094 (+ 126us) log.cc:844] Serialized 907 byte log entry
0903 20:05:32.298806 (+185712us) op_driver.cc:464] REPLICATION: finished
0903 20:05:32.300134 (+ 1328us) write_op.cc:301] APPLY: starting
0903 20:05:32.300153 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:32.300213 (+ 60us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:32.300240 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:32.300321 (+ 81us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:32.300363 (+ 42us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0903 20:05:32.300372 (+ 9us) write_op.cc:312] APPLY: finished
0903 20:05:32.300475 (+ 103us) log.cc:844] Serialized 29 byte log entry
0903 20:05:32.300543 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:32.300564 (+ 21us) write_op.cc:454] Released schema lock
0903 20:05:32.300571 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0903 20:05:32.389235 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:32.394613 (+ 5378us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:32.394621 (+ 8us) write_op.cc:435] Acquired schema lock
0903 20:05:32.394624 (+ 3us) tablet.cc:660] Decoding operations
0903 20:05:32.397334 (+ 2710us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:32.397349 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:32.397355 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0903 20:05:32.398653 (+ 1298us) tablet.cc:699] Row locks acquired
0903 20:05:32.398656 (+ 3us) write_op.cc:260] PREPARE: finished
0903 20:05:32.398934 (+ 278us) write_op.cc:270] Start()
0903 20:05:32.399534 (+ 600us) write_op.cc:276] Timestamp: P: 1756929932398891 usec, L: 0
0903 20:05:32.399544 (+ 10us) op_driver.cc:348] REPLICATION: starting
0903 20:05:32.404504 (+ 4960us) log.cc:844] Serialized 356 byte log entry
0903 20:05:32.466592 (+ 62088us) op_driver.cc:464] REPLICATION: finished
0903 20:05:32.467812 (+ 1220us) write_op.cc:301] APPLY: starting
0903 20:05:32.467836 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:32.467886 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:32.467929 (+ 43us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:32.467994 (+ 65us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:32.468033 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0903 20:05:32.468044 (+ 11us) write_op.cc:312] APPLY: finished
0903 20:05:32.468141 (+ 97us) log.cc:844] Serialized 23 byte log entry
0903 20:05:32.468476 (+ 335us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:32.468502 (+ 26us) write_op.cc:454] Released schema lock
0903 20:05:32.468511 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":199219,"reported_tablets":39,"child_traces":[["op",{"apply.queue_time_us":542,"apply.run_cpu_time_us":512,"apply.run_wall_time_us":513,"num_ops":2,"prepare.queue_time_us":12914,"prepare.run_cpu_time_us":494,"prepare.run_wall_time_us":493,"replication_time_us":185078}],["op",{"apply.queue_time_us":1137,"num_ops":1,"prepare.queue_time_us":82971,"prepare.run_cpu_time_us":1362,"prepare.run_wall_time_us":17306,"replication_time_us":67004}]]}
I20250903 20:05:32.472394 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bf69315f07a470bb2b897ad7c590729 with delete_type TABLET_DATA_DELETED (replaced by tablet 17514a2af39b402abbe8263d7c1abc7f) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:32.472563 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet 63a1f26fac3b4b939ebb79c6a07d8f8e with delete_type TABLET_DATA_DELETED (replaced by tablet 4789bd72aa454b639df76bd5ad379850) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:32.472771 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 512908bc4d6941ec81b145ee5795405f with delete_type TABLET_DATA_DELETED (replaced by tablet 1598824a733a4a93bed043e81b86abf3) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:32.473814 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet 541278ca9ffb4e41b341139c2faa337f with delete_type TABLET_DATA_DELETED (replaced by tablet e9126e4a18c546c28927f22c9f16426e) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:32.474737 27684 ts_tablet_manager.cc:1918] T 8a223f75045345ae833f52bcc8f239f9 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:32.475097 27684 log.cc:1199] T 8a223f75045345ae833f52bcc8f239f9 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/8a223f75045345ae833f52bcc8f239f9
I20250903 20:05:32.475813 27684 ts_tablet_manager.cc:1939] T 8a223f75045345ae833f52bcc8f239f9 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:32.476706 27692 tablet_bootstrap.cc:492] T a253e3399622409e92299069b19d3698 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:32.478195 27692 tablet_bootstrap.cc:654] T a253e3399622409e92299069b19d3698 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:32.480376 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 913) took 3313 ms. Trace:
I20250903 20:05:32.480803 27684 rpcz_store.cc:276] 0903 20:05:29.166875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:29.166942 (+ 67us) service_pool.cc:225] Handling call
0903 20:05:32.449084 (+3282142us) ts_tablet_manager.cc:1145] Deleting tablet 8a223f75045345ae833f52bcc8f239f9
0903 20:05:32.449093 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.453366 (+ 4273us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.474567 (+ 21201us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.480314 (+ 5747us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3281903}
I20250903 20:05:32.482081 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 8a223f75045345ae833f52bcc8f239f9 (table default.Table_04bd610b6ae24395b30dc33c4d118258 [id=02a2c253dfb64f2cac5f70bde132aceb]) successfully deleted
I20250903 20:05:32.480839 27695 tablet_bootstrap.cc:492] T 06a79706f1184fa49d10182ba669b0ab P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:32.483326 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 921) took 3182 ms. Trace:
I20250903 20:05:32.483692 27684 rpcz_store.cc:276] 0903 20:05:29.301002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:29.301060 (+ 58us) service_pool.cc:225] Handling call
0903 20:05:32.482707 (+3181647us) ts_tablet_manager.cc:1145] Deleting tablet 53b00774b745462091321ea47fcc2114
0903 20:05:32.482772 (+ 65us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.483215 (+ 443us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3181235}
W20250903 20:05:32.484802 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 53b00774b745462091321ea47fcc2114 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53b00774b745462091321ea47fcc2114 already in progress: creating tablet
I20250903 20:05:32.485944 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 922) took 3170 ms. Trace:
I20250903 20:05:32.486559 27684 rpcz_store.cc:276] 0903 20:05:29.315756 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:29.315816 (+ 60us) service_pool.cc:225] Handling call
0903 20:05:32.485405 (+3169589us) ts_tablet_manager.cc:1145] Deleting tablet 31186292b25f4c3cb23597f66dcbd282
0903 20:05:32.485416 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.485815 (+ 399us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3169278}
W20250903 20:05:32.487633 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 31186292b25f4c3cb23597f66dcbd282 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31186292b25f4c3cb23597f66dcbd282 already in progress: creating tablet
I20250903 20:05:32.488742 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 923) took 2837 ms. Trace:
W20250903 20:05:32.489094 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ec8cea2d64b642689d0bc11295745ef0 consensus state has no leader
I20250903 20:05:32.489193 27684 rpcz_store.cc:276] 0903 20:05:29.651741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:29.664777 (+ 13036us) service_pool.cc:225] Handling call
0903 20:05:32.488230 (+2823453us) ts_tablet_manager.cc:1145] Deleting tablet cc1bc205bf39447bb9c3c7863a0959cf
0903 20:05:32.488244 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.488680 (+ 436us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2823098}
W20250903 20:05:32.490437 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet cc1bc205bf39447bb9c3c7863a0959cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc1bc205bf39447bb9c3c7863a0959cf already in progress: creating tablet
I20250903 20:05:32.491103 27684 tablet_replica.cc:331] T d1c96296a579440c887ba6c2a93f7204 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:32.491887 27684 raft_consensus.cc:2241] T d1c96296a579440c887ba6c2a93f7204 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:32.492482 27684 raft_consensus.cc:2270] T d1c96296a579440c887ba6c2a93f7204 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.494647 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 653eade3172344d7aa801edf906ae1d4
I20250903 20:05:32.495220 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e0f0013beb934622b6c69bbf05cc1ede" } new_table_name: "default.Table_df7e7287722a45988929dfd3f0ff3eb6" modify_external_catalogs: true
I20250903 20:05:32.495466 27692 tablet_bootstrap.cc:492] T a253e3399622409e92299069b19d3698 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:32.495919 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet 653eade3172344d7aa801edf906ae1d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:32.496309 27692 ts_tablet_manager.cc:1397] T a253e3399622409e92299069b19d3698 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.002s
I20250903 20:05:32.496856 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_de5aee21a7e341a89dc0e5af838dfbcc" } new_table_name: "default.Table_3ee4dfa744d34536bab52e4ba4f64dbd" modify_external_catalogs: true
I20250903 20:05:32.498077 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet e35574d360334150bb494e13f55d3656
I20250903 20:05:32.499275 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet e35574d360334150bb494e13f55d3656 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:32.500306 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f632fa586b124749a0763c5c17dfb9b3" } new_table_name: "default.Table_7b9de0532ff4407086b7f341fedfb3af" modify_external_catalogs: true
I20250903 20:05:32.500317 27692 raft_consensus.cc:357] T a253e3399622409e92299069b19d3698 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.501152 27692 raft_consensus.cc:383] T a253e3399622409e92299069b19d3698 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:32.501494 27692 raft_consensus.cc:738] T a253e3399622409e92299069b19d3698 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:32.502626 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_a765aae93cdc4ade9f0d2be91c3b3ecf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:32.502312 27692 consensus_queue.cc:260] T a253e3399622409e92299069b19d3698 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:32.503248 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a765aae93cdc4ade9f0d2be91c3b3ecf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:32.500550 27684 ts_tablet_manager.cc:1905] T d1c96296a579440c887ba6c2a93f7204 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:32.504899 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet e35574d360334150bb494e13f55d3656 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:32.505748 27183 consensus_peers.cc:489] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4f3230f50ec4c95b528fead8c191224. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:32.506100 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_348db3632ae743b9b7d995887e840104"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:32.506559 27692 ts_tablet_manager.cc:1428] T a253e3399622409e92299069b19d3698 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.010s user 0.001s sys 0.002s
W20250903 20:05:32.506727 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_348db3632ae743b9b7d995887e840104 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:32.507227 27692 tablet_bootstrap.cc:492] T 06a79706f1184fa49d10182ba669b0ab P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:32.508145 27350 tablet_service.cc:1515] Processing DeleteTablet for tablet e35574d360334150bb494e13f55d3656 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.508471 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet 653eade3172344d7aa801edf906ae1d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:32.508729 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 653eade3172344d7aa801edf906ae1d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.510463 27692 tablet_bootstrap.cc:654] T 06a79706f1184fa49d10182ba669b0ab P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:32.519577 27695 tablet_bootstrap.cc:654] T 06a79706f1184fa49d10182ba669b0ab P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:32.523592 28808 consensus_queue.cc:1035] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:32.526400 27057 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 67ca1dcf6fc7435db555392235f9900f (table default.Table_d3945d5a36d84041b35394333dc704f9 [id=5568f788d1414c83b4359b5adf28632a]). Expected version 2 got 0
I20250903 20:05:32.527133 27057 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 9e98cd1d9bd246e182e2969d147e68bf (table default.Table_28a829a9988442049d50e6ae254e6274 [id=b27f08d56c3c43e6a09a465fdda4d3db]). Expected version 1 got 0
I20250903 20:05:32.527438 27057 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet fc634116a78f45dd833034ddda90e9bf (table default.Table_78e7a5b76ebc469396d8a3d670ab63ee [id=fb43441ca8f2415d99a628e81ef1642f]). Expected version 2 got 0
I20250903 20:05:32.528337 27057 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 2059ccbd6d884085886d7d564afe03aa (table default.Table_ea5cb66a574f4466bf7e7f635f22b426 [id=91abc1158ea047849f398926e3695927]). Expected version 1 got 0
I20250903 20:05:32.530306 27685 ts_tablet_manager.cc:1918] T 961ade36f7d14eba91bb16d5eae2e254 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:32.530494 27685 log.cc:1199] T 961ade36f7d14eba91bb16d5eae2e254 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/961ade36f7d14eba91bb16d5eae2e254
I20250903 20:05:32.530985 27685 ts_tablet_manager.cc:1939] T 961ade36f7d14eba91bb16d5eae2e254 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:32.531404 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 797) took 5914 ms. Trace:
I20250903 20:05:32.531560 27685 rpcz_store.cc:276] 0903 20:05:26.616635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.617210 (+ 575us) service_pool.cc:225] Handling call
0903 20:05:32.410729 (+5793519us) ts_tablet_manager.cc:1145] Deleting tablet 961ade36f7d14eba91bb16d5eae2e254
0903 20:05:32.410753 (+ 24us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.525656 (+114903us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.530161 (+ 4505us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.531386 (+ 1225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5793169}
I20250903 20:05:32.532310 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 961ade36f7d14eba91bb16d5eae2e254 (table default.Table_459903503b5148d09d3cb8f2f5519fab [id=3c69612aaf164dc899dcc252d85636f6]) successfully deleted
I20250903 20:05:32.532721 27685 tablet_replica.cc:331] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:32.533037 27685 raft_consensus.cc:2241] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:32.534114 27685 raft_consensus.cc:2270] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.534873 27685 ts_tablet_manager.cc:1905] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:32.538538 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5934304e4ebd46f0a04e7c58c6607379 consensus state has no leader
I20250903 20:05:32.538976 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_73a4ac2afbad4654ac623cd581a45eb2" } modify_external_catalogs: true
I20250903 20:05:32.539155 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_73a4ac2afbad4654ac623cd581a45eb2" } modify_external_catalogs: true
I20250903 20:05:32.541409 27684 ts_tablet_manager.cc:1918] T d1c96296a579440c887ba6c2a93f7204 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250903 20:05:32.541981 27066 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dd27901224d6442389e8fc09c6b498ab consensus state has no leader
I20250903 20:05:32.542761 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2e9e07bea789491685a849ec5dea7d73" } modify_external_catalogs: true
I20250903 20:05:32.543001 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_191beed070214f6eac98ee16e25e34f8" } modify_external_catalogs: true
I20250903 20:05:32.543177 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_191beed070214f6eac98ee16e25e34f8" } modify_external_catalogs: true
W20250903 20:05:32.543402 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a785adcd0bd943fc97b201b6d29073f4 consensus state has no leader
I20250903 20:05:32.543938 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b7bf9efdd5044f078553d1b4a33825fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:32.544703 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7bf9efdd5044f078553d1b4a33825fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:32.543022 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2e9e07bea789491685a849ec5dea7d73" } modify_external_catalogs: true
I20250903 20:05:32.547545 27685 ts_tablet_manager.cc:1918] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:32.547729 27685 log.cc:1199] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/1bcb9d5f010f4297bab78c0183889238
I20250903 20:05:32.548151 27685 ts_tablet_manager.cc:1939] T 1bcb9d5f010f4297bab78c0183889238 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:32.549638 28874 consensus_queue.cc:1035] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:32.550642 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_77dba1c3fea74301973e35bf475df57a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:32.551354 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77dba1c3fea74301973e35bf475df57a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:32.553239 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 801) took 5733 ms. Trace:
I20250903 20:05:32.545372 27692 tablet_bootstrap.cc:492] T 06a79706f1184fa49d10182ba669b0ab P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:32.553444 27692 ts_tablet_manager.cc:1397] T 06a79706f1184fa49d10182ba669b0ab P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.009s
I20250903 20:05:32.553392 27685 rpcz_store.cc:276] 0903 20:05:26.819766 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:26.819825 (+ 59us) service_pool.cc:225] Handling call
0903 20:05:32.532671 (+5712846us) ts_tablet_manager.cc:1145] Deleting tablet 1bcb9d5f010f4297bab78c0183889238
0903 20:05:32.532681 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.540128 (+ 7447us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.547431 (+ 7303us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.553222 (+ 5791us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5712635}
I20250903 20:05:32.554009 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 1bcb9d5f010f4297bab78c0183889238 (table default.Table_adf05e59a7be478b89666c3162ac83f1 [id=e69321e60a3a4e71a463a0011c0c6d65]) successfully deleted
I20250903 20:05:32.554327 27692 raft_consensus.cc:357] T 06a79706f1184fa49d10182ba669b0ab P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:32.554505 27692 raft_consensus.cc:383] T 06a79706f1184fa49d10182ba669b0ab P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:32.554563 27692 raft_consensus.cc:738] T 06a79706f1184fa49d10182ba669b0ab P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:32.554752 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 815) took 5530 ms. Trace:
I20250903 20:05:32.554750 27692 consensus_queue.cc:260] T 06a79706f1184fa49d10182ba669b0ab P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:32.554869 27685 rpcz_store.cc:276] 0903 20:05:27.024552 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.024624 (+ 72us) service_pool.cc:225] Handling call
0903 20:05:32.554375 (+5529751us) ts_tablet_manager.cc:1145] Deleting tablet 553443eda16c44fa84b1db22b1363808
0903 20:05:32.554386 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.554735 (+ 349us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5529519}
I20250903 20:05:32.555284 27692 ts_tablet_manager.cc:1428] T 06a79706f1184fa49d10182ba669b0ab P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:32.555578 27692 tablet_bootstrap.cc:492] T fc634116a78f45dd833034ddda90e9bf P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
W20250903 20:05:32.555609 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 553443eda16c44fa84b1db22b1363808 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 553443eda16c44fa84b1db22b1363808 already in progress: creating tablet
I20250903 20:05:32.555707 28567 consensus_queue.cc:1035] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:32.556140 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 816) took 5531 ms. Trace:
I20250903 20:05:32.556274 27685 rpcz_store.cc:276] 0903 20:05:27.024661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.024894 (+ 233us) service_pool.cc:225] Handling call
0903 20:05:32.556014 (+5531120us) ts_tablet_manager.cc:1145] Deleting tablet e0b9172ebb484382adb86e597f5cbc3b
0903 20:05:32.556024 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.556124 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5530980}
W20250903 20:05:32.556821 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet e0b9172ebb484382adb86e597f5cbc3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0b9172ebb484382adb86e597f5cbc3b already in progress: creating tablet
I20250903 20:05:32.557307 27692 tablet_bootstrap.cc:654] T fc634116a78f45dd833034ddda90e9bf P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:32.558279 27684 log.cc:1199] T d1c96296a579440c887ba6c2a93f7204 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/d1c96296a579440c887ba6c2a93f7204
I20250903 20:05:32.558614 27684 ts_tablet_manager.cc:1939] T d1c96296a579440c887ba6c2a93f7204 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:32.559437 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 924) took 2762 ms. Trace:
I20250903 20:05:32.559532 27684 rpcz_store.cc:276] 0903 20:05:29.797198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:29.797248 (+ 50us) service_pool.cc:225] Handling call
0903 20:05:32.490959 (+2693711us) ts_tablet_manager.cc:1145] Deleting tablet d1c96296a579440c887ba6c2a93f7204
0903 20:05:32.490971 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.514523 (+ 23552us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.541305 (+ 26782us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.559410 (+ 18105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2693362}
I20250903 20:05:32.559863 27684 tablet_replica.cc:331] T f2629f46704e4a0fa7922533928ca2ed P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:32.560050 27684 raft_consensus.cc:2241] T f2629f46704e4a0fa7922533928ca2ed P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:32.560114 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet d1c96296a579440c887ba6c2a93f7204 (table default.Table_66babebee1c9462ea057c8489d43846d [id=a4d648e314504b68bc297abdbccad46f]) successfully deleted
I20250903 20:05:32.560223 27684 raft_consensus.cc:2270] T f2629f46704e4a0fa7922533928ca2ed P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:32.560281 27183 consensus_peers.cc:489] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a223f75045345ae833f52bcc8f239f9. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:32.560806 27684 ts_tablet_manager.cc:1905] T f2629f46704e4a0fa7922533928ca2ed P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:32.560791 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 818) took 5535 ms. Trace:
I20250903 20:05:32.561028 27685 rpcz_store.cc:276] 0903 20:05:27.024899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.024946 (+ 47us) service_pool.cc:225] Handling call
0903 20:05:32.557264 (+5532318us) ts_tablet_manager.cc:1145] Deleting tablet ea3f82f4e4244219ab0919f3e341aead
0903 20:05:32.557274 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.560771 (+ 3497us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5532144}
W20250903 20:05:32.562496 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet ea3f82f4e4244219ab0919f3e341aead because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea3f82f4e4244219ab0919f3e341aead already in progress: creating tablet
I20250903 20:05:32.563143 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 817) took 5538 ms. Trace:
I20250903 20:05:32.563496 27685 rpcz_store.cc:276] 0903 20:05:27.024789 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.025089 (+ 300us) service_pool.cc:225] Handling call
0903 20:05:32.562948 (+5537859us) ts_tablet_manager.cc:1145] Deleting tablet 44fef873a0644dd988322847f333ec95
0903 20:05:32.562980 (+ 32us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.563119 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5537706}
I20250903 20:05:32.563798 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 8330ab3e4b6f4560b9e41a9df280c211
I20250903 20:05:32.564556 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet a09771c026a9474c9f61ed259d843cca
W20250903 20:05:32.566792 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 44fef873a0644dd988322847f333ec95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44fef873a0644dd988322847f333ec95 already in progress: creating tablet
I20250903 20:05:32.567301 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 8330ab3e4b6f4560b9e41a9df280c211 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:32.567844 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet a09771c026a9474c9f61ed259d843cca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.568089 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet a09771c026a9474c9f61ed259d843cca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:32.568359 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 8330ab3e4b6f4560b9e41a9df280c211 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.568373 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet a09771c026a9474c9f61ed259d843cca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:32.569257 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 819) took 5542 ms. Trace:
I20250903 20:05:32.569824 27685 rpcz_store.cc:276] 0903 20:05:27.026352 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.026402 (+ 50us) service_pool.cc:225] Handling call
0903 20:05:32.569075 (+5542673us) ts_tablet_manager.cc:1145] Deleting tablet 81852d8e06ae4c6bb715a88d8c398bfc
0903 20:05:32.569091 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.569233 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5542494}
I20250903 20:05:32.570436 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet 8330ab3e4b6f4560b9e41a9df280c211 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:32.570657 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 81852d8e06ae4c6bb715a88d8c398bfc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81852d8e06ae4c6bb715a88d8c398bfc already in progress: creating tablet
I20250903 20:05:32.571086 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 820) took 5350 ms. Trace:
I20250903 20:05:32.571342 27064 master_service.cc:948] ReplaceTablet: received request to replace tablet 931404d9b3844d908baaed3190d57b56 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:32.571483 27685 rpcz_store.cc:276] 0903 20:05:27.220284 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.220343 (+ 59us) service_pool.cc:225] Handling call
0903 20:05:32.570973 (+5350630us) ts_tablet_manager.cc:1145] Deleting tablet ce3be9d44147420d953e716a347352c9
0903 20:05:32.570984 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.571071 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5350406}
I20250903 20:05:32.572890 27685 tablet_replica.cc:331] T 6788b1683f984949816edfc5b68de8db P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:32.573588 27685 raft_consensus.cc:2241] T 6788b1683f984949816edfc5b68de8db P 89f8f0ee68b940db9ce250c0604f934a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:32.573813 27685 raft_consensus.cc:2270] T 6788b1683f984949816edfc5b68de8db P 89f8f0ee68b940db9ce250c0604f934a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.574448 27685 ts_tablet_manager.cc:1905] T 6788b1683f984949816edfc5b68de8db P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:32.575001 27684 ts_tablet_manager.cc:1918] T f2629f46704e4a0fa7922533928ca2ed P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:32.574857 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_d1ef54568fa7465bbb2e459a8d50cf6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:32.575132 27684 log.cc:1199] T f2629f46704e4a0fa7922533928ca2ed P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/f2629f46704e4a0fa7922533928ca2ed
I20250903 20:05:32.575532 27684 ts_tablet_manager.cc:1939] T f2629f46704e4a0fa7922533928ca2ed P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:32.576470 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 928) took 2462 ms. Trace:
I20250903 20:05:32.576593 27684 rpcz_store.cc:276] 0903 20:05:30.113748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.114311 (+ 563us) service_pool.cc:225] Handling call
0903 20:05:32.559821 (+2445510us) ts_tablet_manager.cc:1145] Deleting tablet f2629f46704e4a0fa7922533928ca2ed
0903 20:05:32.559829 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.571765 (+ 11936us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.574898 (+ 3133us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.576440 (+ 1542us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2445319}
I20250903 20:05:32.576956 27684 tablet_replica.cc:331] T 90baf83357494a0aae325908c118b471 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:32.577142 27684 raft_consensus.cc:2241] T 90baf83357494a0aae325908c118b471 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:32.577236 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet f2629f46704e4a0fa7922533928ca2ed (table default.Table_ae457e422e0c4314aeaecd7755413366 [id=2051ecb140f24088911aad60fdb9082b]) successfully deleted
I20250903 20:05:32.577332 27684 raft_consensus.cc:2270] T 90baf83357494a0aae325908c118b471 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.577844 27684 ts_tablet_manager.cc:1905] T 90baf83357494a0aae325908c118b471 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:32.577841 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e289a239e8554a8891f3c7ef328c0007"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:32.578501 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e289a239e8554a8891f3c7ef328c0007 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:32.585726 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1ef54568fa7465bbb2e459a8d50cf6e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:32.586665 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4bcd65022b7b4502983b6b58809047e3 consensus state has no leader
I20250903 20:05:32.591540 27685 ts_tablet_manager.cc:1918] T 6788b1683f984949816edfc5b68de8db P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:32.591768 27685 log.cc:1199] T 6788b1683f984949816edfc5b68de8db P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/6788b1683f984949816edfc5b68de8db
I20250903 20:05:32.592164 27685 ts_tablet_manager.cc:1939] T 6788b1683f984949816edfc5b68de8db P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:32.592558 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 821) took 5344 ms. Trace:
I20250903 20:05:32.592696 27685 rpcz_store.cc:276] 0903 20:05:27.248416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.319913 (+ 71497us) service_pool.cc:225] Handling call
0903 20:05:32.572842 (+5252929us) ts_tablet_manager.cc:1145] Deleting tablet 6788b1683f984949816edfc5b68de8db
0903 20:05:32.572852 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.583167 (+ 10315us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.591422 (+ 8255us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.592542 (+ 1120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5244409}
I20250903 20:05:32.593237 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 6788b1683f984949816edfc5b68de8db (table default.Table_48f65d1e5e0b4836bb4df770bd94b4e0 [id=45702f222451484c80915421a0cdeb68]) successfully deleted
I20250903 20:05:32.593688 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 834) took 5139 ms. Trace:
I20250903 20:05:32.593804 27685 rpcz_store.cc:276] 0903 20:05:27.454543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.469877 (+ 15334us) service_pool.cc:225] Handling call
0903 20:05:32.593560 (+5123683us) ts_tablet_manager.cc:1145] Deleting tablet cf99a74acfdb4a17bbbce3a07467e9c1
0903 20:05:32.593570 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.593671 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5123493}
W20250903 20:05:32.594377 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet cf99a74acfdb4a17bbbce3a07467e9c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf99a74acfdb4a17bbbce3a07467e9c1 already in progress: creating tablet
I20250903 20:05:32.594776 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 835) took 5140 ms. Trace:
I20250903 20:05:32.594908 27685 rpcz_store.cc:276] 0903 20:05:27.454639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.454686 (+ 47us) service_pool.cc:225] Handling call
0903 20:05:32.594671 (+5139985us) ts_tablet_manager.cc:1145] Deleting tablet 30cc0d4ba8d444f1b98232f906ea0c7b
0903 20:05:32.594680 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.594758 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5112708}
I20250903 20:05:32.595157 27684 ts_tablet_manager.cc:1918] T 90baf83357494a0aae325908c118b471 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:32.595273 27684 log.cc:1199] T 90baf83357494a0aae325908c118b471 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/90baf83357494a0aae325908c118b471
W20250903 20:05:32.595470 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 30cc0d4ba8d444f1b98232f906ea0c7b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30cc0d4ba8d444f1b98232f906ea0c7b already in progress: creating tablet
I20250903 20:05:32.595638 27684 ts_tablet_manager.cc:1939] T 90baf83357494a0aae325908c118b471 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:32.596140 28808 raft_consensus.cc:491] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:32.596274 28808 raft_consensus.cc:513] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:32.596463 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 929) took 2482 ms. Trace:
I20250903 20:05:32.596552 27684 rpcz_store.cc:276] 0903 20:05:30.113873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.114603 (+ 730us) service_pool.cc:225] Handling call
0903 20:05:32.576896 (+2462293us) ts_tablet_manager.cc:1145] Deleting tablet 90baf83357494a0aae325908c118b471
0903 20:05:32.576908 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.592020 (+ 15112us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.595047 (+ 3027us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.596442 (+ 1395us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2462155}
I20250903 20:05:32.596894 27684 tablet_replica.cc:331] T 673c45f3f2a14da9bd306011d396eaca P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:32.596892 28808 leader_election.cc:290] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:32.597024 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 90baf83357494a0aae325908c118b471 (table default.Table_75f638a7c40e48e584a02949e6f864fd [id=7e9a08baa86e406c85192816b4a5d26d]) successfully deleted
I20250903 20:05:32.597095 27684 raft_consensus.cc:2241] T 673c45f3f2a14da9bd306011d396eaca P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:32.597231 27684 raft_consensus.cc:2270] T 673c45f3f2a14da9bd306011d396eaca P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.597746 27684 ts_tablet_manager.cc:1905] T 673c45f3f2a14da9bd306011d396eaca P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:32.598860 27517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "673c45f3f2a14da9bd306011d396eaca" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:32.599161 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8 (table default.Table_943bea896c744410a2a4d80ffb9ca439 [id=a950fbaecd7341989961ff3ea9a713e0]) successfully deleted
W20250903 20:05:32.599727 27184 leader_election.cc:343] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 673c45f3f2a14da9bd306011d396eaca
W20250903 20:05:32.600813 27183 leader_election.cc:343] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: Tablet not RUNNING: STOPPED
I20250903 20:05:32.600951 27183 leader_election.cc:304] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:32.601460 28947 raft_consensus.cc:2747] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:32.602144 27687 tablet_replica.cc:331] T 512908bc4d6941ec81b145ee5795405f P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:32.602388 27687 raft_consensus.cc:2241] T 512908bc4d6941ec81b145ee5795405f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:32.602596 27687 raft_consensus.cc:2270] T 512908bc4d6941ec81b145ee5795405f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.595933 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 831) took 5141 ms. Trace:
I20250903 20:05:32.603099 27685 rpcz_store.cc:276] 0903 20:05:27.454263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.461346 (+ 7083us) service_pool.cc:225] Handling call
0903 20:05:32.595822 (+5134476us) ts_tablet_manager.cc:1145] Deleting tablet 91169fff8a6649828648ac97ccf65fdc
0903 20:05:32.595833 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.595917 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5113808}
W20250903 20:05:32.603660 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 91169fff8a6649828648ac97ccf65fdc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91169fff8a6649828648ac97ccf65fdc already in progress: creating tablet
I20250903 20:05:32.604074 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 833) took 5149 ms. Trace:
I20250903 20:05:32.604241 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 931404d9b3844d908baaed3190d57b56
I20250903 20:05:32.606566 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 931404d9b3844d908baaed3190d57b56 with delete_type TABLET_DATA_DELETED (replaced by tablet 52ee6ba19a734329acde7608327f2a78) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:32.606827 27064 catalog_manager.cc:6652] ReplaceTablet: tablet 931404d9b3844d908baaed3190d57b56 deleted and replaced by tablet 52ee6ba19a734329acde7608327f2a78
I20250903 20:05:32.608335 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 931404d9b3844d908baaed3190d57b56 with delete_type TABLET_DATA_DELETED (replaced by tablet 52ee6ba19a734329acde7608327f2a78) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.616699 27687 ts_tablet_manager.cc:1905] T 512908bc4d6941ec81b145ee5795405f P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:32.618705 27695 tablet_bootstrap.cc:492] T 06a79706f1184fa49d10182ba669b0ab P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:32.607990 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet 931404d9b3844d908baaed3190d57b56 with delete_type TABLET_DATA_DELETED (replaced by tablet 52ee6ba19a734329acde7608327f2a78) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:32.618897 27695 ts_tablet_manager.cc:1397] T 06a79706f1184fa49d10182ba669b0ab P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.141s user 0.009s sys 0.009s
I20250903 20:05:32.604188 27685 rpcz_store.cc:276] 0903 20:05:27.454459 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.461660 (+ 7201us) service_pool.cc:225] Handling call
0903 20:05:32.603959 (+5142299us) ts_tablet_manager.cc:1145] Deleting tablet 112c0dccb60c4c7c9efc164d9c8443ea
0903 20:05:32.603970 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.604057 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5121879}
W20250903 20:05:32.619705 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 112c0dccb60c4c7c9efc164d9c8443ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 112c0dccb60c4c7c9efc164d9c8443ea already in progress: creating tablet
I20250903 20:05:32.619884 27695 raft_consensus.cc:357] T 06a79706f1184fa49d10182ba669b0ab P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:32.620059 27695 raft_consensus.cc:383] T 06a79706f1184fa49d10182ba669b0ab P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:32.620123 27695 raft_consensus.cc:738] T 06a79706f1184fa49d10182ba669b0ab P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:32.620311 27695 consensus_queue.cc:260] T 06a79706f1184fa49d10182ba669b0ab P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:32.620723 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 832) took 5166 ms. Trace:
I20250903 20:05:32.620983 27685 rpcz_store.cc:276] 0903 20:05:27.454373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.461862 (+ 7489us) service_pool.cc:225] Handling call
0903 20:05:32.620438 (+5158576us) ts_tablet_manager.cc:1145] Deleting tablet a686ea49a29a4dd7b6e5dce5e1f9935a
0903 20:05:32.620605 (+ 167us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.620706 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5138305}
I20250903 20:05:32.620800 27695 ts_tablet_manager.cc:1428] T 06a79706f1184fa49d10182ba669b0ab P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:32.622603 27695 tablet_bootstrap.cc:492] T 7345305820be4d32a3ceb227f045f009 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:32.624125 27695 tablet_bootstrap.cc:654] T 7345305820be4d32a3ceb227f045f009 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:32.617381 27693 tablet_bootstrap.cc:492] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:32.624843 27693 ts_tablet_manager.cc:1397] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.164s user 0.002s sys 0.003s
I20250903 20:05:32.621730 28567 consensus_queue.cc:1035] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:32.626574 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 830) took 5222 ms. Trace:
I20250903 20:05:32.626658 27685 rpcz_store.cc:276] 0903 20:05:27.403750 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.462520 (+ 58770us) service_pool.cc:225] Handling call
0903 20:05:32.626432 (+5163912us) ts_tablet_manager.cc:1145] Deleting tablet edac2355f67045a6acffaae723d9186f
0903 20:05:32.626444 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.626554 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5144264}
W20250903 20:05:32.627123 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet edac2355f67045a6acffaae723d9186f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet edac2355f67045a6acffaae723d9186f already in progress: creating tablet
I20250903 20:05:32.627485 27685 tablet_replica.cc:331] T 33fb6b357bda43e7abeaed5634183ef6 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:32.627669 27685 raft_consensus.cc:2241] T 33fb6b357bda43e7abeaed5634183ef6 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:32.629632 27685 raft_consensus.cc:2270] T 33fb6b357bda43e7abeaed5634183ef6 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.635604 27685 ts_tablet_manager.cc:1905] T 33fb6b357bda43e7abeaed5634183ef6 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:32.641369 27684 ts_tablet_manager.cc:1918] T 673c45f3f2a14da9bd306011d396eaca P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:32.642798 27684 log.cc:1199] T 673c45f3f2a14da9bd306011d396eaca P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/673c45f3f2a14da9bd306011d396eaca
I20250903 20:05:32.645207 27687 ts_tablet_manager.cc:1918] T 512908bc4d6941ec81b145ee5795405f P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:32.645893 27684 ts_tablet_manager.cc:1939] T 673c45f3f2a14da9bd306011d396eaca P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:32.651746 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 930) took 2537 ms. Trace:
I20250903 20:05:32.651991 27684 rpcz_store.cc:276] 0903 20:05:30.113982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.114759 (+ 777us) service_pool.cc:225] Handling call
0903 20:05:32.596821 (+2482062us) ts_tablet_manager.cc:1145] Deleting tablet 673c45f3f2a14da9bd306011d396eaca
0903 20:05:32.596831 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.627595 (+ 30764us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.641232 (+ 13637us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.651712 (+ 10480us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2481950}
I20250903 20:05:32.652654 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 673c45f3f2a14da9bd306011d396eaca (table default.Table_43ea308240ac4628a72434608a22b975 [id=8de1ad04c15648d9bcccfc56efc9e7fb]) successfully deleted
I20250903 20:05:32.653271 27684 tablet_replica.cc:331] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:32.653535 27684 raft_consensus.cc:2241] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:32.653966 27684 raft_consensus.cc:2270] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.655026 27684 ts_tablet_manager.cc:1905] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:32.670019 27695 tablet_bootstrap.cc:492] T 7345305820be4d32a3ceb227f045f009 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:32.670187 27695 ts_tablet_manager.cc:1397] T 7345305820be4d32a3ceb227f045f009 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.005s
I20250903 20:05:32.670915 27695 raft_consensus.cc:357] T 7345305820be4d32a3ceb227f045f009 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.671101 27695 raft_consensus.cc:383] T 7345305820be4d32a3ceb227f045f009 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:32.671160 27695 raft_consensus.cc:738] T 7345305820be4d32a3ceb227f045f009 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:32.671415 27695 consensus_queue.cc:260] T 7345305820be4d32a3ceb227f045f009 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.671880 27695 ts_tablet_manager.cc:1428] T 7345305820be4d32a3ceb227f045f009 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250903 20:05:32.672164 27695 tablet_bootstrap.cc:492] T d1c96296a579440c887ba6c2a93f7204 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:32.673533 27695 tablet_bootstrap.cc:654] T d1c96296a579440c887ba6c2a93f7204 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:32.675251 27692 tablet_bootstrap.cc:492] T fc634116a78f45dd833034ddda90e9bf P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:32.675479 27692 ts_tablet_manager.cc:1397] T fc634116a78f45dd833034ddda90e9bf P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.120s user 0.019s sys 0.021s
I20250903 20:05:32.676364 27692 raft_consensus.cc:357] T fc634116a78f45dd833034ddda90e9bf P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:32.676633 27692 raft_consensus.cc:383] T fc634116a78f45dd833034ddda90e9bf P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:32.676735 27692 raft_consensus.cc:738] T fc634116a78f45dd833034ddda90e9bf P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:32.677026 27692 consensus_queue.cc:260] T fc634116a78f45dd833034ddda90e9bf P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:32.677625 27692 ts_tablet_manager.cc:1428] T fc634116a78f45dd833034ddda90e9bf P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:32.677959 27692 tablet_bootstrap.cc:492] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:32.679824 27692 tablet_bootstrap.cc:654] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:32.687191 27695 tablet_bootstrap.cc:492] T d1c96296a579440c887ba6c2a93f7204 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:32.687352 27695 ts_tablet_manager.cc:1397] T d1c96296a579440c887ba6c2a93f7204 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.004s
I20250903 20:05:32.681154 27684 ts_tablet_manager.cc:1918] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:32.688014 27695 raft_consensus.cc:357] T d1c96296a579440c887ba6c2a93f7204 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.688192 27695 raft_consensus.cc:383] T d1c96296a579440c887ba6c2a93f7204 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:32.688246 27695 raft_consensus.cc:738] T d1c96296a579440c887ba6c2a93f7204 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:32.688151 27684 log.cc:1199] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/5f27f525fdab407280aa23df1ae71b43
I20250903 20:05:32.688478 27695 consensus_queue.cc:260] T d1c96296a579440c887ba6c2a93f7204 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.688854 27684 ts_tablet_manager.cc:1939] T 5f27f525fdab407280aa23df1ae71b43 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:32.688962 27695 ts_tablet_manager.cc:1428] T d1c96296a579440c887ba6c2a93f7204 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250903 20:05:32.689268 27695 tablet_bootstrap.cc:492] T 976322efa9464ed7a2175d3e3de51dab P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:32.689944 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 933) took 2575 ms. Trace:
I20250903 20:05:32.690136 27684 rpcz_store.cc:276] 0903 20:05:30.114251 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.114807 (+ 556us) service_pool.cc:225] Handling call
0903 20:05:32.653207 (+2538400us) ts_tablet_manager.cc:1145] Deleting tablet 5f27f525fdab407280aa23df1ae71b43
0903 20:05:32.653219 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.659746 (+ 6527us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.681030 (+ 21284us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.689920 (+ 8890us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2538199}
I20250903 20:05:32.690806 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 5f27f525fdab407280aa23df1ae71b43 (table default.Table_1c679659094d4666aaaf68f76f667890 [id=61a25739b03042f7bd5bb693b68656d5]) successfully deleted
I20250903 20:05:32.691234 27684 tablet_replica.cc:331] T a72b85dfb34047528b2b28ff0d40f828 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:32.691262 27695 tablet_bootstrap.cc:654] T 976322efa9464ed7a2175d3e3de51dab P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:32.691522 27684 raft_consensus.cc:2241] T a72b85dfb34047528b2b28ff0d40f828 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:32.691762 27684 raft_consensus.cc:2270] T a72b85dfb34047528b2b28ff0d40f828 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.691848 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_95b24ad570fd42158c2e3dc68edf37a8" } new_table_name: "default.Table_c0a41c4a69a84977906c16593147161d" modify_external_catalogs: true
I20250903 20:05:32.692394 27684 ts_tablet_manager.cc:1905] T a72b85dfb34047528b2b28ff0d40f828 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:32.694586 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_7e7bbe82a45f42f2bb0d0a2093f129ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:32.695369 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e7bbe82a45f42f2bb0d0a2093f129ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:32.697490 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_08918e8b515e4487bf4aee0a33234bd5" } modify_external_catalogs: true
I20250903 20:05:32.697719 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_08918e8b515e4487bf4aee0a33234bd5" } modify_external_catalogs: true
I20250903 20:05:32.703462 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_9ae576ba1bb8422b9b57aa5a78deddf5" } modify_external_catalogs: true
I20250903 20:05:32.703755 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_9ae576ba1bb8422b9b57aa5a78deddf5" } modify_external_catalogs: true
W20250903 20:05:32.706511 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9ac48450b5aa448f9f040e88dc2f98bd consensus state has no leader
I20250903 20:05:32.707162 27692 tablet_bootstrap.cc:492] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:32.707336 27692 ts_tablet_manager.cc:1397] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20250903 20:05:32.707278 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_566a0ec0444945f4933ae71fbd4866e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:32.707888 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_566a0ec0444945f4933ae71fbd4866e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:32.708109 27692 raft_consensus.cc:357] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.708284 27692 raft_consensus.cc:383] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:32.708331 27692 raft_consensus.cc:738] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:32.708568 27692 consensus_queue.cc:260] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.709057 27692 ts_tablet_manager.cc:1428] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:32.709345 27692 tablet_bootstrap.cc:492] T 4bcd65022b7b4502983b6b58809047e3 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:32.711138 27692 tablet_bootstrap.cc:654] T 4bcd65022b7b4502983b6b58809047e3 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:32.712783 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 9367b3a22f624c4087848099de32c080
I20250903 20:05:32.714345 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet 9367b3a22f624c4087848099de32c080 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:32.715044 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_67537b4f25e1452d98364cea8557ad42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:32.715864 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67537b4f25e1452d98364cea8557ad42 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:32.716013 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 9367b3a22f624c4087848099de32c080 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:32.716418 27684 ts_tablet_manager.cc:1918] T a72b85dfb34047528b2b28ff0d40f828 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:32.716531 27684 log.cc:1199] T a72b85dfb34047528b2b28ff0d40f828 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/a72b85dfb34047528b2b28ff0d40f828
I20250903 20:05:32.716907 27684 ts_tablet_manager.cc:1939] T a72b85dfb34047528b2b28ff0d40f828 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:32.717710 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 931) took 2603 ms. Trace:
I20250903 20:05:32.717811 27684 rpcz_store.cc:276] 0903 20:05:30.114054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.114918 (+ 864us) service_pool.cc:225] Handling call
0903 20:05:32.691164 (+2576246us) ts_tablet_manager.cc:1145] Deleting tablet a72b85dfb34047528b2b28ff0d40f828
0903 20:05:32.691175 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.713158 (+ 21983us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.716300 (+ 3142us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.717688 (+ 1388us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2576111}
I20250903 20:05:32.718130 27695 tablet_bootstrap.cc:492] T 976322efa9464ed7a2175d3e3de51dab P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:32.718318 27695 ts_tablet_manager.cc:1397] T 976322efa9464ed7a2175d3e3de51dab P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.006s
I20250903 20:05:32.718355 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet a72b85dfb34047528b2b28ff0d40f828 (table default.Table_2fa1d4a13c1840cd88c36e5fa2c6475f [id=d7b6b14b360e45a59034d2eed00e2e62]) successfully deleted
I20250903 20:05:32.719139 27695 raft_consensus.cc:357] T 976322efa9464ed7a2175d3e3de51dab P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.719338 27695 raft_consensus.cc:383] T 976322efa9464ed7a2175d3e3de51dab P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:32.719389 27695 raft_consensus.cc:738] T 976322efa9464ed7a2175d3e3de51dab P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:32.719569 27692 tablet_bootstrap.cc:492] T 4bcd65022b7b4502983b6b58809047e3 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:32.719722 27692 ts_tablet_manager.cc:1397] T 4bcd65022b7b4502983b6b58809047e3 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20250903 20:05:32.720513 27692 raft_consensus.cc:357] T 4bcd65022b7b4502983b6b58809047e3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.720692 27692 raft_consensus.cc:383] T 4bcd65022b7b4502983b6b58809047e3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:32.720758 27692 raft_consensus.cc:738] T 4bcd65022b7b4502983b6b58809047e3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:32.720954 27692 consensus_queue.cc:260] T 4bcd65022b7b4502983b6b58809047e3 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.721516 27692 ts_tablet_manager.cc:1428] T 4bcd65022b7b4502983b6b58809047e3 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:32.721573 27695 consensus_queue.cc:260] T 976322efa9464ed7a2175d3e3de51dab P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.722097 27695 ts_tablet_manager.cc:1428] T 976322efa9464ed7a2175d3e3de51dab P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250903 20:05:32.722424 27695 tablet_bootstrap.cc:492] T 200c06f296b94eab94f35f312c5a443b P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:32.724287 27692 tablet_bootstrap.cc:492] T 4c18a98d5c0f4a43ae91c9130047b681 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:32.726445 27692 tablet_bootstrap.cc:654] T 4c18a98d5c0f4a43ae91c9130047b681 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:32.727217 27060 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 9e98cd1d9bd246e182e2969d147e68bf (table default.Table_28a829a9988442049d50e6ae254e6274 [id=b27f08d56c3c43e6a09a465fdda4d3db]). Expected version 1 got 0
I20250903 20:05:32.727566 27060 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 67ca1dcf6fc7435db555392235f9900f (table default.Table_d3945d5a36d84041b35394333dc704f9 [id=5568f788d1414c83b4359b5adf28632a]). Expected version 2 got 0
I20250903 20:05:32.727661 27684 tablet_replica.cc:331] T 466e501e424e46bc98e8bbc64f97005e P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:32.727869 27684 raft_consensus.cc:2241] T 466e501e424e46bc98e8bbc64f97005e P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:32.728075 27684 raft_consensus.cc:2270] T 466e501e424e46bc98e8bbc64f97005e P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.728139 27060 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 2059ccbd6d884085886d7d564afe03aa (table default.Table_ea5cb66a574f4466bf7e7f635f22b426 [id=91abc1158ea047849f398926e3695927]). Expected version 1 got 0
I20250903 20:05:32.728667 27684 ts_tablet_manager.cc:1905] T 466e501e424e46bc98e8bbc64f97005e P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:32.730094 27060 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 976322efa9464ed7a2175d3e3de51dab (table default.Table_1ce23fe2aa6d4aceb99b7f02a2e6d04b [id=4e05a1e86a4d4319b51f2ff36f3db76b]). Expected version 1 got 0
I20250903 20:05:32.735651 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet df9d0f3662a245d7bc7cc184bc9ef3db
I20250903 20:05:32.737763 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet df9d0f3662a245d7bc7cc184bc9ef3db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:32.737875 27059 catalog_manager.cc:5582] T 67ca1dcf6fc7435db555392235f9900f P e72ef1dbcc5143cf9f04a808f101c80f reported cstate change: term changed from 0 to 1, leader changed from <none> to e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67). New cstate: current_term: 1 leader_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: HEALTHY } } }
I20250903 20:05:32.738296 27059 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 67ca1dcf6fc7435db555392235f9900f (table default.Table_d3945d5a36d84041b35394333dc704f9 [id=5568f788d1414c83b4359b5adf28632a]). Expected version 2 got 0
I20250903 20:05:32.738536 27059 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 2059ccbd6d884085886d7d564afe03aa (table default.Table_ea5cb66a574f4466bf7e7f635f22b426 [id=91abc1158ea047849f398926e3695927]). Expected version 1 got 0
I20250903 20:05:32.738857 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet df9d0f3662a245d7bc7cc184bc9ef3db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:32.739526 27183 consensus_peers.cc:489] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2629f46704e4a0fa7922533928ca2ed. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:32.740294 27684 ts_tablet_manager.cc:1918] T 466e501e424e46bc98e8bbc64f97005e P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:32.740411 27684 log.cc:1199] T 466e501e424e46bc98e8bbc64f97005e P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/466e501e424e46bc98e8bbc64f97005e
I20250903 20:05:32.740653 27059 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 9e98cd1d9bd246e182e2969d147e68bf (table default.Table_28a829a9988442049d50e6ae254e6274 [id=b27f08d56c3c43e6a09a465fdda4d3db]). Expected version 1 got 0
I20250903 20:05:32.740844 27684 ts_tablet_manager.cc:1939] T 466e501e424e46bc98e8bbc64f97005e P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:32.741058 27059 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 7345305820be4d32a3ceb227f045f009 (table default.Table_5f0199a660844c15a254a062c1540498 [id=52f969e50b9d466cbd14de6f252b7706]). Expected version 1 got 0
I20250903 20:05:32.741379 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_27ad52e9c3794a6f8b8959abe991f5f0" } new_table_name: "default.Table_6b96753f5228409aa07aa6f517d8b39a" modify_external_catalogs: true
I20250903 20:05:32.741707 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 932) took 2627 ms. Trace:
I20250903 20:05:32.741392 27059 catalog_manager.cc:5582] T 976322efa9464ed7a2175d3e3de51dab P e72ef1dbcc5143cf9f04a808f101c80f reported cstate change: term changed from 0 to 1, leader changed from <none> to e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67). New cstate: current_term: 1 leader_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:32.743803 27059 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 976322efa9464ed7a2175d3e3de51dab (table default.Table_1ce23fe2aa6d4aceb99b7f02a2e6d04b [id=4e05a1e86a4d4319b51f2ff36f3db76b]). Expected version 1 got 0
I20250903 20:05:32.744421 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0dbf85eafa35406cba9418cfa377f180" } new_table_name: "default.Table_271d79e67006404794743b3897946e9a" modify_external_catalogs: true
I20250903 20:05:32.744733 27136 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet df9d0f3662a245d7bc7cc184bc9ef3db
I20250903 20:05:32.741789 27684 rpcz_store.cc:276] 0903 20:05:30.114148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.115085 (+ 937us) service_pool.cc:225] Handling call
0903 20:05:32.721683 (+2606598us) ts_tablet_manager.cc:1145] Deleting tablet 466e501e424e46bc98e8bbc64f97005e
0903 20:05:32.727600 (+ 5917us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.731878 (+ 4278us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.740176 (+ 8298us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.741679 (+ 1503us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2606484}
I20250903 20:05:32.746402 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c4b81b1196c847c09dcf2d9281c75b26" } new_table_name: "default.Table_df4279ff3754441397a70e7e154bb125" modify_external_catalogs: true
I20250903 20:05:32.746706 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 466e501e424e46bc98e8bbc64f97005e (table default.Table_83ea6dfcee144a4fbc86931567b676bb [id=f321a32c1ac043238c2fc92bfd5ec1ae]) successfully deleted
I20250903 20:05:32.746737 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet df9d0f3662a245d7bc7cc184bc9ef3db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:32.747136 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet df9d0f3662a245d7bc7cc184bc9ef3db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:32.673004 28567 raft_consensus.cc:491] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:32.751266 28567 raft_consensus.cc:513] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:32.751631 27687 log.cc:1199] T 512908bc4d6941ec81b145ee5795405f P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/512908bc4d6941ec81b145ee5795405f
I20250903 20:05:32.751878 28567 leader_election.cc:290] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:32.714600 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 9367b3a22f624c4087848099de32c080 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.752043 27687 ts_tablet_manager.cc:1939] T 512908bc4d6941ec81b145ee5795405f P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:32.737296 27350 tablet_service.cc:1515] Processing DeleteTablet for tablet df9d0f3662a245d7bc7cc184bc9ef3db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.752498 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_12a587f34ab643568acbad040e1ce139"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:32.752990 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 979) took 1468 ms. Trace:
W20250903 20:05:32.753062 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12a587f34ab643568acbad040e1ce139 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:32.753252 27687 rpcz_store.cc:276] 0903 20:05:31.284016 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.284194 (+ 178us) service_pool.cc:225] Handling call
0903 20:05:32.602043 (+1317849us) ts_tablet_manager.cc:1145] Deleting tablet 512908bc4d6941ec81b145ee5795405f
0903 20:05:32.602055 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.641660 (+ 39605us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.645092 (+ 3432us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.752971 (+107879us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1317683}
I20250903 20:05:32.753494 27517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "06a79706f1184fa49d10182ba669b0ab" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:32.753686 27517 raft_consensus.cc:2466] T 06a79706f1184fa49d10182ba669b0ab P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:32.753933 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 512908bc4d6941ec81b145ee5795405f (table default.Table_6d2d1ff2291b43f8ae4c0a6a456f5014 [id=60e7bfe4938540e4a3c34b248356969f]) successfully deleted
I20250903 20:05:32.754107 27320 leader_election.cc:304] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:32.754426 27687 tablet_replica.cc:331] T b7a6d5d8a97a4769a81ec7224d591f8c P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:32.746510 27353 tablet_service.cc:1468] Processing CreateTablet for tablet ed90d6ea02ad4a97949e714c8b47a49b (DEFAULT_TABLE table=default.Table_867d45b3bc9248559147d26a7af1e231 [id=88953bd8338d45ae870a4667731ca5b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.754501 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 946) took 2235 ms. Trace:
I20250903 20:05:32.754568 27684 rpcz_store.cc:276] 0903 20:05:30.518952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.519752 (+ 800us) service_pool.cc:225] Handling call
0903 20:05:32.754363 (+2234611us) ts_tablet_manager.cc:1145] Deleting tablet 07439db2c3b8470eb2b797cc68376aa7
0903 20:05:32.754372 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.754482 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2180679}
I20250903 20:05:32.754835 27353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed90d6ea02ad4a97949e714c8b47a49b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.754907 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 947) took 2235 ms. Trace:
I20250903 20:05:32.754978 27684 rpcz_store.cc:276] 0903 20:05:30.519116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.519919 (+ 803us) service_pool.cc:225] Handling call
0903 20:05:32.754796 (+2234877us) ts_tablet_manager.cc:1145] Deleting tablet 167b359047ac457c9d165ec199cd9163
0903 20:05:32.754806 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.754890 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2181038}
I20250903 20:05:32.755157 27685 ts_tablet_manager.cc:1918] T 33fb6b357bda43e7abeaed5634183ef6 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:32.755285 27685 log.cc:1199] T 33fb6b357bda43e7abeaed5634183ef6 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/33fb6b357bda43e7abeaed5634183ef6
I20250903 20:05:32.755306 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 948) took 2213 ms. Trace:
I20250903 20:05:32.755365 27684 rpcz_store.cc:276] 0903 20:05:30.541420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.542369 (+ 949us) service_pool.cc:225] Handling call
0903 20:05:32.755168 (+2212799us) ts_tablet_manager.cc:1145] Deleting tablet 1f2124e9baae4c819aeb51cef79c9ff3
0903 20:05:32.755178 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.755289 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2173945}
I20250903 20:05:32.755630 27684 tablet_replica.cc:331] T e8bc99de4ffe43d58351d170b7c61d8a P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:32.755646 27685 ts_tablet_manager.cc:1939] T 33fb6b357bda43e7abeaed5634183ef6 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
W20250903 20:05:32.755752 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 07439db2c3b8470eb2b797cc68376aa7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07439db2c3b8470eb2b797cc68376aa7 already in progress: creating tablet
I20250903 20:05:32.755856 27684 raft_consensus.cc:2241] T e8bc99de4ffe43d58351d170b7c61d8a P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
W20250903 20:05:32.755985 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 167b359047ac457c9d165ec199cd9163 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 167b359047ac457c9d165ec199cd9163 already in progress: creating tablet
I20250903 20:05:32.756048 27684 raft_consensus.cc:2270] T e8bc99de4ffe43d58351d170b7c61d8a P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.756580 27684 ts_tablet_manager.cc:1905] T e8bc99de4ffe43d58351d170b7c61d8a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:32.746160 27348 tablet_service.cc:1515] Processing DeleteTablet for tablet df9d0f3662a245d7bc7cc184bc9ef3db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.747437 28916 raft_consensus.cc:491] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:32.758492 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "06a79706f1184fa49d10182ba669b0ab" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:32.758556 28916 raft_consensus.cc:513] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.758685 27240 raft_consensus.cc:2466] T 06a79706f1184fa49d10182ba669b0ab P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:32.754456 28567 raft_consensus.cc:2802] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:32.758890 28567 raft_consensus.cc:491] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:32.758997 28567 raft_consensus.cc:3058] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:32.759219 28916 leader_election.cc:290] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:32.759935 27361 tablet_service.cc:1468] Processing CreateTablet for tablet fb38fbf2160945cdbca04130003a4db2 (DEFAULT_TABLE table=default.Table_0570092e479a4896841a51ffcf2a88d6 [id=eb50c93863804517916f979d77f8b1bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.760028 29044 consensus_queue.cc:1035] T 67ca1dcf6fc7435db555392235f9900f P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:32.760324 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb38fbf2160945cdbca04130003a4db2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:32.760715 27320 leader_election.cc:343] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: d1c96296a579440c887ba6c2a93f7204
W20250903 20:05:32.761636 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 1f2124e9baae4c819aeb51cef79c9ff3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f2124e9baae4c819aeb51cef79c9ff3 already in progress: creating tablet
I20250903 20:05:32.762012 27061 master_service.cc:948] ReplaceTablet: received request to replace tablet 3c47c5fca0ee4349b2667494f45af5d0 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:32.762524 27348 tablet_service.cc:1468] Processing CreateTablet for tablet df9d0f3662a245d7bc7cc184bc9ef3db (DEFAULT_TABLE table=default.Table_9ae576ba1bb8422b9b57aa5a78deddf5 [id=a11f008b954749bd8e03f623cef969ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.762879 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df9d0f3662a245d7bc7cc184bc9ef3db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.763042 27692 tablet_bootstrap.cc:492] T 4c18a98d5c0f4a43ae91c9130047b681 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:32.763178 27692 ts_tablet_manager.cc:1397] T 4c18a98d5c0f4a43ae91c9130047b681 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.039s user 0.001s sys 0.007s
I20250903 20:05:32.763856 27506 tablet_service.cc:1468] Processing CreateTablet for tablet ed90d6ea02ad4a97949e714c8b47a49b (DEFAULT_TABLE table=default.Table_867d45b3bc9248559147d26a7af1e231 [id=88953bd8338d45ae870a4667731ca5b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.763999 27692 raft_consensus.cc:357] T 4c18a98d5c0f4a43ae91c9130047b681 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.764163 27692 raft_consensus.cc:383] T 4c18a98d5c0f4a43ae91c9130047b681 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:32.764245 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed90d6ea02ad4a97949e714c8b47a49b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.764230 27692 raft_consensus.cc:738] T 4c18a98d5c0f4a43ae91c9130047b681 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:32.764490 27692 consensus_queue.cc:260] T 4c18a98d5c0f4a43ae91c9130047b681 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:32.764745 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3739ac27204c4eb18747a0ae0d9254eb consensus state has no leader
I20250903 20:05:32.765017 27692 ts_tablet_manager.cc:1428] T 4c18a98d5c0f4a43ae91c9130047b681 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250903 20:05:32.765360 27692 tablet_bootstrap.cc:492] T e877d56bbbfa4c5a92d025d0311ce5c6 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:32.766165 27499 tablet_service.cc:1468] Processing CreateTablet for tablet df9d0f3662a245d7bc7cc184bc9ef3db (DEFAULT_TABLE table=default.Table_9ae576ba1bb8422b9b57aa5a78deddf5 [id=a11f008b954749bd8e03f623cef969ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.766551 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df9d0f3662a245d7bc7cc184bc9ef3db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.766850 27684 ts_tablet_manager.cc:1918] T e8bc99de4ffe43d58351d170b7c61d8a P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250903 20:05:32.766968 27684 log.cc:1199] T e8bc99de4ffe43d58351d170b7c61d8a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/e8bc99de4ffe43d58351d170b7c61d8a
I20250903 20:05:32.767498 27684 ts_tablet_manager.cc:1939] T e8bc99de4ffe43d58351d170b7c61d8a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:32.768522 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 950) took 2042 ms. Trace:
I20250903 20:05:32.768635 27684 rpcz_store.cc:276] 0903 20:05:30.725766 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.742407 (+ 16641us) service_pool.cc:225] Handling call
0903 20:05:32.755586 (+2013179us) ts_tablet_manager.cc:1145] Deleting tablet e8bc99de4ffe43d58351d170b7c61d8a
0903 20:05:32.755595 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.759575 (+ 3980us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.766729 (+ 7154us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.768488 (+ 1759us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2012795}
I20250903 20:05:32.770618 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 847) took 5129 ms. Trace:
I20250903 20:05:32.770723 27685 rpcz_store.cc:276] 0903 20:05:27.640729 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.642434 (+ 1705us) service_pool.cc:225] Handling call
0903 20:05:32.627430 (+4984996us) ts_tablet_manager.cc:1145] Deleting tablet 33fb6b357bda43e7abeaed5634183ef6
0903 20:05:32.627442 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.669721 (+ 42279us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.755032 (+ 85311us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.770586 (+ 15554us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4984806}
I20250903 20:05:32.772476 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet e8bc99de4ffe43d58351d170b7c61d8a (table default.Table_10a7df6626f24353ae6661cbbdf3efb0 [id=416ff29f66d14564826a6734dde1c604]) successfully deleted
I20250903 20:05:32.773372 27504 tablet_service.cc:1468] Processing CreateTablet for tablet fb38fbf2160945cdbca04130003a4db2 (DEFAULT_TABLE table=default.Table_0570092e479a4896841a51ffcf2a88d6 [id=eb50c93863804517916f979d77f8b1bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.773350 27222 tablet_service.cc:1468] Processing CreateTablet for tablet ed90d6ea02ad4a97949e714c8b47a49b (DEFAULT_TABLE table=default.Table_867d45b3bc9248559147d26a7af1e231 [id=88953bd8338d45ae870a4667731ca5b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.773782 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb38fbf2160945cdbca04130003a4db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.773787 27226 tablet_service.cc:1468] Processing CreateTablet for tablet df9d0f3662a245d7bc7cc184bc9ef3db (DEFAULT_TABLE table=default.Table_9ae576ba1bb8422b9b57aa5a78deddf5 [id=a11f008b954749bd8e03f623cef969ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.775063 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed90d6ea02ad4a97949e714c8b47a49b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.775506 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 33fb6b357bda43e7abeaed5634183ef6 (table default.Table_7a6b693ddf594ff6821359a4a7fbee01 [id=fb19a730ea6343fbba98571261d7a894]) successfully deleted
I20250903 20:05:32.776628 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 9ed340a66af14ba18bdc3019703bc5e1 (DEFAULT_TABLE table=default.Table_bbc905e2f2804b9ead3a69b83cf50fde [id=999c439c64e044de8ed3638f58608b02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.776849 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 9bda74f7a49a482bbd1aac89b0cec8c5 (DEFAULT_TABLE table=default.Table_05b4f4b3f80c45df89e4f38ce76f42a0 [id=046828b2c9904b6c89fc2da5fea34373]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.776997 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ed340a66af14ba18bdc3019703bc5e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.777150 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bda74f7a49a482bbd1aac89b0cec8c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.778419 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df9d0f3662a245d7bc7cc184bc9ef3db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.778522 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 951) took 1764 ms. Trace:
I20250903 20:05:32.778605 27684 rpcz_store.cc:276] 0903 20:05:31.013856 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.013913 (+ 57us) service_pool.cc:225] Handling call
0903 20:05:32.778359 (+1764446us) ts_tablet_manager.cc:1145] Deleting tablet 90776f6369744cec8ce380b7695c44f1
0903 20:05:32.778370 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.778496 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1764207}
I20250903 20:05:32.779124 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 952) took 1680 ms. Trace:
I20250903 20:05:32.779196 27684 rpcz_store.cc:276] 0903 20:05:31.099147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.099217 (+ 70us) service_pool.cc:225] Handling call
0903 20:05:32.779020 (+1679803us) ts_tablet_manager.cc:1145] Deleting tablet 96c9875830b246b3a9117536254cbc4e
0903 20:05:32.779028 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.779109 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1679582}
I20250903 20:05:32.779611 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 965) took 1188 ms. Trace:
I20250903 20:05:32.779678 27684 rpcz_store.cc:276] 0903 20:05:31.591003 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.591058 (+ 55us) service_pool.cc:225] Handling call
0903 20:05:32.779519 (+1188461us) ts_tablet_manager.cc:1145] Deleting tablet 17514a2af39b402abbe8263d7c1abc7f
0903 20:05:32.779528 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.779598 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1188236}
I20250903 20:05:32.780053 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 964) took 1189 ms. Trace:
I20250903 20:05:32.780117 27684 rpcz_store.cc:276] 0903 20:05:31.590873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.592766 (+ 1893us) service_pool.cc:225] Handling call
0903 20:05:32.779962 (+1187196us) ts_tablet_manager.cc:1145] Deleting tablet 3f303233e6c347289650c6cf83f8127f
0903 20:05:32.779972 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.780039 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1187008}
I20250903 20:05:32.780537 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 963) took 1193 ms. Trace:
I20250903 20:05:32.780604 27684 rpcz_store.cc:276] 0903 20:05:31.586845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.600229 (+ 13384us) service_pool.cc:225] Handling call
0903 20:05:32.780431 (+1180202us) ts_tablet_manager.cc:1145] Deleting tablet e64c4a9179b6491ba92df75e0cc1bc50
0903 20:05:32.780440 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.780511 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1178133}
I20250903 20:05:32.781183 27684 tablet_replica.cc:331] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:32.781458 27684 raft_consensus.cc:2241] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:32.781838 27684 raft_consensus.cc:2270] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.776626 27215 tablet_service.cc:1468] Processing CreateTablet for tablet fb38fbf2160945cdbca04130003a4db2 (DEFAULT_TABLE table=default.Table_0570092e479a4896841a51ffcf2a88d6 [id=eb50c93863804517916f979d77f8b1bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.782536 27684 ts_tablet_manager.cc:1905] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:32.782625 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb38fbf2160945cdbca04130003a4db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.782857 29044 raft_consensus.cc:491] T 7345305820be4d32a3ceb227f045f009 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:32.783020 29044 raft_consensus.cc:513] T 7345305820be4d32a3ceb227f045f009 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.783654 29044 leader_election.cc:290] T 7345305820be4d32a3ceb227f045f009 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:32.784665 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7345305820be4d32a3ceb227f045f009" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:32.784849 27249 raft_consensus.cc:2466] T 7345305820be4d32a3ceb227f045f009 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:32.785391 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b13e911191fc47e1a83d725043ba1f30" } modify_external_catalogs: true
I20250903 20:05:32.785646 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b13e911191fc47e1a83d725043ba1f30" } modify_external_catalogs: true
I20250903 20:05:32.785709 27462 leader_election.cc:304] T 7345305820be4d32a3ceb227f045f009 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:32.786000 29044 raft_consensus.cc:2802] T 7345305820be4d32a3ceb227f045f009 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:32.786108 29044 raft_consensus.cc:491] T 7345305820be4d32a3ceb227f045f009 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:32.786188 29044 raft_consensus.cc:3058] T 7345305820be4d32a3ceb227f045f009 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:32.786456 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1c96296a579440c887ba6c2a93f7204" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:32.786624 27240 raft_consensus.cc:2466] T d1c96296a579440c887ba6c2a93f7204 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:32.787321 27321 leader_election.cc:304] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:32.788122 29053 raft_consensus.cc:2802] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:32.788220 29053 raft_consensus.cc:491] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:32.788288 29053 raft_consensus.cc:3058] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:32.788538 27352 tablet_service.cc:1468] Processing CreateTablet for tablet 9ed340a66af14ba18bdc3019703bc5e1 (DEFAULT_TABLE table=default.Table_bbc905e2f2804b9ead3a69b83cf50fde [id=999c439c64e044de8ed3638f58608b02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.788756 29044 raft_consensus.cc:513] T 7345305820be4d32a3ceb227f045f009 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.789004 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 8e37674dd6f94feab26741033409f23c (DEFAULT_TABLE table=default.Table_f573c7cc8d7240c0bf60622441b33a69 [id=41adb50030d24515b6896b0159fcc1b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.789388 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e37674dd6f94feab26741033409f23c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.790300 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7345305820be4d32a3ceb227f045f009" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:32.790480 27239 raft_consensus.cc:3058] T 7345305820be4d32a3ceb227f045f009 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:32.790573 27684 ts_tablet_manager.cc:1918] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 20:05:32.790736 27684 log.cc:1199] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/c8a293c6fa034e9fb29682439cb3bcc2
I20250903 20:05:32.791160 27684 ts_tablet_manager.cc:1939] T c8a293c6fa034e9fb29682439cb3bcc2 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:32.747771 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_473b25f4188347ffba83a46006e66cf1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:32.793322 27239 raft_consensus.cc:2466] T 7345305820be4d32a3ceb227f045f009 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:32.793442 27684 tablet_replica.cc:331] T 607c9a9cbc5540e5bd0fa436fcba564c P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:32.793818 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ed340a66af14ba18bdc3019703bc5e1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:32.793817 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_473b25f4188347ffba83a46006e66cf1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:32.793769 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7345305820be4d32a3ceb227f045f009" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
W20250903 20:05:32.794402 27462 leader_election.cc:343] T 7345305820be4d32a3ceb227f045f009 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:32.794551 27684 raft_consensus.cc:2241] T 607c9a9cbc5540e5bd0fa436fcba564c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:32.794741 27684 raft_consensus.cc:2270] T 607c9a9cbc5540e5bd0fa436fcba564c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.795619 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7345305820be4d32a3ceb227f045f009" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:32.796039 28567 raft_consensus.cc:513] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:32.796422 27462 leader_election.cc:304] T 7345305820be4d32a3ceb227f045f009 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:32.796646 28567 leader_election.cc:290] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:32.796715 28710 raft_consensus.cc:2802] T 7345305820be4d32a3ceb227f045f009 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:32.796856 28710 raft_consensus.cc:695] T 7345305820be4d32a3ceb227f045f009 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
W20250903 20:05:32.796867 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 90776f6369744cec8ce380b7695c44f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90776f6369744cec8ce380b7695c44f1 already in progress: creating tablet
W20250903 20:05:32.797102 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 96c9875830b246b3a9117536254cbc4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96c9875830b246b3a9117536254cbc4e already in progress: creating tablet
I20250903 20:05:32.797150 28710 consensus_queue.cc:237] T 7345305820be4d32a3ceb227f045f009 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:32.797361 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 17514a2af39b402abbe8263d7c1abc7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17514a2af39b402abbe8263d7c1abc7f already in progress: creating tablet
W20250903 20:05:32.797565 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 3f303233e6c347289650c6cf83f8127f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f303233e6c347289650c6cf83f8127f already in progress: creating tablet
W20250903 20:05:32.798063 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet e64c4a9179b6491ba92df75e0cc1bc50 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e64c4a9179b6491ba92df75e0cc1bc50 already in progress: creating tablet
W20250903 20:05:32.798596 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet b5bff419ed31453e80ceec23ea05caad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5bff419ed31453e80ceec23ea05caad already in progress: creating tablet
I20250903 20:05:32.798871 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet c8a293c6fa034e9fb29682439cb3bcc2 (table default.Table_397211088a974a09bd7f031d5c7ca1cc [id=5cfc8827e4ca4f4a94424c48d6f71eb5]) successfully deleted
W20250903 20:05:32.799161 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 2478fe89414b4845b0ba377c1cde98b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2478fe89414b4845b0ba377c1cde98b7 already in progress: creating tablet
W20250903 20:05:32.799482 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 4e34d6d5d68e4395aed78821ea077c1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e34d6d5d68e4395aed78821ea077c1b already in progress: creating tablet
W20250903 20:05:32.799758 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 1ad2ca3e72434b69ad48e533da72e8fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ad2ca3e72434b69ad48e533da72e8fd already in progress: creating tablet
W20250903 20:05:32.800158 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 6d21f38759d743af972635f8b5b8d8d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d21f38759d743af972635f8b5b8d8d2 already in progress: creating tablet
I20250903 20:05:32.800185 27350 tablet_service.cc:1468] Processing CreateTablet for tablet 9bda74f7a49a482bbd1aac89b0cec8c5 (DEFAULT_TABLE table=default.Table_05b4f4b3f80c45df89e4f38ce76f42a0 [id=046828b2c9904b6c89fc2da5fea34373]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.800611 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bda74f7a49a482bbd1aac89b0cec8c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.801615 29044 leader_election.cc:290] T 7345305820be4d32a3ceb227f045f009 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:32.805724 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 9bda74f7a49a482bbd1aac89b0cec8c5 (DEFAULT_TABLE table=default.Table_05b4f4b3f80c45df89e4f38ce76f42a0 [id=046828b2c9904b6c89fc2da5fea34373]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.806879 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bda74f7a49a482bbd1aac89b0cec8c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.808081 27684 ts_tablet_manager.cc:1905] T 607c9a9cbc5540e5bd0fa436fcba564c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:32.809979 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d53bf94124e34deb92bb2ac1939ac70b consensus state has no leader
I20250903 20:05:32.810519 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b1d886984546451495e327fac36f5e49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:32.811197 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1d886984546451495e327fac36f5e49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:32.811483 27685 tablet_replica.cc:331] T f0efbb7eefb84ab6bee5b9625b91e298 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:32.811708 27685 raft_consensus.cc:2241] T f0efbb7eefb84ab6bee5b9625b91e298 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:32.811894 27685 raft_consensus.cc:2270] T f0efbb7eefb84ab6bee5b9625b91e298 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.812526 27685 ts_tablet_manager.cc:1905] T f0efbb7eefb84ab6bee5b9625b91e298 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:32.813583 27066 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cd14f3d496ea429c98bcb73ca2d8a47d consensus state has no leader
I20250903 20:05:32.813994 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1eee3233aac9463982bf7a5c2f3e098b" } modify_external_catalogs: true
I20250903 20:05:32.814250 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1eee3233aac9463982bf7a5c2f3e098b" } modify_external_catalogs: true
I20250903 20:05:32.816198 27695 tablet_bootstrap.cc:654] T 200c06f296b94eab94f35f312c5a443b P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:32.818569 27183 consensus_peers.cc:489] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0347eab6c8642beabdc2d7e7bbfdc9b. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:32.819253 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet a9f43781299e47a8aecc37a0b73d7a64
I20250903 20:05:32.820122 27365 tablet_service.cc:1468] Processing CreateTablet for tablet e7fd7cbf5a3048558720c7e258be308c (DEFAULT_TABLE table=default.Table_d6bdbaa18dba41bc997a529a1d2d322d [id=feda95c0e0c74dfd8803e026aae32a57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.820106 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_23a41a80bee1454e983b6dff5b8e1c8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:32.820123 27349 tablet_service.cc:1468] Processing CreateTablet for tablet c2da49d4d4ce4dc782d0b5ce8f74efbc (DEFAULT_TABLE table=default.Table_edc3f0ba095c4b65b0bb1f12d3f231c5 [id=44d492b191c84d7ba5be2dea19a8a169]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:32.820765 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23a41a80bee1454e983b6dff5b8e1c8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:32.820746 27226 tablet_service.cc:1468] Processing CreateTablet for tablet 23f315a1fa3d4c749994de7d0a0df775 (DEFAULT_TABLE table=default.Table_2f3c5f77e6bd4a0ebf0ac41e63589e79 [id=708a748c53be42049d5b4d1ca7713bc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.821051 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7fd7cbf5a3048558720c7e258be308c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.821098 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23f315a1fa3d4c749994de7d0a0df775. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.822296 27347 tablet_service.cc:1468] Processing CreateTablet for tablet fa2014bccb5e4f0a91cbc9845df4634d (DEFAULT_TABLE table=default.Table_051d82784c114e9f96c5717143bbc65d [id=2f0e4907f017416bb5790e27fc79aeb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.822363 27057 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_b44b0ef404e14710b769f7df4cd92ddc [id=fdc7899c8b334d309b834ae1daa14fa0] alter complete (version 5)
I20250903 20:05:32.822849 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa2014bccb5e4f0a91cbc9845df4634d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.824566 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet a9f43781299e47a8aecc37a0b73d7a64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.824816 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2da49d4d4ce4dc782d0b5ce8f74efbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.826922 27357 tablet_service.cc:1468] Processing CreateTablet for tablet 9e95a606cc2e4266984dbbe2969b3d13 (DEFAULT_TABLE table=default.Table_58b22c5102fc41568b4619bd19092925 [id=69e5c8d9217049c292cd63a493b03dff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.827339 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e95a606cc2e4266984dbbe2969b3d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.831009 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 8e37674dd6f94feab26741033409f23c (DEFAULT_TABLE table=default.Table_f573c7cc8d7240c0bf60622441b33a69 [id=41adb50030d24515b6896b0159fcc1b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.831446 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e37674dd6f94feab26741033409f23c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.832448 27213 tablet_service.cc:1468] Processing CreateTablet for tablet e7fd7cbf5a3048558720c7e258be308c (DEFAULT_TABLE table=default.Table_d6bdbaa18dba41bc997a529a1d2d322d [id=feda95c0e0c74dfd8803e026aae32a57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.832558 27221 tablet_service.cc:1515] Processing DeleteTablet for tablet a9f43781299e47a8aecc37a0b73d7a64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:32.832780 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7fd7cbf5a3048558720c7e258be308c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.832769 27684 ts_tablet_manager.cc:1918] T 607c9a9cbc5540e5bd0fa436fcba564c P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:32.832904 27684 log.cc:1199] T 607c9a9cbc5540e5bd0fa436fcba564c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/607c9a9cbc5540e5bd0fa436fcba564c
I20250903 20:05:32.832918 27212 tablet_service.cc:1468] Processing CreateTablet for tablet fa2014bccb5e4f0a91cbc9845df4634d (DEFAULT_TABLE table=default.Table_051d82784c114e9f96c5717143bbc65d [id=2f0e4907f017416bb5790e27fc79aeb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.833217 27212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa2014bccb5e4f0a91cbc9845df4634d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.833367 27684 ts_tablet_manager.cc:1939] T 607c9a9cbc5540e5bd0fa436fcba564c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:32.837353 27216 tablet_service.cc:1515] Processing DeleteTablet for tablet 567630e2a93645e58f9bf05ffff87b32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:32.837913 27227 tablet_service.cc:1515] Processing DeleteTablet for tablet 5002b7e8118c49b6a247cb8dc4ca1aea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:32.838301 27227 tablet_service.cc:1515] Processing DeleteTablet for tablet 02b3e262dd7841dfbb0520377eb94743 with delete_type TABLET_DATA_DELETED (replaced by tablet fa544dfdfff04a1e97676e562aeb0f40) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:32.838532 29053 raft_consensus.cc:513] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.839263 27227 tablet_service.cc:1515] Processing DeleteTablet for tablet a0f79831866b40c49c5b27ce2014c3f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:32.839592 29053 leader_election.cc:290] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:32.839751 27227 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bcb9d5f010f4297bab78c0183889238 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a9e55f0811345f7afa71645f38ebdaf) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:32.840184 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet 8fff36d5a80e45af8a256c3908555e08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:32.843346 27213 tablet_service.cc:1515] Processing DeleteTablet for tablet a253e3399622409e92299069b19d3698 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:32.843605 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e027444cbd5545a38f6e4f35e1ed90c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:32.844249 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e027444cbd5545a38f6e4f35e1ed90c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:32.845547 27685 ts_tablet_manager.cc:1918] T f0efbb7eefb84ab6bee5b9625b91e298 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:32.845685 27685 log.cc:1199] T f0efbb7eefb84ab6bee5b9625b91e298 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/f0efbb7eefb84ab6bee5b9625b91e298
I20250903 20:05:32.846107 27685 ts_tablet_manager.cc:1939] T f0efbb7eefb84ab6bee5b9625b91e298 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:32.846192 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet a253e3399622409e92299069b19d3698 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.847129 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet d1c96296a579440c887ba6c2a93f7204 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:29 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.847445 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet 512908bc4d6941ec81b145ee5795405f with delete_type TABLET_DATA_DELETED (replaced by tablet 1598824a733a4a93bed043e81b86abf3) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.848080 27359 tablet_service.cc:1468] Processing CreateTablet for tablet d53bf94124e34deb92bb2ac1939ac70b (DEFAULT_TABLE table=default.Table_271d79e67006404794743b3897946e9a [id=9160d17e677942f5bf459d72fac75365]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.848526 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d53bf94124e34deb92bb2ac1939ac70b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.853006 27684 tablet_replica.cc:331] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:32.854418 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 846) took 5213 ms. Trace:
I20250903 20:05:32.854955 27684 raft_consensus.cc:2241] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:32.855535 27685 rpcz_store.cc:276] 0903 20:05:27.640618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.642698 (+ 2080us) service_pool.cc:225] Handling call
0903 20:05:32.771046 (+5128348us) ts_tablet_manager.cc:1145] Deleting tablet f0efbb7eefb84ab6bee5b9625b91e298
0903 20:05:32.774700 (+ 3654us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.837451 (+ 62751us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.845436 (+ 7985us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.854395 (+ 8959us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5128215}
I20250903 20:05:32.856849 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 848) took 5188 ms. Trace:
I20250903 20:05:32.857038 27684 raft_consensus.cc:2270] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.857395 27353 tablet_service.cc:1468] Processing CreateTablet for tablet 80e057f690a542e7a91cbd790ed15c59 (DEFAULT_TABLE table=default.Table_db15e7b90ba74a349a32034316f18378 [id=03c241f456ae462f9c17b9c2b504ecc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.857849 27353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80e057f690a542e7a91cbd790ed15c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.861212 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 607c9a9cbc5540e5bd0fa436fcba564c (table default.Table_5b44669e91a8427c8e7d96b866fdb3b5 [id=96167b0d9f9b47168a9fc59635b7136d]) successfully deleted
I20250903 20:05:32.862704 27350 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.862694 27494 tablet_service.cc:1468] Processing CreateTablet for tablet e7fd7cbf5a3048558720c7e258be308c (DEFAULT_TABLE table=default.Table_d6bdbaa18dba41bc997a529a1d2d322d [id=feda95c0e0c74dfd8803e026aae32a57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.863139 27500 tablet_service.cc:1468] Processing CreateTablet for tablet ca9fffda31624dc0827e6234ab02bb3e (DEFAULT_TABLE table=default.Table_dbeec832140d416aa6667b1958c1abf2 [id=29dcf515ea874dc4a811102221089817]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.863210 27489 tablet_service.cc:1468] Processing CreateTablet for tablet 23f315a1fa3d4c749994de7d0a0df775 (DEFAULT_TABLE table=default.Table_2f3c5f77e6bd4a0ebf0ac41e63589e79 [id=708a748c53be42049d5b4d1ca7713bc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.863238 27497 tablet_service.cc:1468] Processing CreateTablet for tablet 1355ab18761f4ab894d10d95abe5fec5 (DEFAULT_TABLE table=default.Table_74cf241e5c694a219f63258fb4b582c6 [id=7fddd5bd19eb4622829651eba1bf7c7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.863235 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet a9f43781299e47a8aecc37a0b73d7a64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:32.863445 27496 tablet_service.cc:1468] Processing CreateTablet for tablet 80e057f690a542e7a91cbd790ed15c59 (DEFAULT_TABLE table=default.Table_db15e7b90ba74a349a32034316f18378 [id=03c241f456ae462f9c17b9c2b504ecc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.863546 27503 tablet_service.cc:1468] Processing CreateTablet for tablet d53bf94124e34deb92bb2ac1939ac70b (DEFAULT_TABLE table=default.Table_271d79e67006404794743b3897946e9a [id=9160d17e677942f5bf459d72fac75365]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.863788 27685 rpcz_store.cc:276] 0903 20:05:27.668166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.668716 (+ 550us) service_pool.cc:225] Handling call
0903 20:05:32.856716 (+5188000us) ts_tablet_manager.cc:1145] Deleting tablet 733f8024e1fa401ca9614a97db91a19b
0903 20:05:32.856727 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.856831 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5187784}
I20250903 20:05:32.862944 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 02b3e262dd7841dfbb0520377eb94743 with delete_type TABLET_DATA_DELETED (replaced by tablet fa544dfdfff04a1e97676e562aeb0f40) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.864285 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 850) took 4955 ms. Trace:
I20250903 20:05:32.864378 27685 rpcz_store.cc:276] 0903 20:05:27.908914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.909011 (+ 97us) service_pool.cc:225] Handling call
0903 20:05:32.864116 (+4955105us) ts_tablet_manager.cc:1145] Deleting tablet 475d05d087c34159b093b92cc308bcaf
0903 20:05:32.864130 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.864266 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4954877}
I20250903 20:05:32.864706 27488 tablet_service.cc:1468] Processing CreateTablet for tablet c2da49d4d4ce4dc782d0b5ce8f74efbc (DEFAULT_TABLE table=default.Table_edc3f0ba095c4b65b0bb1f12d3f231c5 [id=44d492b191c84d7ba5be2dea19a8a169]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.864799 27495 tablet_service.cc:1468] Processing CreateTablet for tablet fa2014bccb5e4f0a91cbc9845df4634d (DEFAULT_TABLE table=default.Table_051d82784c114e9f96c5717143bbc65d [id=2f0e4907f017416bb5790e27fc79aeb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.864766 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 9e95a606cc2e4266984dbbe2969b3d13 (DEFAULT_TABLE table=default.Table_58b22c5102fc41568b4619bd19092925 [id=69e5c8d9217049c292cd63a493b03dff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.864663 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 8e37674dd6f94feab26741033409f23c (DEFAULT_TABLE table=default.Table_f573c7cc8d7240c0bf60622441b33a69 [id=41adb50030d24515b6896b0159fcc1b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.865042 27494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7fd7cbf5a3048558720c7e258be308c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.865290 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2da49d4d4ce4dc782d0b5ce8f74efbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.865541 27498 tablet_service.cc:1468] Processing CreateTablet for tablet 361156ca43a444b1b07e1d1fab48c234 (DEFAULT_TABLE table=default.Table_c7ba7021073b42a8b57cb33054f9a141 [id=62966767ca30457f86722ec3ab6cbd32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.858701 27213 tablet_service.cc:1468] Processing CreateTablet for tablet 80e057f690a542e7a91cbd790ed15c59 (DEFAULT_TABLE table=default.Table_db15e7b90ba74a349a32034316f18378 [id=03c241f456ae462f9c17b9c2b504ecc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.865875 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca9fffda31624dc0827e6234ab02bb3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.866106 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d53bf94124e34deb92bb2ac1939ac70b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.866385 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e95a606cc2e4266984dbbe2969b3d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.869716 27489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23f315a1fa3d4c749994de7d0a0df775. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.870379 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80e057f690a542e7a91cbd790ed15c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.870985 27498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 361156ca43a444b1b07e1d1fab48c234. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.871217 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa2014bccb5e4f0a91cbc9845df4634d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.872915 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1355ab18761f4ab894d10d95abe5fec5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:32.859894 27184 consensus_peers.cc:489] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2629f46704e4a0fa7922533928ca2ed. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:32.879792 27227 tablet_service.cc:1468] Processing CreateTablet for tablet 361156ca43a444b1b07e1d1fab48c234 (DEFAULT_TABLE table=default.Table_c7ba7021073b42a8b57cb33054f9a141 [id=62966767ca30457f86722ec3ab6cbd32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.880218 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 361156ca43a444b1b07e1d1fab48c234. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.880823 27226 tablet_service.cc:1468] Processing CreateTablet for tablet d53bf94124e34deb92bb2ac1939ac70b (DEFAULT_TABLE table=default.Table_271d79e67006404794743b3897946e9a [id=9160d17e677942f5bf459d72fac75365]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.881167 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d53bf94124e34deb92bb2ac1939ac70b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.883137 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e37674dd6f94feab26741033409f23c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.886485 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80e057f690a542e7a91cbd790ed15c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.886888 27684 ts_tablet_manager.cc:1905] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:32.887555 27212 tablet_service.cc:1468] Processing CreateTablet for tablet 9e95a606cc2e4266984dbbe2969b3d13 (DEFAULT_TABLE table=default.Table_58b22c5102fc41568b4619bd19092925 [id=69e5c8d9217049c292cd63a493b03dff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.887882 27212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e95a606cc2e4266984dbbe2969b3d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.888559 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet f0efbb7eefb84ab6bee5b9625b91e298 (table default.Table_781d39d6b06f49c28fa93c3692c7b5f7 [id=8d664216ea8e47fdaba51b2d6ca9d0a5]) successfully deleted
I20250903 20:05:32.893071 27499 tablet_service.cc:1468] Processing CreateTablet for tablet 9ed340a66af14ba18bdc3019703bc5e1 (DEFAULT_TABLE table=default.Table_bbc905e2f2804b9ead3a69b83cf50fde [id=999c439c64e044de8ed3638f58608b02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.893430 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet a0f79831866b40c49c5b27ce2014c3f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.888018 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 851) took 4977 ms. Trace:
I20250903 20:05:32.892552 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 1355ab18761f4ab894d10d95abe5fec5 (DEFAULT_TABLE table=default.Table_74cf241e5c694a219f63258fb4b582c6 [id=7fddd5bd19eb4622829651eba1bf7c7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.898376 27685 rpcz_store.cc:276] 0903 20:05:27.910453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.910724 (+ 271us) service_pool.cc:225] Handling call
0903 20:05:32.887890 (+4977166us) ts_tablet_manager.cc:1145] Deleting tablet 5fe1de01bf7a4d418e765ea3e8cbecc1
0903 20:05:32.887900 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.888001 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4976982}
I20250903 20:05:32.898661 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1355ab18761f4ab894d10d95abe5fec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.899214 27348 tablet_service.cc:1468] Processing CreateTablet for tablet 23f315a1fa3d4c749994de7d0a0df775 (DEFAULT_TABLE table=default.Table_2f3c5f77e6bd4a0ebf0ac41e63589e79 [id=708a748c53be42049d5b4d1ca7713bc4]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:32.899618 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 733f8024e1fa401ca9614a97db91a19b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 733f8024e1fa401ca9614a97db91a19b already in progress: creating tablet
I20250903 20:05:32.898958 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 852) took 4986 ms. Trace:
I20250903 20:05:32.900015 27685 rpcz_store.cc:276] 0903 20:05:27.912817 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:27.912888 (+ 71us) service_pool.cc:225] Handling call
0903 20:05:32.898831 (+4985943us) ts_tablet_manager.cc:1145] Deleting tablet 3e930dbaac0e40ee8f175ccf87592d31
0903 20:05:32.898841 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.898941 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4985740}
I20250903 20:05:32.894138 27222 tablet_service.cc:1468] Processing CreateTablet for tablet c2da49d4d4ce4dc782d0b5ce8f74efbc (DEFAULT_TABLE table=default.Table_edc3f0ba095c4b65b0bb1f12d3f231c5 [id=44d492b191c84d7ba5be2dea19a8a169]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.900985 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ed340a66af14ba18bdc3019703bc5e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.893813 27215 tablet_service.cc:1468] Processing CreateTablet for tablet ca9fffda31624dc0827e6234ab02bb3e (DEFAULT_TABLE table=default.Table_dbeec832140d416aa6667b1958c1abf2 [id=29dcf515ea874dc4a811102221089817]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.901934 27685 tablet_replica.cc:331] T 1bad3c509460422396a17a3c4ae3d1e8 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:32.902370 27685 raft_consensus.cc:2241] T 1bad3c509460422396a17a3c4ae3d1e8 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:32.902665 27685 raft_consensus.cc:2270] T 1bad3c509460422396a17a3c4ae3d1e8 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.903370 27685 ts_tablet_manager.cc:1905] T 1bad3c509460422396a17a3c4ae3d1e8 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:32.904359 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca9fffda31624dc0827e6234ab02bb3e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:32.904696 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 475d05d087c34159b093b92cc308bcaf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 475d05d087c34159b093b92cc308bcaf already in progress: creating tablet
I20250903 20:05:32.908048 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2da49d4d4ce4dc782d0b5ce8f74efbc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:32.911008 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 5fe1de01bf7a4d418e765ea3e8cbecc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5fe1de01bf7a4d418e765ea3e8cbecc1 already in progress: creating tablet
I20250903 20:05:32.914913 27684 ts_tablet_manager.cc:1918] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:32.915285 27695 tablet_bootstrap.cc:492] T 200c06f296b94eab94f35f312c5a443b P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:32.915432 27695 ts_tablet_manager.cc:1397] T 200c06f296b94eab94f35f312c5a443b P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.193s user 0.003s sys 0.000s
I20250903 20:05:32.897439 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 361156ca43a444b1b07e1d1fab48c234 (DEFAULT_TABLE table=default.Table_c7ba7021073b42a8b57cb33054f9a141 [id=62966767ca30457f86722ec3ab6cbd32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.916083 27684 log.cc:1199] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/1f6cd4d475f147fe8dfcc7e3a73dedf8
I20250903 20:05:32.897119 27350 tablet_service.cc:1468] Processing CreateTablet for tablet ca9fffda31624dc0827e6234ab02bb3e (DEFAULT_TABLE table=default.Table_dbeec832140d416aa6667b1958c1abf2 [id=29dcf515ea874dc4a811102221089817]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.901973 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 1355ab18761f4ab894d10d95abe5fec5 (DEFAULT_TABLE table=default.Table_74cf241e5c694a219f63258fb4b582c6 [id=7fddd5bd19eb4622829651eba1bf7c7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:32.916510 28874 raft_consensus.cc:491] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:32.916632 28874 raft_consensus.cc:513] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.917248 28874 leader_election.cc:290] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:32.917444 27684 ts_tablet_manager.cc:1939] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:32.918172 27517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a253e3399622409e92299069b19d3698" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:32.918426 27517 raft_consensus.cc:2466] T a253e3399622409e92299069b19d3698 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:32.918823 27183 leader_election.cc:304] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:32.919242 28808 raft_consensus.cc:2802] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:32.919307 28808 raft_consensus.cc:491] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:32.919384 28808 raft_consensus.cc:3058] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:32.921775 27685 ts_tablet_manager.cc:1918] T 1bad3c509460422396a17a3c4ae3d1e8 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:32.921916 28808 raft_consensus.cc:513] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.922449 27684 tablet_replica.cc:331] T 4af8c03c9d834405b4e25d35441db961 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:32.922734 27684 raft_consensus.cc:2241] T 4af8c03c9d834405b4e25d35441db961 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:32.921904 27685 log.cc:1199] T 1bad3c509460422396a17a3c4ae3d1e8 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/1bad3c509460422396a17a3c4ae3d1e8
I20250903 20:05:32.924614 27517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a253e3399622409e92299069b19d3698" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:32.924809 27517 raft_consensus.cc:3058] T a253e3399622409e92299069b19d3698 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:32.925088 27684 raft_consensus.cc:2270] T 4af8c03c9d834405b4e25d35441db961 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.926106 27684 ts_tablet_manager.cc:1905] T 4af8c03c9d834405b4e25d35441db961 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:32.922456 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 3e930dbaac0e40ee8f175ccf87592d31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e930dbaac0e40ee8f175ccf87592d31 already in progress: creating tablet
I20250903 20:05:32.924000 27685 ts_tablet_manager.cc:1939] T 1bad3c509460422396a17a3c4ae3d1e8 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:32.926855 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 854) took 4550 ms. Trace:
W20250903 20:05:32.927330 27052 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet a785adcd0bd943fc97b201b6d29073f4 (table default.Table_7b9de0532ff4407086b7f341fedfb3af [id=5f1249a1893e4f83a8fa045d14637183], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet a785adcd0bd943fc97b201b6d29073f4 consensus state has no leader
I20250903 20:05:32.927397 27685 rpcz_store.cc:276] 0903 20:05:28.376467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.376903 (+ 436us) service_pool.cc:225] Handling call
0903 20:05:32.900438 (+4523535us) ts_tablet_manager.cc:1145] Deleting tablet 1bad3c509460422396a17a3c4ae3d1e8
0903 20:05:32.900448 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.908389 (+ 7941us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.921643 (+ 13254us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.926839 (+ 5196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2816,"tablet-delete.queue_time_us":4523308}
I20250903 20:05:32.927793 27517 raft_consensus.cc:2466] T a253e3399622409e92299069b19d3698 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:32.928561 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a253e3399622409e92299069b19d3698" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:32.928771 27379 raft_consensus.cc:2466] T a253e3399622409e92299069b19d3698 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:32.928957 27685 tablet_replica.cc:331] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:32.928097 28808 leader_election.cc:290] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:32.928587 27183 leader_election.cc:304] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:32.929541 28808 raft_consensus.cc:2802] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:32.929786 28808 raft_consensus.cc:695] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:32.930101 28808 consensus_queue.cc:237] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.932230 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a253e3399622409e92299069b19d3698" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:32.932446 27379 raft_consensus.cc:3058] T a253e3399622409e92299069b19d3698 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:32.935395 27379 raft_consensus.cc:2466] T a253e3399622409e92299069b19d3698 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:32.937975 27695 raft_consensus.cc:357] T 200c06f296b94eab94f35f312c5a443b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.939244 27695 raft_consensus.cc:383] T 200c06f296b94eab94f35f312c5a443b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:32.939544 27695 raft_consensus.cc:738] T 200c06f296b94eab94f35f312c5a443b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:32.940127 27695 consensus_queue.cc:260] T 200c06f296b94eab94f35f312c5a443b P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.941480 27695 ts_tablet_manager.cc:1428] T 200c06f296b94eab94f35f312c5a443b P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20250903 20:05:32.938766 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8 (table default.Table_943bea896c744410a2a4d80ffb9ca439 [id=a950fbaecd7341989961ff3ea9a713e0]) successfully deleted
W20250903 20:05:32.942173 27051 catalog_manager.cc:4895] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 6788b1683f984949816edfc5b68de8db because the tablet was not found. No further retry: Not found: Tablet not found: 6788b1683f984949816edfc5b68de8db
I20250903 20:05:32.942595 27695 tablet_bootstrap.cc:492] T e877d56bbbfa4c5a92d025d0311ce5c6 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:32.947199 27684 ts_tablet_manager.cc:1918] T 4af8c03c9d834405b4e25d35441db961 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:32.947679 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_8b2d83116755450387b770b591314010" } new_table_name: "default.Table_9dc948a62eca49f29840200652ee0a06" modify_external_catalogs: true
I20250903 20:05:32.929155 27685 raft_consensus.cc:2241] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:32.949426 27685 raft_consensus.cc:2270] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.950107 27685 ts_tablet_manager.cc:1905] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:32.950531 27684 log.cc:1199] T 4af8c03c9d834405b4e25d35441db961 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/4af8c03c9d834405b4e25d35441db961
I20250903 20:05:32.951504 27684 ts_tablet_manager.cc:1939] T 4af8c03c9d834405b4e25d35441db961 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:32.953764 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 4af8c03c9d834405b4e25d35441db961 (table default.Table_9eac2fd82a524aae943ebd76428f95eb [id=058a64daaf104dfdace7d05d52388b79]) successfully deleted
W20250903 20:05:32.955925 27051 catalog_manager.cc:4895] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 53881ad30bc843839008ace4012a3f70 because the tablet was not found. No further retry: Not found: Tablet not found: 53881ad30bc843839008ace4012a3f70
I20250903 20:05:32.956820 27684 tablet_replica.cc:331] T 20c25c84c6fb4609a1964a4ff3c526da P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:32.957513 27684 raft_consensus.cc:2241] T 20c25c84c6fb4609a1964a4ff3c526da P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:32.958505 27684 raft_consensus.cc:2270] T 20c25c84c6fb4609a1964a4ff3c526da P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.959517 27684 ts_tablet_manager.cc:1905] T 20c25c84c6fb4609a1964a4ff3c526da P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:32.965394 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23f315a1fa3d4c749994de7d0a0df775. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.966020 27687 raft_consensus.cc:2241] T b7a6d5d8a97a4769a81ec7224d591f8c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250903 20:05:32.969764 27183 consensus_peers.cc:489] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e2f98f760cf4df4bc2a10530e2157c6. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:32.975396 27684 ts_tablet_manager.cc:1918] T 20c25c84c6fb4609a1964a4ff3c526da P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:32.975586 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 3c47c5fca0ee4349b2667494f45af5d0
I20250903 20:05:32.975721 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet b32c82c3d8c4420cb6e3a95129550076
I20250903 20:05:32.976162 27061 catalog_manager.cc:6652] ReplaceTablet: tablet 3c47c5fca0ee4349b2667494f45af5d0 deleted and replaced by tablet 2d4df4cc00c9433bb120566a797be986
I20250903 20:05:32.976598 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_762b9caaaec44868b1816efca7d186cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:32.978286 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_762b9caaaec44868b1816efca7d186cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:32.978672 27213 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c47c5fca0ee4349b2667494f45af5d0 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d4df4cc00c9433bb120566a797be986) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:32.979105 27227 tablet_service.cc:1515] Processing DeleteTablet for tablet b32c82c3d8c4420cb6e3a95129550076 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:32.976862 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c47c5fca0ee4349b2667494f45af5d0 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d4df4cc00c9433bb120566a797be986) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:32.977484 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet b32c82c3d8c4420cb6e3a95129550076 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:32.980721 27684 log.cc:1199] T 20c25c84c6fb4609a1964a4ff3c526da P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/20c25c84c6fb4609a1964a4ff3c526da
I20250903 20:05:32.981411 27684 ts_tablet_manager.cc:1939] T 20c25c84c6fb4609a1964a4ff3c526da P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:32.981832 27685 ts_tablet_manager.cc:1918] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:32.983110 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 20c25c84c6fb4609a1964a4ff3c526da (table default.Table_6b435f3c61214c74a632c94e955abcdf [id=c2cf9ce651fc41a9b2b223b998e45d9b]) successfully deleted
I20250903 20:05:32.984403 27684 tablet_replica.cc:331] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:32.984882 27684 raft_consensus.cc:2241] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:32.985498 27684 raft_consensus.cc:2270] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.986552 27684 ts_tablet_manager.cc:1905] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:32.969081 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca9fffda31624dc0827e6234ab02bb3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.984297 27695 tablet_bootstrap.cc:654] T e877d56bbbfa4c5a92d025d0311ce5c6 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:32.993081 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1355ab18761f4ab894d10d95abe5fec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:32.993589 27685 log.cc:1199] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/6f8cfe53b91b41d7ba7f9074e1005af3
I20250903 20:05:32.993981 27685 ts_tablet_manager.cc:1939] T 6f8cfe53b91b41d7ba7f9074e1005af3 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:32.977908 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c47c5fca0ee4349b2667494f45af5d0 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d4df4cc00c9433bb120566a797be986) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.994262 29044 raft_consensus.cc:491] T 4c18a98d5c0f4a43ae91c9130047b681 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:32.994339 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 855) took 4617 ms. Trace:
I20250903 20:05:32.994410 29044 raft_consensus.cc:513] T 4c18a98d5c0f4a43ae91c9130047b681 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:32.994419 27685 rpcz_store.cc:276] 0903 20:05:28.376570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.377161 (+ 591us) service_pool.cc:225] Handling call
0903 20:05:32.928903 (+4551742us) ts_tablet_manager.cc:1145] Deleting tablet 6f8cfe53b91b41d7ba7f9074e1005af3
0903 20:05:32.928915 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:32.972103 (+ 43188us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.981701 (+ 9598us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:32.994320 (+ 12619us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4551571}
I20250903 20:05:32.978101 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet b32c82c3d8c4420cb6e3a95129550076 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:32.994984 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 6f8cfe53b91b41d7ba7f9074e1005af3 (table default.Table_e5def563dd414101b9ee8fbaf7156e29 [id=7e67c0d1d2f34172a987a8ea785c5a6a]) successfully deleted
I20250903 20:05:32.995023 29044 leader_election.cc:290] T 4c18a98d5c0f4a43ae91c9130047b681 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:32.997318 27687 raft_consensus.cc:2270] T b7a6d5d8a97a4769a81ec7224d591f8c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:32.998080 27687 ts_tablet_manager.cc:1905] T b7a6d5d8a97a4769a81ec7224d591f8c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:32.998425 27685 tablet_replica.cc:331] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:32.998718 27685 raft_consensus.cc:2241] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:32.999215 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 361156ca43a444b1b07e1d1fab48c234. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.002576 27685 raft_consensus.cc:2270] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:32.981050 27321 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 189994us
I20250903 20:05:33.003693 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "06a79706f1184fa49d10182ba669b0ab" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:33.003887 27248 raft_consensus.cc:3058] T 06a79706f1184fa49d10182ba669b0ab P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:33.004208 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1c96296a579440c887ba6c2a93f7204" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:33.003695 27685 ts_tablet_manager.cc:1905] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.004334 27239 raft_consensus.cc:3058] T d1c96296a579440c887ba6c2a93f7204 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:33.006425 27248 raft_consensus.cc:2466] T 06a79706f1184fa49d10182ba669b0ab P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:33.006641 27239 raft_consensus.cc:2466] T d1c96296a579440c887ba6c2a93f7204 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:33.008975 27321 leader_election.cc:304] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:33.010051 27321 leader_election.cc:304] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:33.010329 28916 raft_consensus.cc:2802] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:33.010493 28916 raft_consensus.cc:695] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:33.010803 28916 consensus_queue.cc:237] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:33.012226 29053 raft_consensus.cc:2802] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:33.012336 29053 raft_consensus.cc:695] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:33.012543 29053 consensus_queue.cc:237] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.017396 27684 ts_tablet_manager.cc:1918] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.017585 27684 log.cc:1199] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/63a1f26fac3b4b939ebb79c6a07d8f8e
I20250903 20:05:33.018038 27684 ts_tablet_manager.cc:1939] T 63a1f26fac3b4b939ebb79c6a07d8f8e P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:33.019169 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 63a1f26fac3b4b939ebb79c6a07d8f8e (table default.Table_362ac8cb413e426ebf154eae4b6b5c7e [id=10ef25498f7448dbb19cd66d4c6ba713]) successfully deleted
I20250903 20:05:33.019598 27684 tablet_replica.cc:331] T f9709dfd18654c5bb071e39da7c8889f P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:33.019853 27684 raft_consensus.cc:2241] T f9709dfd18654c5bb071e39da7c8889f P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:33.020061 27684 raft_consensus.cc:2270] T f9709dfd18654c5bb071e39da7c8889f P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.020702 27684 ts_tablet_manager.cc:1905] T f9709dfd18654c5bb071e39da7c8889f P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.021745 27685 ts_tablet_manager.cc:1918] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.022619 28567 consensus_queue.cc:1035] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:33.024471 27692 tablet_bootstrap.cc:654] T e877d56bbbfa4c5a92d025d0311ce5c6 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:33.026427 27462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 230002us
W20250903 20:05:33.026873 27462 leader_election.cc:343] T 7345305820be4d32a3ceb227f045f009 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:33.025363 27183 consensus_peers.cc:489] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a72b85dfb34047528b2b28ff0d40f828. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:33.027704 27685 log.cc:1199] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/59a1dd48d76241d7ba817070fbffd853
I20250903 20:05:33.027858 27692 tablet_bootstrap.cc:492] T e877d56bbbfa4c5a92d025d0311ce5c6 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:33.028091 27685 ts_tablet_manager.cc:1939] T 59a1dd48d76241d7ba817070fbffd853 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:33.028429 27692 ts_tablet_manager.cc:1397] T e877d56bbbfa4c5a92d025d0311ce5c6 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.263s user 0.002s sys 0.003s
I20250903 20:05:33.028434 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 856) took 4651 ms. Trace:
I20250903 20:05:33.028506 27685 rpcz_store.cc:276] 0903 20:05:28.376648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.377359 (+ 711us) service_pool.cc:225] Handling call
0903 20:05:32.998153 (+4620794us) ts_tablet_manager.cc:1145] Deleting tablet 59a1dd48d76241d7ba817070fbffd853
0903 20:05:32.998165 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.015762 (+ 17597us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.021654 (+ 5892us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.028416 (+ 6762us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4620635}
I20250903 20:05:33.028976 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 59a1dd48d76241d7ba817070fbffd853 (table default.Table_d342b3c06e1c4b98bc7f07e88844cdc9 [id=34db41bcdc604ee08da11d0e559b4d35]) successfully deleted
I20250903 20:05:33.029664 27692 raft_consensus.cc:357] T e877d56bbbfa4c5a92d025d0311ce5c6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.029995 27692 raft_consensus.cc:383] T e877d56bbbfa4c5a92d025d0311ce5c6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.030128 27692 raft_consensus.cc:738] T e877d56bbbfa4c5a92d025d0311ce5c6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.030412 27685 tablet_replica.cc:331] T 1fbe54dc40dd47daa109ceeb07546b0a P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:33.030426 27692 consensus_queue.cc:260] T e877d56bbbfa4c5a92d025d0311ce5c6 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.030632 27685 raft_consensus.cc:2241] T 1fbe54dc40dd47daa109ceeb07546b0a P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:33.030797 27685 raft_consensus.cc:2270] T 1fbe54dc40dd47daa109ceeb07546b0a P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.031229 27692 ts_tablet_manager.cc:1428] T e877d56bbbfa4c5a92d025d0311ce5c6 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250903 20:05:33.031476 27685 ts_tablet_manager.cc:1905] T 1fbe54dc40dd47daa109ceeb07546b0a P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.031635 27692 tablet_bootstrap.cc:492] T 45f7068bafaf48d290ded9d1272c0e46 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:33.033083 28947 raft_consensus.cc:491] T fc634116a78f45dd833034ddda90e9bf P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:33.033144 27692 tablet_bootstrap.cc:654] T 45f7068bafaf48d290ded9d1272c0e46 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:33.033242 28947 raft_consensus.cc:513] T fc634116a78f45dd833034ddda90e9bf P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:33.033852 28947 leader_election.cc:290] T fc634116a78f45dd833034ddda90e9bf P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:33.034287 27209 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:56514 (request call id 152) took 1191 ms. Trace:
I20250903 20:05:33.034364 27209 rpcz_store.cc:276] 0903 20:05:31.842398 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.864728 (+ 22330us) service_pool.cc:225] Handling call
0903 20:05:33.034265 (+1169537us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250903 20:05:33.034662 27517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc634116a78f45dd833034ddda90e9bf" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:33.034809 27684 ts_tablet_manager.cc:1918] T f9709dfd18654c5bb071e39da7c8889f P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.034832 27517 raft_consensus.cc:2466] T fc634116a78f45dd833034ddda90e9bf P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:33.034895 27684 log.cc:1199] T f9709dfd18654c5bb071e39da7c8889f P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/f9709dfd18654c5bb071e39da7c8889f
I20250903 20:05:33.035244 27684 ts_tablet_manager.cc:1939] T f9709dfd18654c5bb071e39da7c8889f P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:33.035370 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc634116a78f45dd833034ddda90e9bf" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:33.035521 27183 leader_election.cc:304] T fc634116a78f45dd833034ddda90e9bf P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
W20250903 20:05:33.036017 27184 leader_election.cc:343] T fc634116a78f45dd833034ddda90e9bf P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:33.036156 28947 raft_consensus.cc:2802] T fc634116a78f45dd833034ddda90e9bf P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:33.036232 28947 raft_consensus.cc:491] T fc634116a78f45dd833034ddda90e9bf P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:33.036314 28947 raft_consensus.cc:3058] T fc634116a78f45dd833034ddda90e9bf P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:33.036350 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet f9709dfd18654c5bb071e39da7c8889f (table default.Table_876161e56d1d4c4c9d6726471530d965 [id=39cbb7f0d8e145c6b5f283133017b80f]) successfully deleted
I20250903 20:05:33.038815 28947 raft_consensus.cc:513] T fc634116a78f45dd833034ddda90e9bf P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:33.040128 27517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc634116a78f45dd833034ddda90e9bf" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:33.040284 27517 raft_consensus.cc:3058] T fc634116a78f45dd833034ddda90e9bf P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:33.040751 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc634116a78f45dd833034ddda90e9bf" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
W20250903 20:05:33.041623 27184 leader_election.cc:343] T fc634116a78f45dd833034ddda90e9bf P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:33.041970 27684 tablet_replica.cc:331] T 0a9b7770432942da9373d5100c71bd6a P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:33.042145 27684 raft_consensus.cc:2241] T 0a9b7770432942da9373d5100c71bd6a P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:33.042331 27684 raft_consensus.cc:2270] T 0a9b7770432942da9373d5100c71bd6a P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.042670 27517 raft_consensus.cc:2466] T fc634116a78f45dd833034ddda90e9bf P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:33.042917 27684 ts_tablet_manager.cc:1905] T 0a9b7770432942da9373d5100c71bd6a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.043112 27183 leader_election.cc:304] T fc634116a78f45dd833034ddda90e9bf P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:33.043397 28808 raft_consensus.cc:2802] T fc634116a78f45dd833034ddda90e9bf P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:33.043582 28808 raft_consensus.cc:695] T fc634116a78f45dd833034ddda90e9bf P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:33.043803 28808 consensus_queue.cc:237] T fc634116a78f45dd833034ddda90e9bf P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:33.045219 28947 leader_election.cc:290] T fc634116a78f45dd833034ddda90e9bf P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
W20250903 20:05:33.046145 27461 connection.cc:537] client connection to 127.26.65.125:44059 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 20:05:33.046193 27051 connection.cc:537] client connection to 127.26.65.125:44059 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250903 20:05:33.046401 27051 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 2 similar messages]
I20250903 20:05:33.052166 27685 ts_tablet_manager.cc:1918] T 1fbe54dc40dd47daa109ceeb07546b0a P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250903 20:05:33.056505 27320 connection.cc:537] client connection to 127.26.65.125:44059 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250903 20:05:33.057525 27685 log.cc:1199] T 1fbe54dc40dd47daa109ceeb07546b0a P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/1fbe54dc40dd47daa109ceeb07546b0a
I20250903 20:05:33.057807 27684 ts_tablet_manager.cc:1918] T 0a9b7770432942da9373d5100c71bd6a P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.058171 27685 ts_tablet_manager.cc:1939] T 1fbe54dc40dd47daa109ceeb07546b0a P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
W20250903 20:05:33.058326 27183 connection.cc:537] client connection to 127.26.65.125:44059 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250903 20:05:33.059764 27684 log.cc:1199] T 0a9b7770432942da9373d5100c71bd6a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/0a9b7770432942da9373d5100c71bd6a
I20250903 20:05:33.060262 27684 ts_tablet_manager.cc:1939] T 0a9b7770432942da9373d5100c71bd6a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
W20250903 20:05:33.059639 29121 negotiation.cc:337] Failed RPC negotiation. Trace:
0903 20:05:33.049050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.26.65.125:44059 (local address 127.26.65.67:54903)
0903 20:05:33.059289 (+ 10239us) negotiation.cc:107] Waiting for socket to connect
0903 20:05:33.059393 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.26.65.125:44059: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":10120,"thread_start_us":10038,"threads_started":1}
I20250903 20:05:33.064102 27684 tablet_replica.cc:331] T b0d79c78e0024b7ba3bb3356be7c8ceb P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:33.064548 27684 raft_consensus.cc:2241] T b0d79c78e0024b7ba3bb3356be7c8ceb P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:33.064517 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_66ee6081d8504c119f365516fc8be9f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:33.064626 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_789d6e974e2f44ff8d04b1a39adcb6a7" } new_table_name: "default.Table_e0fec991c20c4b07a1f0d59074f9d720" modify_external_catalogs: true
W20250903 20:05:33.065474 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66ee6081d8504c119f365516fc8be9f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:33.066437 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 16549a6d9aa9432eb5cee19f69ce6957
I20250903 20:05:33.068365 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f8bbfa6b86694cdab5ec1347c36a69f3" } modify_external_catalogs: true
I20250903 20:05:33.068641 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f8bbfa6b86694cdab5ec1347c36a69f3" } modify_external_catalogs: true
I20250903 20:05:33.069243 27684 raft_consensus.cc:2270] T b0d79c78e0024b7ba3bb3356be7c8ceb P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.070130 27684 ts_tablet_manager.cc:1905] T b0d79c78e0024b7ba3bb3356be7c8ceb P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.070358 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet 16549a6d9aa9432eb5cee19f69ce6957 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:33.073329 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 857) took 4696 ms. Trace:
I20250903 20:05:32.739493 27693 raft_consensus.cc:357] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.074451 27685 rpcz_store.cc:276] 0903 20:05:28.376766 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.377539 (+ 773us) service_pool.cc:225] Handling call
0903 20:05:33.030364 (+4652825us) ts_tablet_manager.cc:1145] Deleting tablet 1fbe54dc40dd47daa109ceeb07546b0a
0903 20:05:33.030374 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.040402 (+ 10028us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.052026 (+ 11624us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.073300 (+ 21274us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4652700}
I20250903 20:05:33.075220 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 1fbe54dc40dd47daa109ceeb07546b0a (table default.Table_975e53d37e054938a5b63957252ff50d [id=f261e84232234244824d0232efbb4399]) successfully deleted
I20250903 20:05:33.075626 27685 tablet_replica.cc:331] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:33.076839 27685 raft_consensus.cc:2241] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:33.077085 27693 raft_consensus.cc:383] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.077183 27693 raft_consensus.cc:738] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.077419 27693 consensus_queue.cc:260] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.077549 27685 raft_consensus.cc:2270] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.077850 27350 tablet_service.cc:1515] Processing DeleteTablet for tablet 16549a6d9aa9432eb5cee19f69ce6957 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:33.077921 27693 ts_tablet_manager.cc:1428] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.339s user 0.001s sys 0.000s
I20250903 20:05:33.078188 27693 tablet_bootstrap.cc:492] T 7345305820be4d32a3ceb227f045f009 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:33.078570 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c18a98d5c0f4a43ae91c9130047b681" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:33.079597 27693 tablet_bootstrap.cc:654] T 7345305820be4d32a3ceb227f045f009 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:33.082060 27517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "06a79706f1184fa49d10182ba669b0ab" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:33.082258 27517 raft_consensus.cc:3058] T 06a79706f1184fa49d10182ba669b0ab P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:33.082918 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet 16549a6d9aa9432eb5cee19f69ce6957 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:33.078558 27685 ts_tablet_manager.cc:1905] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.085289 27517 raft_consensus.cc:2466] T 06a79706f1184fa49d10182ba669b0ab P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:33.086917 27692 tablet_bootstrap.cc:492] T 45f7068bafaf48d290ded9d1272c0e46 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:33.087121 27692 ts_tablet_manager.cc:1397] T 45f7068bafaf48d290ded9d1272c0e46 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.003s
W20250903 20:05:33.087502 27462 leader_election.cc:343] T 4c18a98d5c0f4a43ae91c9130047b681 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:33.087852 27692 raft_consensus.cc:357] T 45f7068bafaf48d290ded9d1272c0e46 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.090759 29108 raft_consensus.cc:491] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:33.090981 29108 raft_consensus.cc:513] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.092387 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5002b7e8118c49b6a247cb8dc4ca1aea" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:33.092605 27249 raft_consensus.cc:2466] T 5002b7e8118c49b6a247cb8dc4ca1aea P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:33.093987 27321 leader_election.cc:304] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:33.094391 28567 raft_consensus.cc:2802] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:33.094480 28567 raft_consensus.cc:491] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:33.094555 28567 raft_consensus.cc:3058] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:33.099649 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 0a9b7770432942da9373d5100c71bd6a (table default.Table_6656b728e12a482fa9e7893ef4842ac7 [id=6788b524a8654e34a4d9ee8cc8fc84d3]) successfully deleted
I20250903 20:05:33.100622 28567 raft_consensus.cc:513] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:33.101259 27051 consensus_peers.cc:489] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 -> Peer 35dd69f3fa7a4fd8843b97f8a5d68296 (127.26.65.125:44059): Couldn't send request to peer 35dd69f3fa7a4fd8843b97f8a5d68296. Status: Network error: Client connection negotiation failed: client connection to 127.26.65.125:44059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:33.101668 27685 ts_tablet_manager.cc:1918] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.104219 28567 leader_election.cc:290] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:33.105019 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5002b7e8118c49b6a247cb8dc4ca1aea" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:33.105543 27249 raft_consensus.cc:3058] T 5002b7e8118c49b6a247cb8dc4ca1aea P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:33.107899 29108 leader_election.cc:290] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:33.108379 27249 raft_consensus.cc:2466] T 5002b7e8118c49b6a247cb8dc4ca1aea P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:33.108671 27685 log.cc:1199] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/a4f3230f50ec4c95b528fead8c191224
I20250903 20:05:33.108978 27321 leader_election.cc:304] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:33.109192 27685 ts_tablet_manager.cc:1939] T a4f3230f50ec4c95b528fead8c191224 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:33.109726 29108 raft_consensus.cc:2802] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:33.088014 29044 consensus_queue.cc:1035] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:33.110107 27692 raft_consensus.cc:383] T 45f7068bafaf48d290ded9d1272c0e46 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.110230 29108 raft_consensus.cc:695] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:33.110285 27692 raft_consensus.cc:738] T 45f7068bafaf48d290ded9d1272c0e46 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.093617 27684 ts_tablet_manager.cc:1918] T b0d79c78e0024b7ba3bb3356be7c8ceb P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 20:05:33.110569 27684 log.cc:1199] T b0d79c78e0024b7ba3bb3356be7c8ceb P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/b0d79c78e0024b7ba3bb3356be7c8ceb
I20250903 20:05:33.110621 29108 consensus_queue.cc:237] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.110613 27692 consensus_queue.cc:260] T 45f7068bafaf48d290ded9d1272c0e46 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.111203 27692 ts_tablet_manager.cc:1428] T 45f7068bafaf48d290ded9d1272c0e46 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20250903 20:05:33.111490 27692 tablet_bootstrap.cc:492] T bf459db3327640f88574fca7b5f2ca98 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:33.111721 27684 ts_tablet_manager.cc:1939] T b0d79c78e0024b7ba3bb3356be7c8ceb P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:33.106155 29109 raft_consensus.cc:491] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:33.112519 29109 raft_consensus.cc:513] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.113157 27692 tablet_bootstrap.cc:654] T bf459db3327640f88574fca7b5f2ca98 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:33.113126 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 858) took 4736 ms. Trace:
I20250903 20:05:33.113217 27685 rpcz_store.cc:276] 0903 20:05:28.376854 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.377688 (+ 834us) service_pool.cc:225] Handling call
0903 20:05:33.075576 (+4697888us) ts_tablet_manager.cc:1145] Deleting tablet a4f3230f50ec4c95b528fead8c191224
0903 20:05:33.075586 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.088271 (+ 12685us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.101539 (+ 13268us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.113102 (+ 11563us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4697736}
I20250903 20:05:33.113693 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet a4f3230f50ec4c95b528fead8c191224 (table default.Table_be001b7aec594b93a1b45e9313b772ca [id=7a045d80e0eb42cb95fdca51d339da0d]) successfully deleted
I20250903 20:05:33.114428 27685 tablet_replica.cc:331] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:33.114943 27685 raft_consensus.cc:2241] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:33.115129 27685 raft_consensus.cc:2270] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.115813 27685 ts_tablet_manager.cc:1905] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.120083 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet 567630e2a93645e58f9bf05ffff87b32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:33.120570 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet 8fff36d5a80e45af8a256c3908555e08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:33.121068 27059 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.26.65.67:58355 (request call id 152) took 783 ms (client timeout 1000 ms). Trace:
I20250903 20:05:33.121518 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet 120ca61b3da742dc9a2efa483ead7e97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:33.121889 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet 02b3e262dd7841dfbb0520377eb94743 with delete_type TABLET_DATA_DELETED (replaced by tablet fa544dfdfff04a1e97676e562aeb0f40) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:33.121333 27059 rpcz_store.cc:269] 0903 20:05:32.337880 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.473435 (+135555us) service_pool.cc:225] Handling call
0903 20:05:33.120972 (+647537us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:33.069020 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:33.069183 (+ 163us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:33.069190 (+ 7us) write_op.cc:435] Acquired schema lock
0903 20:05:33.069194 (+ 4us) tablet.cc:660] Decoding operations
0903 20:05:33.069341 (+ 147us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:33.069355 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:33.069362 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0903 20:05:33.069496 (+ 134us) tablet.cc:699] Row locks acquired
0903 20:05:33.069501 (+ 5us) write_op.cc:260] PREPARE: finished
0903 20:05:33.069539 (+ 38us) write_op.cc:270] Start()
0903 20:05:33.069578 (+ 39us) write_op.cc:276] Timestamp: P: 1756929933069491 usec, L: 0
0903 20:05:33.069584 (+ 6us) op_driver.cc:348] REPLICATION: starting
0903 20:05:33.069801 (+ 217us) log.cc:844] Serialized 907 byte log entry
0903 20:05:33.114024 (+ 44223us) op_driver.cc:464] REPLICATION: finished
0903 20:05:33.114179 (+ 155us) write_op.cc:301] APPLY: starting
0903 20:05:33.114200 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:33.114294 (+ 94us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:33.114323 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:33.114408 (+ 85us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:33.114446 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0903 20:05:33.114454 (+ 8us) write_op.cc:312] APPLY: finished
0903 20:05:33.114550 (+ 96us) log.cc:844] Serialized 29 byte log entry
0903 20:05:33.114625 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:33.114652 (+ 27us) write_op.cc:454] Released schema lock
0903 20:05:33.114663 (+ 11us) write_op.cc:341] FINISH: Updating metrics
0903 20:05:33.114751 (+ 88us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:33.114754 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":76475,"reported_tablets":22,"child_traces":[["op",{"apply.queue_time_us":69,"apply.run_cpu_time_us":570,"apply.run_wall_time_us":571,"num_ops":2,"prepare.queue_time_us":249335,"prepare.run_cpu_time_us":928,"prepare.run_wall_time_us":20474,"raft.queue_time_us":40,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":253,"replication_time_us":44380}]]}
I20250903 20:05:33.122579 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet 4af8c03c9d834405b4e25d35441db961 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:33.122648 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 988) took 1022 ms. Trace:
I20250903 20:05:33.122824 27684 rpcz_store.cc:276] 0903 20:05:32.100076 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.100140 (+ 64us) service_pool.cc:225] Handling call
0903 20:05:33.064046 (+963906us) ts_tablet_manager.cc:1145] Deleting tablet b0d79c78e0024b7ba3bb3356be7c8ceb
0903 20:05:33.064057 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.073931 (+ 9874us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.093506 (+ 19575us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.122623 (+ 29117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":963629}
I20250903 20:05:33.122972 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet 607c9a9cbc5540e5bd0fa436fcba564c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:33.123410 27494 tablet_service.cc:1515] Processing DeleteTablet for tablet 63a1f26fac3b4b939ebb79c6a07d8f8e with delete_type TABLET_DATA_DELETED (replaced by tablet 4789bd72aa454b639df76bd5ad379850) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:33.123770 27499 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f27f525fdab407280aa23df1ae71b43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:33.124408 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet b0d79c78e0024b7ba3bb3356be7c8ceb (table default.Table_94e9e32ef1134f07ade1cdf8da69ead6 [id=e248f9c4b9c9406395aaea7ba3feb9ba]) successfully deleted
I20250903 20:05:33.123412 27684 tablet_replica.cc:331] T 1bf69315f07a470bb2b897ad7c590729 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:33.125665 27684 raft_consensus.cc:2241] T 1bf69315f07a470bb2b897ad7c590729 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:33.125965 27684 raft_consensus.cc:2270] T 1bf69315f07a470bb2b897ad7c590729 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.113126 29109 leader_election.cc:290] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:33.127140 27684 ts_tablet_manager.cc:1905] T 1bf69315f07a470bb2b897ad7c590729 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.132701 29044 raft_consensus.cc:491] T 5002b7e8118c49b6a247cb8dc4ca1aea P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:33.132851 29044 raft_consensus.cc:513] T 5002b7e8118c49b6a247cb8dc4ca1aea P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.133396 29044 leader_election.cc:290] T 5002b7e8118c49b6a247cb8dc4ca1aea P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:33.134019 27687 ts_tablet_manager.cc:1918] T b7a6d5d8a97a4769a81ec7224d591f8c P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.134153 27687 log.cc:1199] T b7a6d5d8a97a4769a81ec7224d591f8c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/b7a6d5d8a97a4769a81ec7224d591f8c
I20250903 20:05:33.134532 27687 ts_tablet_manager.cc:1939] T b7a6d5d8a97a4769a81ec7224d591f8c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:33.135277 27695 tablet_bootstrap.cc:492] T e877d56bbbfa4c5a92d025d0311ce5c6 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:33.135455 27695 ts_tablet_manager.cc:1397] T e877d56bbbfa4c5a92d025d0311ce5c6 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.193s user 0.019s sys 0.020s
I20250903 20:05:33.136265 27695 raft_consensus.cc:357] T e877d56bbbfa4c5a92d025d0311ce5c6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.136453 27695 raft_consensus.cc:383] T e877d56bbbfa4c5a92d025d0311ce5c6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.136523 27695 raft_consensus.cc:738] T e877d56bbbfa4c5a92d025d0311ce5c6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.136739 27695 consensus_queue.cc:260] T e877d56bbbfa4c5a92d025d0311ce5c6 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.136781 28947 raft_consensus.cc:491] T 976322efa9464ed7a2175d3e3de51dab P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:33.136963 28947 raft_consensus.cc:513] T 976322efa9464ed7a2175d3e3de51dab P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.137233 27695 ts_tablet_manager.cc:1428] T e877d56bbbfa4c5a92d025d0311ce5c6 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:33.137516 27695 tablet_bootstrap.cc:492] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:33.137591 28947 leader_election.cc:290] T 976322efa9464ed7a2175d3e3de51dab P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:33.139107 27695 tablet_bootstrap.cc:654] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:33.139194 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "976322efa9464ed7a2175d3e3de51dab" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:33.141435 29109 raft_consensus.cc:491] T 4bcd65022b7b4502983b6b58809047e3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:33.141649 29109 raft_consensus.cc:513] T 4bcd65022b7b4502983b6b58809047e3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.141737 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ac73bfe5b3b342f6ad66cd0c724af31a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:33.142563 29109 leader_election.cc:290] T 4bcd65022b7b4502983b6b58809047e3 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:33.142560 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac73bfe5b3b342f6ad66cd0c724af31a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:33.144266 27462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 52128.2us
I20250903 20:05:33.144603 27685 ts_tablet_manager.cc:1918] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:33.140476 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_334d7a39ba4b4c2babe0c815dec6a6c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:33.145851 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_334d7a39ba4b4c2babe0c815dec6a6c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:33.149269 27684 ts_tablet_manager.cc:1918] T 1bf69315f07a470bb2b897ad7c590729 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.153035 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ba9ea3be44c64888a491bcc40e505a32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:33.153609 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba9ea3be44c64888a491bcc40e505a32 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:33.149544 27517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5002b7e8118c49b6a247cb8dc4ca1aea" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:33.153771 27517 raft_consensus.cc:2466] T 5002b7e8118c49b6a247cb8dc4ca1aea P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:33.149883 27508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5002b7e8118c49b6a247cb8dc4ca1aea" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:33.154014 27508 raft_consensus.cc:3058] T 5002b7e8118c49b6a247cb8dc4ca1aea P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:33.156520 27508 raft_consensus.cc:2466] T 5002b7e8118c49b6a247cb8dc4ca1aea P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:33.157060 27684 log.cc:1199] T 1bf69315f07a470bb2b897ad7c590729 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/1bf69315f07a470bb2b897ad7c590729
I20250903 20:05:33.159022 27684 ts_tablet_manager.cc:1939] T 1bf69315f07a470bb2b897ad7c590729 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
W20250903 20:05:33.159853 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7dc5214512af449bbc537f111a3d8dbc consensus state has no leader
I20250903 20:05:33.160166 27684 tablet_replica.cc:331] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:33.160418 27684 raft_consensus.cc:2241] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:33.160718 27684 raft_consensus.cc:2270] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.161389 27684 ts_tablet_manager.cc:1905] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.146095 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "200c06f296b94eab94f35f312c5a443b" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:33.163924 27685 log.cc:1199] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/673c45f3f2a14da9bd306011d396eaca
I20250903 20:05:33.164011 27239 raft_consensus.cc:2466] T 200c06f296b94eab94f35f312c5a443b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:33.164247 27685 ts_tablet_manager.cc:1939] T 673c45f3f2a14da9bd306011d396eaca P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:33.146950 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5002b7e8118c49b6a247cb8dc4ca1aea" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:33.164458 27248 raft_consensus.cc:2391] T 5002b7e8118c49b6a247cb8dc4ca1aea P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
I20250903 20:05:33.164551 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 863) took 4737 ms. Trace:
I20250903 20:05:33.164619 27685 rpcz_store.cc:276] 0903 20:05:28.426808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.426862 (+ 54us) service_pool.cc:225] Handling call
0903 20:05:33.114375 (+4687513us) ts_tablet_manager.cc:1145] Deleting tablet 673c45f3f2a14da9bd306011d396eaca
0903 20:05:33.114387 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.129497 (+ 15110us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.144490 (+ 14993us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.164531 (+ 20041us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":251,"tablet-delete.queue_time_us":4687293}
I20250903 20:05:33.165159 27462 leader_election.cc:304] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:33.165735 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 1bf69315f07a470bb2b897ad7c590729 (table default.Table_5fb98b65f51b4d628de6bed5d36b53d1 [id=b82c8f8f0f6b415580c2b590f05c0f7a]) successfully deleted
I20250903 20:05:33.165963 29128 raft_consensus.cc:2802] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:33.166108 29128 raft_consensus.cc:491] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:33.166275 29128 raft_consensus.cc:3058] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:33.156857 27183 consensus_peers.cc:489] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:33.148182 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4bcd65022b7b4502983b6b58809047e3" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:33.170748 27462 leader_election.cc:343] T 4bcd65022b7b4502983b6b58809047e3 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:33.171025 27183 consensus_peers.cc:489] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0d79c78e0024b7ba3bb3356be7c8ceb. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:33.172271 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 673c45f3f2a14da9bd306011d396eaca (table default.Table_43ea308240ac4628a72434608a22b975 [id=8de1ad04c15648d9bcccfc56efc9e7fb]) successfully deleted
I20250903 20:05:33.172405 27695 tablet_bootstrap.cc:492] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:33.172569 27695 ts_tablet_manager.cc:1397] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20250903 20:05:33.172777 27685 tablet_replica.cc:331] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:33.173112 27685 raft_consensus.cc:2241] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:33.173414 27684 ts_tablet_manager.cc:1918] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.173666 27685 raft_consensus.cc:2270] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.173560 27695 raft_consensus.cc:357] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.173866 27695 raft_consensus.cc:383] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.173933 27695 raft_consensus.cc:738] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.174067 27684 log.cc:1199] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/b7a6d5d8a97a4769a81ec7224d591f8c
I20250903 20:05:33.174178 27685 ts_tablet_manager.cc:1905] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.175308 27684 ts_tablet_manager.cc:1939] T b7a6d5d8a97a4769a81ec7224d591f8c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:33.174147 27695 consensus_queue.cc:260] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.176719 27695 ts_tablet_manager.cc:1428] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250903 20:05:33.177170 27695 tablet_bootstrap.cc:492] T 4bcd65022b7b4502983b6b58809047e3 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:33.178962 27695 tablet_bootstrap.cc:654] T 4bcd65022b7b4502983b6b58809047e3 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:33.179123 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet b7a6d5d8a97a4769a81ec7224d591f8c (table default.Table_f752a52fa3ba4ac99bc29428fa871318 [id=21b4fd8022f6461691f87200e040e264]) successfully deleted
I20250903 20:05:33.179545 27684 tablet_replica.cc:331] T 541278ca9ffb4e41b341139c2faa337f P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:33.179807 27684 raft_consensus.cc:2241] T 541278ca9ffb4e41b341139c2faa337f P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:33.180043 27684 raft_consensus.cc:2270] T 541278ca9ffb4e41b341139c2faa337f P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.187935 27684 ts_tablet_manager.cc:1905] T 541278ca9ffb4e41b341139c2faa337f P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.191195 28903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "35bdb0090d8747059b5ee103e786e979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8857b5d4ea1e4f45aafb8e029837cca2" member_type: VOTER last_known_addr { host: "127.26.65.126" port: 39015 } } peers { permanent_uuid: "35dd69f3fa7a4fd8843b97f8a5d68296" member_type: VOTER last_known_addr { host: "127.26.65.125" port: 44059 } } peers { permanent_uuid: "35bdb0090d8747059b5ee103e786e979" member_type: VOTER last_known_addr { host: "127.26.65.124" port: 40911 } } }
I20250903 20:05:33.191419 28903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 [sys.catalog]: This master's current role is: LEADER
I20250903 20:05:33.191803 27249 raft_consensus.cc:3058] T 976322efa9464ed7a2175d3e3de51dab P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:33.192185 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_81922f27b2de428ba78422c5be13fcd9" } modify_external_catalogs: true
W20250903 20:05:33.192286 27183 consensus_peers.cc:489] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1f6cd4d475f147fe8dfcc7e3a73dedf8. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:33.192348 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_81922f27b2de428ba78422c5be13fcd9" } modify_external_catalogs: true
I20250903 20:05:33.194744 27249 raft_consensus.cc:1273] T 976322efa9464ed7a2175d3e3de51dab P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Refusing update from remote peer e72ef1dbcc5143cf9f04a808f101c80f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 20:05:33.195693 29109 consensus_queue.cc:1035] T 976322efa9464ed7a2175d3e3de51dab P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:33.201442 27685 ts_tablet_manager.cc:1918] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.204205 27684 ts_tablet_manager.cc:1918] T 541278ca9ffb4e41b341139c2faa337f P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.205139 27684 log.cc:1199] T 541278ca9ffb4e41b341139c2faa337f P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/541278ca9ffb4e41b341139c2faa337f
I20250903 20:05:33.205595 27684 ts_tablet_manager.cc:1939] T 541278ca9ffb4e41b341139c2faa337f P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:33.206667 27685 log.cc:1199] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/8a223f75045345ae833f52bcc8f239f9
I20250903 20:05:33.206883 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 541278ca9ffb4e41b341139c2faa337f (table default.Table_01215977307c453e9236e5bd72bc8427 [id=7ae71054c4d34d66a210e7c8f7460d4b]) successfully deleted
I20250903 20:05:33.205040 27692 tablet_bootstrap.cc:492] T bf459db3327640f88574fca7b5f2ca98 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:33.207309 27692 ts_tablet_manager.cc:1397] T bf459db3327640f88574fca7b5f2ca98 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.096s user 0.001s sys 0.002s
I20250903 20:05:33.207332 27685 ts_tablet_manager.cc:1939] T 8a223f75045345ae833f52bcc8f239f9 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:33.207656 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 864) took 4780 ms. Trace:
I20250903 20:05:33.208312 27692 raft_consensus.cc:357] T bf459db3327640f88574fca7b5f2ca98 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.208561 27692 raft_consensus.cc:383] T bf459db3327640f88574fca7b5f2ca98 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.208655 27692 raft_consensus.cc:738] T bf459db3327640f88574fca7b5f2ca98 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.208940 27692 consensus_queue.cc:260] T bf459db3327640f88574fca7b5f2ca98 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.209661 27692 ts_tablet_manager.cc:1428] T bf459db3327640f88574fca7b5f2ca98 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:33.209998 27692 tablet_bootstrap.cc:492] T 46f49fd718be4a2996d781ab31268217 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:33.211488 27692 tablet_bootstrap.cc:654] T 46f49fd718be4a2996d781ab31268217 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:33.211716 27684 tablet_replica.cc:331] T 512908bc4d6941ec81b145ee5795405f P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:33.212425 27684 raft_consensus.cc:2241] T 512908bc4d6941ec81b145ee5795405f P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:33.212623 27684 raft_consensus.cc:2270] T 512908bc4d6941ec81b145ee5795405f P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.213618 27684 ts_tablet_manager.cc:1905] T 512908bc4d6941ec81b145ee5795405f P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.138406 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "976322efa9464ed7a2175d3e3de51dab" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:33.149462 27320 leader_election.cc:343] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: d1c96296a579440c887ba6c2a93f7204
I20250903 20:05:33.165839 27693 tablet_bootstrap.cc:492] T 7345305820be4d32a3ceb227f045f009 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
W20250903 20:05:33.214433 27320 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 64975.3us
I20250903 20:05:33.214604 27693 ts_tablet_manager.cc:1397] T 7345305820be4d32a3ceb227f045f009 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.136s user 0.003s sys 0.000s
I20250903 20:05:33.215265 27693 raft_consensus.cc:357] T 7345305820be4d32a3ceb227f045f009 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.215423 27693 raft_consensus.cc:383] T 7345305820be4d32a3ceb227f045f009 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.215478 27693 raft_consensus.cc:738] T 7345305820be4d32a3ceb227f045f009 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.215710 27693 consensus_queue.cc:260] T 7345305820be4d32a3ceb227f045f009 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.215947 29130 consensus_queue.cc:1035] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:33.216217 27693 ts_tablet_manager.cc:1428] T 7345305820be4d32a3ceb227f045f009 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:33.216427 27382 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4bcd65022b7b4502983b6b58809047e3" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:33.217319 27462 leader_election.cc:343] T 4bcd65022b7b4502983b6b58809047e3 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:33.217339 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 970) took 1934 ms. Trace:
I20250903 20:05:33.217420 27462 leader_election.cc:304] T 4bcd65022b7b4502983b6b58809047e3 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:33.217408 27687 rpcz_store.cc:276] 0903 20:05:31.283110 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.284303 (+ 1193us) service_pool.cc:225] Handling call
0903 20:05:32.754203 (+1469900us) ts_tablet_manager.cc:1145] Deleting tablet b7a6d5d8a97a4769a81ec7224d591f8c
0903 20:05:32.754238 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.001180 (+246942us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.133904 (+132724us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.217320 (+ 83416us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1958,"tablet-delete.queue_time_us":1469719}
I20250903 20:05:33.217860 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet b7a6d5d8a97a4769a81ec7224d591f8c (table default.Table_f752a52fa3ba4ac99bc29428fa871318 [id=21b4fd8022f6461691f87200e040e264]) successfully deleted
I20250903 20:05:33.218012 28710 consensus_queue.cc:1035] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:33.218427 28710 raft_consensus.cc:2747] T 4bcd65022b7b4502983b6b58809047e3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:33.218555 28710 consensus_queue.cc:1035] T 67ca1dcf6fc7435db555392235f9900f P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:33.218936 27384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c18a98d5c0f4a43ae91c9130047b681" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:33.219347 27376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5002b7e8118c49b6a247cb8dc4ca1aea" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:33.219651 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "200c06f296b94eab94f35f312c5a443b" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:33.220206 27379 raft_consensus.cc:2391] T 976322efa9464ed7a2175d3e3de51dab P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89f8f0ee68b940db9ce250c0604f934a in current term 1: Already voted for candidate e72ef1dbcc5143cf9f04a808f101c80f in this term.
I20250903 20:05:33.216487 27693 tablet_bootstrap.cc:492] T fc634116a78f45dd833034ddda90e9bf P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:33.230105 27693 tablet_bootstrap.cc:654] T fc634116a78f45dd833034ddda90e9bf P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:33.220670 27184 leader_election.cc:304] T 976322efa9464ed7a2175d3e3de51dab P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:33.231295 29128 raft_consensus.cc:513] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.231858 29128 leader_election.cc:290] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:33.231886 27685 rpcz_store.cc:276] 0903 20:05:28.427278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.427333 (+ 55us) service_pool.cc:225] Handling call
0903 20:05:33.172725 (+4745392us) ts_tablet_manager.cc:1145] Deleting tablet 8a223f75045345ae833f52bcc8f239f9
0903 20:05:33.172734 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.184925 (+ 12191us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.201321 (+ 16396us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.207639 (+ 6318us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4745169}
I20250903 20:05:33.232591 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 8a223f75045345ae833f52bcc8f239f9 (table default.Table_04bd610b6ae24395b30dc33c4d118258 [id=02a2c253dfb64f2cac5f70bde132aceb]) successfully deleted
I20250903 20:05:33.232997 27685 tablet_replica.cc:331] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:33.233229 27685 raft_consensus.cc:2241] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:33.233626 27685 raft_consensus.cc:2270] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.234683 27685 ts_tablet_manager.cc:1905] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.235574 27684 ts_tablet_manager.cc:1918] T 512908bc4d6941ec81b145ee5795405f P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.235761 27684 log.cc:1199] T 512908bc4d6941ec81b145ee5795405f P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/512908bc4d6941ec81b145ee5795405f
I20250903 20:05:33.237350 27687 tablet_replica.cc:331] T 5f27f525fdab407280aa23df1ae71b43 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:33.237771 28874 raft_consensus.cc:2747] T 976322efa9464ed7a2175d3e3de51dab P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:33.238363 27684 ts_tablet_manager.cc:1939] T 512908bc4d6941ec81b145ee5795405f P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:33.239254 27687 raft_consensus.cc:2241] T 5f27f525fdab407280aa23df1ae71b43 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250903 20:05:33.239560 27462 leader_election.cc:343] T 4c18a98d5c0f4a43ae91c9130047b681 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:33.239729 27462 leader_election.cc:304] T 4c18a98d5c0f4a43ae91c9130047b681 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:33.239996 28710 raft_consensus.cc:2747] T 4c18a98d5c0f4a43ae91c9130047b681 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:33.240260 27462 leader_election.cc:304] T 5002b7e8118c49b6a247cb8dc4ca1aea P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:33.240454 28710 raft_consensus.cc:2747] T 5002b7e8118c49b6a247cb8dc4ca1aea P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:33.241243 27462 leader_election.cc:343] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:33.241211 28710 consensus_queue.cc:1035] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250903 20:05:33.241714 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f63b9a066ae94ef3afe3badaef5bcafd consensus state has no leader
I20250903 20:05:33.241727 29144 consensus_queue.cc:1035] T 976322efa9464ed7a2175d3e3de51dab P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:33.245568 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "200c06f296b94eab94f35f312c5a443b" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:33.239403 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "200c06f296b94eab94f35f312c5a443b" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:33.250355 27687 raft_consensus.cc:2270] T 5f27f525fdab407280aa23df1ae71b43 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.251204 27687 ts_tablet_manager.cc:1905] T 5f27f525fdab407280aa23df1ae71b43 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:33.247501 27183 consensus_peers.cc:489] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:33.254601 27249 raft_consensus.cc:3058] T 200c06f296b94eab94f35f312c5a443b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:33.257463 27249 raft_consensus.cc:2466] T 200c06f296b94eab94f35f312c5a443b P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:33.273825 27684 tablet_replica.cc:331] T 726f4d920f3e47fea41b4e12c021c988 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:33.274147 27684 raft_consensus.cc:2241] T 726f4d920f3e47fea41b4e12c021c988 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:33.274318 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 512908bc4d6941ec81b145ee5795405f (table default.Table_6d2d1ff2291b43f8ae4c0a6a456f5014 [id=60e7bfe4938540e4a3c34b248356969f]) successfully deleted
I20250903 20:05:33.274447 27684 raft_consensus.cc:2270] T 726f4d920f3e47fea41b4e12c021c988 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.275832 27684 ts_tablet_manager.cc:1905] T 726f4d920f3e47fea41b4e12c021c988 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.284847 27684 ts_tablet_manager.cc:1918] T 726f4d920f3e47fea41b4e12c021c988 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.285120 27684 log.cc:1199] T 726f4d920f3e47fea41b4e12c021c988 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/726f4d920f3e47fea41b4e12c021c988
I20250903 20:05:33.285530 27684 ts_tablet_manager.cc:1939] T 726f4d920f3e47fea41b4e12c021c988 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:33.286530 29151 tablet.cc:1722] T 976322efa9464ed7a2175d3e3de51dab P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:33.286793 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 726f4d920f3e47fea41b4e12c021c988 (table default.Table_a6608fc0162446a480d8e8ecb07db28f [id=66b42d6c99de4096998cba421c5b4663]) successfully deleted
W20250903 20:05:33.286803 27462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 44193.7us
I20250903 20:05:33.287739 27692 tablet_bootstrap.cc:492] T 46f49fd718be4a2996d781ab31268217 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:33.289122 27692 ts_tablet_manager.cc:1397] T 46f49fd718be4a2996d781ab31268217 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.079s user 0.006s sys 0.000s
I20250903 20:05:33.290025 27692 raft_consensus.cc:357] T 46f49fd718be4a2996d781ab31268217 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.290280 27692 raft_consensus.cc:383] T 46f49fd718be4a2996d781ab31268217 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.290371 27692 raft_consensus.cc:738] T 46f49fd718be4a2996d781ab31268217 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.290603 27692 consensus_queue.cc:260] T 46f49fd718be4a2996d781ab31268217 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.291132 27692 ts_tablet_manager.cc:1428] T 46f49fd718be4a2996d781ab31268217 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:33.291566 27692 tablet_bootstrap.cc:492] T ed096b452ba747f89e91edc15e6e3ac1 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
W20250903 20:05:33.292752 27462 leader_election.cc:343] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:33.293303 27462 leader_election.cc:304] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:33.293648 29044 raft_consensus.cc:2802] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:33.293821 29044 raft_consensus.cc:695] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:33.293862 29109 consensus_queue.cc:1035] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:33.294071 29044 consensus_queue.cc:237] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.299708 27695 tablet_bootstrap.cc:492] T 4bcd65022b7b4502983b6b58809047e3 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:33.299881 27695 ts_tablet_manager.cc:1397] T 4bcd65022b7b4502983b6b58809047e3 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.123s user 0.003s sys 0.000s
I20250903 20:05:33.301237 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5fb98b65f51b4d628de6bed5d36b53d1" } new_table_name: "default.Table_d6ccf445d4a743398a91f41658b38f8b" modify_external_catalogs: true
I20250903 20:05:33.301872 27685 ts_tablet_manager.cc:1918] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.302876 27692 tablet_bootstrap.cc:654] T ed096b452ba747f89e91edc15e6e3ac1 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:33.301995 27685 log.cc:1199] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/e0347eab6c8642beabdc2d7e7bbfdc9b
I20250903 20:05:33.305835 27685 ts_tablet_manager.cc:1939] T e0347eab6c8642beabdc2d7e7bbfdc9b P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:33.307240 27693 tablet_bootstrap.cc:492] T fc634116a78f45dd833034ddda90e9bf P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:33.307382 27693 ts_tablet_manager.cc:1397] T fc634116a78f45dd833034ddda90e9bf P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.091s user 0.003s sys 0.000s
I20250903 20:05:33.310437 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 865) took 4882 ms. Trace:
I20250903 20:05:33.310544 27685 rpcz_store.cc:276] 0903 20:05:28.427722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.427849 (+ 127us) service_pool.cc:225] Handling call
0903 20:05:33.232939 (+4805090us) ts_tablet_manager.cc:1145] Deleting tablet e0347eab6c8642beabdc2d7e7bbfdc9b
0903 20:05:33.232951 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.269517 (+ 36566us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.301756 (+ 32239us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.310415 (+ 8659us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4804891}
I20250903 20:05:33.311607 27685 tablet_replica.cc:331] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:33.312114 27685 raft_consensus.cc:2241] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:33.313335 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet e0347eab6c8642beabdc2d7e7bbfdc9b (table default.Table_187e391a6d6d4d0d96d4bb8a0581face [id=54ebade293b7466aa39f00ec876f27eb]) successfully deleted
I20250903 20:05:33.315886 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f92253d7e7aa42949cfa903765b446ca" } new_table_name: "default.Table_258f358903dd4f6fa7e310d72562a570" modify_external_catalogs: true
I20250903 20:05:33.319363 27684 tablet_replica.cc:331] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:33.300776 27695 raft_consensus.cc:357] T 4bcd65022b7b4502983b6b58809047e3 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.319505 27695 raft_consensus.cc:383] T 4bcd65022b7b4502983b6b58809047e3 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.319592 27695 raft_consensus.cc:738] T 4bcd65022b7b4502983b6b58809047e3 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.319952 27695 consensus_queue.cc:260] T 4bcd65022b7b4502983b6b58809047e3 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.321317 27693 raft_consensus.cc:357] T fc634116a78f45dd833034ddda90e9bf P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:33.321871 27684 raft_consensus.cc:2241] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:33.322192 27693 raft_consensus.cc:383] T fc634116a78f45dd833034ddda90e9bf P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.325260 29154 tablet.cc:1722] T 976322efa9464ed7a2175d3e3de51dab P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:33.323876 29053 raft_consensus.cc:491] T 7345305820be4d32a3ceb227f045f009 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:33.326485 29053 raft_consensus.cc:513] T 7345305820be4d32a3ceb227f045f009 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.327260 29053 leader_election.cc:290] T 7345305820be4d32a3ceb227f045f009 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:33.327860 27693 raft_consensus.cc:738] T fc634116a78f45dd833034ddda90e9bf P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.328207 27693 consensus_queue.cc:260] T fc634116a78f45dd833034ddda90e9bf P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:33.329329 27693 ts_tablet_manager.cc:1428] T fc634116a78f45dd833034ddda90e9bf P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.022s user 0.003s sys 0.000s
I20250903 20:05:33.329667 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7345305820be4d32a3ceb227f045f009" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:33.330014 27693 tablet_bootstrap.cc:492] T 200c06f296b94eab94f35f312c5a443b P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:33.330024 27249 raft_consensus.cc:2391] T 7345305820be4d32a3ceb227f045f009 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 51578fb7bceb4e119b3ab1e58c4f5528 in current term 1: Already voted for candidate e72ef1dbcc5143cf9f04a808f101c80f in this term.
I20250903 20:05:33.332628 27693 tablet_bootstrap.cc:654] T 200c06f296b94eab94f35f312c5a443b P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:33.334648 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_564e21bb80a94d5888b76f1471d9b937" } new_table_name: "default.Table_c7086111ba3d4f73b8cf50389b3ac155" modify_external_catalogs: true
I20250903 20:05:33.336335 27059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3ee4dfa744d34536bab52e4ba4f64dbd" } modify_external_catalogs: true
I20250903 20:05:33.336556 27059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3ee4dfa744d34536bab52e4ba4f64dbd" } modify_external_catalogs: true
I20250903 20:05:33.322944 29128 raft_consensus.cc:491] T bf459db3327640f88574fca7b5f2ca98 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:33.338402 29128 raft_consensus.cc:513] T bf459db3327640f88574fca7b5f2ca98 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.338573 27684 raft_consensus.cc:2270] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.339093 29128 leader_election.cc:290] T bf459db3327640f88574fca7b5f2ca98 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:33.339370 27684 ts_tablet_manager.cc:1905] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.339725 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf459db3327640f88574fca7b5f2ca98" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:33.340329 27462 leader_election.cc:343] T bf459db3327640f88574fca7b5f2ca98 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:33.343155 27508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7345305820be4d32a3ceb227f045f009" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:33.343772 27685 raft_consensus.cc:2270] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.344416 27685 ts_tablet_manager.cc:1905] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.344843 27695 ts_tablet_manager.cc:1428] T 4bcd65022b7b4502983b6b58809047e3 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.045s user 0.003s sys 0.000s
I20250903 20:05:33.345118 27695 tablet_bootstrap.cc:492] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:33.352372 27692 tablet_bootstrap.cc:492] T ed096b452ba747f89e91edc15e6e3ac1 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:33.352787 27695 tablet_bootstrap.cc:654] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:33.355374 27692 ts_tablet_manager.cc:1397] T ed096b452ba747f89e91edc15e6e3ac1 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.064s user 0.001s sys 0.005s
I20250903 20:05:33.356410 27692 raft_consensus.cc:357] T ed096b452ba747f89e91edc15e6e3ac1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.356731 27685 ts_tablet_manager.cc:1918] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.357736 27692 raft_consensus.cc:383] T ed096b452ba747f89e91edc15e6e3ac1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.357847 27692 raft_consensus.cc:738] T ed096b452ba747f89e91edc15e6e3ac1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.357232 27685 log.cc:1199] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/a72b85dfb34047528b2b28ff0d40f828
I20250903 20:05:33.358093 27692 consensus_queue.cc:260] T ed096b452ba747f89e91edc15e6e3ac1 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.360862 27685 ts_tablet_manager.cc:1939] T a72b85dfb34047528b2b28ff0d40f828 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:33.361271 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 866) took 4933 ms. Trace:
I20250903 20:05:33.361395 27685 rpcz_store.cc:276] 0903 20:05:28.428183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.428236 (+ 53us) service_pool.cc:225] Handling call
0903 20:05:33.311554 (+4883318us) ts_tablet_manager.cc:1145] Deleting tablet a72b85dfb34047528b2b28ff0d40f828
0903 20:05:33.311565 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.347500 (+ 35935us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.356582 (+ 9082us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.361253 (+ 4671us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31011,"tablet-delete.queue_time_us":4883139}
I20250903 20:05:33.361619 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_64a66d119ffd44619423ae75b74662ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:33.361848 27685 tablet_replica.cc:331] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:33.362048 27685 raft_consensus.cc:2241] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
W20250903 20:05:33.362249 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64a66d119ffd44619423ae75b74662ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:33.335103 29157 tablet.cc:1722] T 976322efa9464ed7a2175d3e3de51dab P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:33.363585 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet a72b85dfb34047528b2b28ff0d40f828 (table default.Table_2fa1d4a13c1840cd88c36e5fa2c6475f [id=d7b6b14b360e45a59034d2eed00e2e62]) successfully deleted
I20250903 20:05:33.364622 27685 raft_consensus.cc:2270] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.365187 27685 ts_tablet_manager.cc:1905] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.367108 27060 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 67ca1dcf6fc7435db555392235f9900f (table default.Table_d3945d5a36d84041b35394333dc704f9 [id=5568f788d1414c83b4359b5adf28632a]). Expected version 2 got 0
I20250903 20:05:33.367471 27060 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 2059ccbd6d884085886d7d564afe03aa (table default.Table_ea5cb66a574f4466bf7e7f635f22b426 [id=91abc1158ea047849f398926e3695927]). Expected version 1 got 0
I20250903 20:05:33.367697 27060 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 7345305820be4d32a3ceb227f045f009 (table default.Table_5f0199a660844c15a254a062c1540498 [id=52f969e50b9d466cbd14de6f252b7706]). Expected version 1 got 0
I20250903 20:05:33.367883 27060 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 976322efa9464ed7a2175d3e3de51dab (table default.Table_1ce23fe2aa6d4aceb99b7f02a2e6d04b [id=4e05a1e86a4d4319b51f2ff36f3db76b]). Expected version 1 got 0
I20250903 20:05:33.368935 27499 tablet_service.cc:1468] Processing CreateTablet for tablet 4a2e004dc38f4a7dbe7fe44f5d614374 (DEFAULT_TABLE table=default.Table_566a0ec0444945f4933ae71fbd4866e0 [id=c1bf35a1a8a24be4b8f36663c5f9aa59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.369405 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a2e004dc38f4a7dbe7fe44f5d614374. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.344633 27320 leader_election.cc:304] T 7345305820be4d32a3ceb227f045f009 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:33.374493 29053 raft_consensus.cc:3058] T 7345305820be4d32a3ceb227f045f009 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:33.375344 27058 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 67ca1dcf6fc7435db555392235f9900f (table default.Table_d3945d5a36d84041b35394333dc704f9 [id=5568f788d1414c83b4359b5adf28632a]). Expected version 2 got 0
I20250903 20:05:33.375370 27503 tablet_service.cc:1468] Processing CreateTablet for tablet e63c336cb61640809f8f104b3d66a2b2 (DEFAULT_TABLE table=default.Table_348db3632ae743b9b7d995887e840104 [id=caeeb822021e4f37bcf40b31f5b37ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.375568 27058 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 4bcd65022b7b4502983b6b58809047e3 (table default.Table_df7e7287722a45988929dfd3f0ff3eb6 [id=7ee5ea1a352941d7ba044bf48f21848a]). Expected version 2 got 0
I20250903 20:05:33.375758 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e63c336cb61640809f8f104b3d66a2b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.375819 27058 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet fc634116a78f45dd833034ddda90e9bf (table default.Table_78e7a5b76ebc469396d8a3d670ab63ee [id=fb43441ca8f2415d99a628e81ef1642f]). Expected version 2 got 0
I20250903 20:05:33.375856 27494 tablet_service.cc:1468] Processing CreateTablet for tablet 862a2fa0023844bf8ff32bb2f942c920 (DEFAULT_TABLE table=default.Table_7e7bbe82a45f42f2bb0d0a2093f129ad [id=1767a17148844b2d8307826fdf828015]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.376111 27058 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 9e98cd1d9bd246e182e2969d147e68bf (table default.Table_28a829a9988442049d50e6ae254e6274 [id=b27f08d56c3c43e6a09a465fdda4d3db]). Expected version 1 got 0
I20250903 20:05:33.376394 27494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 862a2fa0023844bf8ff32bb2f942c920. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.376423 27058 catalog_manager.cc:5582] T 7345305820be4d32a3ceb227f045f009 P e72ef1dbcc5143cf9f04a808f101c80f reported cstate change: term changed from 0 to 1, leader changed from <none> to e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67). New cstate: current_term: 1 leader_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:33.376806 27058 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 7345305820be4d32a3ceb227f045f009 (table default.Table_5f0199a660844c15a254a062c1540498 [id=52f969e50b9d466cbd14de6f252b7706]). Expected version 1 got 0
I20250903 20:05:33.377067 27058 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 976322efa9464ed7a2175d3e3de51dab (table default.Table_1ce23fe2aa6d4aceb99b7f02a2e6d04b [id=4e05a1e86a4d4319b51f2ff36f3db76b]). Expected version 1 got 0
I20250903 20:05:33.377513 29053 raft_consensus.cc:2747] T 7345305820be4d32a3ceb227f045f009 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:33.378022 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 11cfcd509e6b46e4bca740ff91867739 (DEFAULT_TABLE table=default.Table_d1ef54568fa7465bbb2e459a8d50cf6e [id=9b57e9004dc94098b3dbfefeeeb6bc24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.378067 27500 tablet_service.cc:1468] Processing CreateTablet for tablet 4ed8ed1e7cef458ab96f2cbfc17a3ac8 (DEFAULT_TABLE table=default.Table_67537b4f25e1452d98364cea8557ad42 [id=99285f329e934663830718e9fdaec531]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.378464 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 7174d36d2c7d4e4ea42472a8f7d5a5d5 (DEFAULT_TABLE table=default.Table_e289a239e8554a8891f3c7ef328c0007 [id=6fd1b57afdc04ee685c297576f2a4855]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.378815 27489 tablet_service.cc:1468] Processing CreateTablet for tablet 4440a6c2afb247e0a58a83806cf5b334 (DEFAULT_TABLE table=default.Table_77dba1c3fea74301973e35bf475df57a [id=89ffa8287aa340c59b773c65d39892a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.379170 27497 tablet_service.cc:1468] Processing CreateTablet for tablet 2770e6b0f07d468b98fc431c81a417b0 (DEFAULT_TABLE table=default.Table_a7795c6eb7c64c4ba9f47fbff01490ae [id=f3861cf0e708485cb384c0884a302f64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.379542 27496 tablet_service.cc:1468] Processing CreateTablet for tablet ea716a09d7e44a8bbd97fd855591ca63 (DEFAULT_TABLE table=default.Table_a765aae93cdc4ade9f0d2be91c3b3ecf [id=39e9fe521e2746e3b3294760a97a1ce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.379846 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 88be2ad3e4a74ad591e8edb5fb084ff5 (DEFAULT_TABLE table=default.Table_b7bf9efdd5044f078553d1b4a33825fc [id=32db435337024a7fa112176b86c7b889]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.380191 27495 tablet_service.cc:1468] Processing CreateTablet for tablet 4a4198b4841b4c2585e6c2f5c6a655d4 (DEFAULT_TABLE table=default.Table_36f5d67acb72437493d261988d5ccb28 [id=8c216327759c4647b9cabb512e8cba6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.380409 27213 tablet_service.cc:1468] Processing CreateTablet for tablet 2770e6b0f07d468b98fc431c81a417b0 (DEFAULT_TABLE table=default.Table_a7795c6eb7c64c4ba9f47fbff01490ae [id=f3861cf0e708485cb384c0884a302f64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.380548 27226 tablet_service.cc:1468] Processing CreateTablet for tablet ea716a09d7e44a8bbd97fd855591ca63 (DEFAULT_TABLE table=default.Table_a765aae93cdc4ade9f0d2be91c3b3ecf [id=39e9fe521e2746e3b3294760a97a1ce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.380774 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2770e6b0f07d468b98fc431c81a417b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.380726 27222 tablet_service.cc:1468] Processing CreateTablet for tablet 4a2e004dc38f4a7dbe7fe44f5d614374 (DEFAULT_TABLE table=default.Table_566a0ec0444945f4933ae71fbd4866e0 [id=c1bf35a1a8a24be4b8f36663c5f9aa59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.381001 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a2e004dc38f4a7dbe7fe44f5d614374. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.381033 27488 tablet_service.cc:1468] Processing CreateTablet for tablet 503dba32175c4459892c62d3f80693fb (DEFAULT_TABLE table=default.Table_5cbc8c64bdd34f028ab716135624ab49 [id=68b98ec051114850a10e37b28208f4fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.381248 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea716a09d7e44a8bbd97fd855591ca63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.381209 27217 tablet_service.cc:1468] Processing CreateTablet for tablet e63c336cb61640809f8f104b3d66a2b2 (DEFAULT_TABLE table=default.Table_348db3632ae743b9b7d995887e840104 [id=caeeb822021e4f37bcf40b31f5b37ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.381384 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 862a2fa0023844bf8ff32bb2f942c920 (DEFAULT_TABLE table=default.Table_7e7bbe82a45f42f2bb0d0a2093f129ad [id=1767a17148844b2d8307826fdf828015]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.381491 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e63c336cb61640809f8f104b3d66a2b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.381652 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 862a2fa0023844bf8ff32bb2f942c920. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.347133 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf459db3327640f88574fca7b5f2ca98" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:33.386472 27212 tablet_service.cc:1468] Processing CreateTablet for tablet 11cfcd509e6b46e4bca740ff91867739 (DEFAULT_TABLE table=default.Table_d1ef54568fa7465bbb2e459a8d50cf6e [id=9b57e9004dc94098b3dbfefeeeb6bc24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.386818 27212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11cfcd509e6b46e4bca740ff91867739. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:33.387390 27462 leader_election.cc:343] T bf459db3327640f88574fca7b5f2ca98 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:33.387509 27462 leader_election.cc:304] T bf459db3327640f88574fca7b5f2ca98 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:33.354697 27687 ts_tablet_manager.cc:1918] T 5f27f525fdab407280aa23df1ae71b43 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.387866 29128 raft_consensus.cc:2747] T bf459db3327640f88574fca7b5f2ca98 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:33.387965 27687 log.cc:1199] T 5f27f525fdab407280aa23df1ae71b43 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/5f27f525fdab407280aa23df1ae71b43
I20250903 20:05:33.388458 27687 ts_tablet_manager.cc:1939] T 5f27f525fdab407280aa23df1ae71b43 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:33.389518 27498 tablet_service.cc:1468] Processing CreateTablet for tablet ae2c8631271a4d859877ab8e175b5400 (DEFAULT_TABLE table=default.Table_93cd83ac3d974349b325ce4fa2fc27ac [id=cfa475daa78640a3a5c3564215c7aab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.390462 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 7174d36d2c7d4e4ea42472a8f7d5a5d5 (DEFAULT_TABLE table=default.Table_e289a239e8554a8891f3c7ef328c0007 [id=6fd1b57afdc04ee685c297576f2a4855]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.390785 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7174d36d2c7d4e4ea42472a8f7d5a5d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.390762 27493 tablet_service.cc:1468] Processing CreateTablet for tablet 52ee6ba19a734329acde7608327f2a78 (DEFAULT_TABLE table=default.Table_10275f9babd149008387d2c114944d27 [id=523a2be18abd4730951758e0b8b91a2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.392427 27692 ts_tablet_manager.cc:1428] T ed096b452ba747f89e91edc15e6e3ac1 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.037s user 0.001s sys 0.001s
I20250903 20:05:33.393458 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 980) took 2109 ms. Trace:
I20250903 20:05:33.393541 27687 rpcz_store.cc:276] 0903 20:05:31.284118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.284386 (+ 268us) service_pool.cc:225] Handling call
0903 20:05:33.237295 (+1952909us) ts_tablet_manager.cc:1145] Deleting tablet 5f27f525fdab407280aa23df1ae71b43
0903 20:05:33.237306 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.323226 (+ 85920us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.354573 (+ 31347us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.393436 (+ 38863us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43,"tablet-delete.queue_time_us":1952804}
I20250903 20:05:33.393883 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 976) took 2110 ms. Trace:
I20250903 20:05:33.393952 27687 rpcz_store.cc:276] 0903 20:05:31.283744 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.284495 (+ 751us) service_pool.cc:225] Handling call
0903 20:05:33.393794 (+2109299us) ts_tablet_manager.cc:1145] Deleting tablet 7e2f98f760cf4df4bc2a10530e2157c6
0903 20:05:33.393803 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.393870 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2109162}
I20250903 20:05:33.394200 27687 tablet_replica.cc:331] T 1bf69315f07a470bb2b897ad7c590729 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:33.394392 27687 raft_consensus.cc:2241] T 1bf69315f07a470bb2b897ad7c590729 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:33.394536 27687 raft_consensus.cc:2270] T 1bf69315f07a470bb2b897ad7c590729 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.394632 27227 tablet_service.cc:1468] Processing CreateTablet for tablet 4440a6c2afb247e0a58a83806cf5b334 (DEFAULT_TABLE table=default.Table_77dba1c3fea74301973e35bf475df57a [id=89ffa8287aa340c59b773c65d39892a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.395020 27687 ts_tablet_manager.cc:1905] T 1bf69315f07a470bb2b897ad7c590729 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.395020 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4440a6c2afb247e0a58a83806cf5b334. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.398489 27216 tablet_service.cc:1468] Processing CreateTablet for tablet 4ed8ed1e7cef458ab96f2cbfc17a3ac8 (DEFAULT_TABLE table=default.Table_67537b4f25e1452d98364cea8557ad42 [id=99285f329e934663830718e9fdaec531]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.398823 27216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ed8ed1e7cef458ab96f2cbfc17a3ac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.402580 27221 tablet_service.cc:1468] Processing CreateTablet for tablet 88be2ad3e4a74ad591e8edb5fb084ff5 (DEFAULT_TABLE table=default.Table_b7bf9efdd5044f078553d1b4a33825fc [id=32db435337024a7fa112176b86c7b889]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.402935 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88be2ad3e4a74ad591e8edb5fb084ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.405112 27685 ts_tablet_manager.cc:1918] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.405208 27685 log.cc:1199] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/466e501e424e46bc98e8bbc64f97005e
I20250903 20:05:33.405520 27685 ts_tablet_manager.cc:1939] T 466e501e424e46bc98e8bbc64f97005e P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:33.405813 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 867) took 4977 ms. Trace:
I20250903 20:05:33.405891 27685 rpcz_store.cc:276] 0903 20:05:28.428550 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.428599 (+ 49us) service_pool.cc:225] Handling call
0903 20:05:33.361803 (+4933204us) ts_tablet_manager.cc:1145] Deleting tablet 466e501e424e46bc98e8bbc64f97005e
0903 20:05:33.361811 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.374304 (+ 12493us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.405013 (+ 30709us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.405796 (+ 783us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4932995}
I20250903 20:05:33.406193 27685 tablet_replica.cc:331] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:33.406400 27685 raft_consensus.cc:2241] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:33.406529 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 4a4198b4841b4c2585e6c2f5c6a655d4 (DEFAULT_TABLE table=default.Table_36f5d67acb72437493d261988d5ccb28 [id=8c216327759c4647b9cabb512e8cba6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.406651 27685 raft_consensus.cc:2270] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.406855 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a4198b4841b4c2585e6c2f5c6a655d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.407181 27685 ts_tablet_manager.cc:1905] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.412516 27685 ts_tablet_manager.cc:1918] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.412627 27685 log.cc:1199] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/0a9b7770432942da9373d5100c71bd6a
I20250903 20:05:33.412976 27685 ts_tablet_manager.cc:1939] T 0a9b7770432942da9373d5100c71bd6a P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:33.413290 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 868) took 4984 ms. Trace:
I20250903 20:05:33.413367 27685 rpcz_store.cc:276] 0903 20:05:28.429007 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.429056 (+ 49us) service_pool.cc:225] Handling call
0903 20:05:33.406152 (+4977096us) ts_tablet_manager.cc:1145] Deleting tablet 0a9b7770432942da9373d5100c71bd6a
0903 20:05:33.406160 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.409714 (+ 3554us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.412425 (+ 2711us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.413271 (+ 846us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4976909}
I20250903 20:05:33.413686 27685 tablet_replica.cc:331] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:33.413854 27685 raft_consensus.cc:2241] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:33.414196 27685 raft_consensus.cc:2270] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.414760 27685 ts_tablet_manager.cc:1905] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.415448 27359 tablet_service.cc:1468] Processing CreateTablet for tablet 7174d36d2c7d4e4ea42472a8f7d5a5d5 (DEFAULT_TABLE table=default.Table_e289a239e8554a8891f3c7ef328c0007 [id=6fd1b57afdc04ee685c297576f2a4855]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.415524 27348 tablet_service.cc:1468] Processing CreateTablet for tablet 2770e6b0f07d468b98fc431c81a417b0 (DEFAULT_TABLE table=default.Table_a7795c6eb7c64c4ba9f47fbff01490ae [id=f3861cf0e708485cb384c0884a302f64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.415935 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 4a2e004dc38f4a7dbe7fe44f5d614374 (DEFAULT_TABLE table=default.Table_566a0ec0444945f4933ae71fbd4866e0 [id=c1bf35a1a8a24be4b8f36663c5f9aa59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.416445 27350 tablet_service.cc:1468] Processing CreateTablet for tablet 862a2fa0023844bf8ff32bb2f942c920 (DEFAULT_TABLE table=default.Table_7e7bbe82a45f42f2bb0d0a2093f129ad [id=1767a17148844b2d8307826fdf828015]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.416889 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 11cfcd509e6b46e4bca740ff91867739 (DEFAULT_TABLE table=default.Table_d1ef54568fa7465bbb2e459a8d50cf6e [id=9b57e9004dc94098b3dbfefeeeb6bc24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.417336 27353 tablet_service.cc:1468] Processing CreateTablet for tablet 4440a6c2afb247e0a58a83806cf5b334 (DEFAULT_TABLE table=default.Table_77dba1c3fea74301973e35bf475df57a [id=89ffa8287aa340c59b773c65d39892a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.417786 27352 tablet_service.cc:1468] Processing CreateTablet for tablet e63c336cb61640809f8f104b3d66a2b2 (DEFAULT_TABLE table=default.Table_348db3632ae743b9b7d995887e840104 [id=caeeb822021e4f37bcf40b31f5b37ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.418258 27357 tablet_service.cc:1468] Processing CreateTablet for tablet ea716a09d7e44a8bbd97fd855591ca63 (DEFAULT_TABLE table=default.Table_a765aae93cdc4ade9f0d2be91c3b3ecf [id=39e9fe521e2746e3b3294760a97a1ce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.418725 27349 tablet_service.cc:1468] Processing CreateTablet for tablet 4ed8ed1e7cef458ab96f2cbfc17a3ac8 (DEFAULT_TABLE table=default.Table_67537b4f25e1452d98364cea8557ad42 [id=99285f329e934663830718e9fdaec531]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.419191 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 88be2ad3e4a74ad591e8edb5fb084ff5 (DEFAULT_TABLE table=default.Table_b7bf9efdd5044f078553d1b4a33825fc [id=32db435337024a7fa112176b86c7b889]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.419662 27347 tablet_service.cc:1468] Processing CreateTablet for tablet 4a4198b4841b4c2585e6c2f5c6a655d4 (DEFAULT_TABLE table=default.Table_36f5d67acb72437493d261988d5ccb28 [id=8c216327759c4647b9cabb512e8cba6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.420156 28874 raft_consensus.cc:491] T 20c25c84c6fb4609a1964a4ff3c526da P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51578fb7bceb4e119b3ab1e58c4f5528)
I20250903 20:05:33.420321 28874 raft_consensus.cc:513] T 20c25c84c6fb4609a1964a4ff3c526da P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:33.421054 28874 leader_election.cc:290] T 20c25c84c6fb4609a1964a4ff3c526da P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:33.421557 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ed8ed1e7cef458ab96f2cbfc17a3ac8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:33.421988 27183 leader_election.cc:343] T 20c25c84c6fb4609a1964a4ff3c526da P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 20c25c84c6fb4609a1964a4ff3c526da
I20250903 20:05:33.429505 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7174d36d2c7d4e4ea42472a8f7d5a5d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.431221 27684 ts_tablet_manager.cc:1918] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.431331 27684 log.cc:1199] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/a0f79831866b40c49c5b27ce2014c3f6
I20250903 20:05:33.431684 27684 ts_tablet_manager.cc:1939] T a0f79831866b40c49c5b27ce2014c3f6 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:33.432473 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1001) took 1132 ms. Trace:
I20250903 20:05:33.432554 27684 rpcz_store.cc:276] 0903 20:05:32.300027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.300094 (+ 67us) service_pool.cc:225] Handling call
0903 20:05:33.319282 (+1019188us) ts_tablet_manager.cc:1145] Deleting tablet a0f79831866b40c49c5b27ce2014c3f6
0903 20:05:33.319297 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.358992 (+ 39695us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.431105 (+ 72113us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.432450 (+ 1345us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1018945}
I20250903 20:05:33.432987 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1003) took 1131 ms. Trace:
I20250903 20:05:33.433051 27684 rpcz_store.cc:276] 0903 20:05:32.301190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.301263 (+ 73us) service_pool.cc:225] Handling call
0903 20:05:33.432892 (+1131629us) ts_tablet_manager.cc:1145] Deleting tablet a4f3230f50ec4c95b528fead8c191224
0903 20:05:33.432902 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.432971 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1131113}
I20250903 20:05:33.433143 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet a0f79831866b40c49c5b27ce2014c3f6 (table default.Table_3d20da97fa86497e9376f8aa69f1c3fd [id=35b0c6c60b9b43b5ad6c376284f45da5]) successfully deleted
I20250903 20:05:33.433295 27684 tablet_replica.cc:331] T 22bc87d1a4804b6c80f77f9692500d63 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:33.433460 27684 raft_consensus.cc:2241] T 22bc87d1a4804b6c80f77f9692500d63 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
W20250903 20:05:33.433614 27051 catalog_manager.cc:4895] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet a4f3230f50ec4c95b528fead8c191224 because the tablet was not found. No further retry: Not found: Tablet not found: a4f3230f50ec4c95b528fead8c191224
I20250903 20:05:33.433604 27684 raft_consensus.cc:2270] T 22bc87d1a4804b6c80f77f9692500d63 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.434185 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7174d36d2c7d4e4ea42472a8f7d5a5d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.434407 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e63c336cb61640809f8f104b3d66a2b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.434458 27692 tablet_bootstrap.cc:492] T 950a6fcd35f64f2d889831fe9cdf503a P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:33.435858 27693 tablet_bootstrap.cc:492] T 200c06f296b94eab94f35f312c5a443b P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:33.435988 27693 ts_tablet_manager.cc:1397] T 200c06f296b94eab94f35f312c5a443b P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.106s user 0.004s sys 0.000s
I20250903 20:05:33.436172 27692 tablet_bootstrap.cc:654] T 950a6fcd35f64f2d889831fe9cdf503a P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:33.436826 27693 raft_consensus.cc:357] T 200c06f296b94eab94f35f312c5a443b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.437001 27693 raft_consensus.cc:383] T 200c06f296b94eab94f35f312c5a443b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.437062 27693 raft_consensus.cc:738] T 200c06f296b94eab94f35f312c5a443b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.437438 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11cfcd509e6b46e4bca740ff91867739. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.437445 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2770e6b0f07d468b98fc431c81a417b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.438309 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11cfcd509e6b46e4bca740ff91867739. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.438419 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a4198b4841b4c2585e6c2f5c6a655d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.441447 27489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4440a6c2afb247e0a58a83806cf5b334. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.443073 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a2e004dc38f4a7dbe7fe44f5d614374. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.446266 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88be2ad3e4a74ad591e8edb5fb084ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.446529 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2770e6b0f07d468b98fc431c81a417b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.449455 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ed8ed1e7cef458ab96f2cbfc17a3ac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.450320 27353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4440a6c2afb247e0a58a83806cf5b334. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.450435 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea716a09d7e44a8bbd97fd855591ca63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.451074 27693 consensus_queue.cc:260] T 200c06f296b94eab94f35f312c5a443b P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.453470 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a4198b4841b4c2585e6c2f5c6a655d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.454433 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88be2ad3e4a74ad591e8edb5fb084ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.458274 27684 ts_tablet_manager.cc:1905] T 22bc87d1a4804b6c80f77f9692500d63 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.458390 28874 raft_consensus.cc:491] T f9709dfd18654c5bb071e39da7c8889f P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51578fb7bceb4e119b3ab1e58c4f5528)
I20250903 20:05:33.458459 27498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae2c8631271a4d859877ab8e175b5400. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.458539 28874 raft_consensus.cc:513] T f9709dfd18654c5bb071e39da7c8889f P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:33.459139 28874 leader_election.cc:290] T f9709dfd18654c5bb071e39da7c8889f P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
W20250903 20:05:33.460390 27183 leader_election.cc:343] T f9709dfd18654c5bb071e39da7c8889f P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: f9709dfd18654c5bb071e39da7c8889f
I20250903 20:05:33.462699 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52ee6ba19a734329acde7608327f2a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.464355 27693 ts_tablet_manager.cc:1428] T 200c06f296b94eab94f35f312c5a443b P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20250903 20:05:33.466626 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 503dba32175c4459892c62d3f80693fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.468755 27685 ts_tablet_manager.cc:1918] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.470021 27693 tablet_bootstrap.cc:492] T 4bcd65022b7b4502983b6b58809047e3 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:33.473412 27693 tablet_bootstrap.cc:654] T 4bcd65022b7b4502983b6b58809047e3 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:33.475631 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea716a09d7e44a8bbd97fd855591ca63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.478350 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 862a2fa0023844bf8ff32bb2f942c920. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.481096 27687 ts_tablet_manager.cc:1918] T 1bf69315f07a470bb2b897ad7c590729 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.481209 27687 log.cc:1199] T 1bf69315f07a470bb2b897ad7c590729 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/1bf69315f07a470bb2b897ad7c590729
I20250903 20:05:33.481562 27687 ts_tablet_manager.cc:1939] T 1bf69315f07a470bb2b897ad7c590729 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:33.482395 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 974) took 2198 ms. Trace:
I20250903 20:05:33.482477 27687 rpcz_store.cc:276] 0903 20:05:31.283481 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.284661 (+ 1180us) service_pool.cc:225] Handling call
0903 20:05:33.394151 (+2109490us) ts_tablet_manager.cc:1145] Deleting tablet 1bf69315f07a470bb2b897ad7c590729
0903 20:05:33.394158 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.398039 (+ 3881us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.480979 (+ 82940us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.482373 (+ 1394us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2109388}
I20250903 20:05:33.485514 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 466e501e424e46bc98e8bbc64f97005e (table default.Table_83ea6dfcee144a4fbc86931567b676bb [id=f321a32c1ac043238c2fc92bfd5ec1ae]) successfully deleted
I20250903 20:05:33.485765 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 0a9b7770432942da9373d5100c71bd6a (table default.Table_6656b728e12a482fa9e7893ef4842ac7 [id=6788b524a8654e34a4d9ee8cc8fc84d3]) successfully deleted
I20250903 20:05:33.468858 27685 log.cc:1199] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/607c9a9cbc5540e5bd0fa436fcba564c
I20250903 20:05:33.485945 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 5f27f525fdab407280aa23df1ae71b43 (table default.Table_1c679659094d4666aaaf68f76f667890 [id=61a25739b03042f7bd5bb693b68656d5]) successfully deleted
W20250903 20:05:33.486195 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 7e2f98f760cf4df4bc2a10530e2157c6 because the tablet was not found. No further retry: Not found: Tablet not found: 7e2f98f760cf4df4bc2a10530e2157c6
I20250903 20:05:33.486369 27685 ts_tablet_manager.cc:1939] T 607c9a9cbc5540e5bd0fa436fcba564c P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:33.486697 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 869) took 5057 ms. Trace:
I20250903 20:05:33.486781 27685 rpcz_store.cc:276] 0903 20:05:28.429377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.429418 (+ 41us) service_pool.cc:225] Handling call
0903 20:05:33.413640 (+4984222us) ts_tablet_manager.cc:1145] Deleting tablet 607c9a9cbc5540e5bd0fa436fcba564c
0903 20:05:33.413649 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.465934 (+ 52285us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.468668 (+ 2734us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.486680 (+ 18012us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4984048}
I20250903 20:05:33.488049 27685 tablet_replica.cc:331] T 20c25c84c6fb4609a1964a4ff3c526da P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:33.488253 27685 raft_consensus.cc:2241] T 20c25c84c6fb4609a1964a4ff3c526da P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:33.488420 27685 raft_consensus.cc:2270] T 20c25c84c6fb4609a1964a4ff3c526da P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.488998 27347 tablet_service.cc:1468] Processing CreateTablet for tablet 52ee6ba19a734329acde7608327f2a78 (DEFAULT_TABLE table=default.Table_10275f9babd149008387d2c114944d27 [id=523a2be18abd4730951758e0b8b91a2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.489404 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52ee6ba19a734329acde7608327f2a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.489866 27684 ts_tablet_manager.cc:1918] T 22bc87d1a4804b6c80f77f9692500d63 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.489961 27684 log.cc:1199] T 22bc87d1a4804b6c80f77f9692500d63 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/22bc87d1a4804b6c80f77f9692500d63
I20250903 20:05:33.489944 27685 ts_tablet_manager.cc:1905] T 20c25c84c6fb4609a1964a4ff3c526da P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.490345 27684 ts_tablet_manager.cc:1939] T 22bc87d1a4804b6c80f77f9692500d63 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:33.491060 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1002) took 1190 ms. Trace:
I20250903 20:05:33.491138 27684 rpcz_store.cc:276] 0903 20:05:32.300902 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.300970 (+ 68us) service_pool.cc:225] Handling call
0903 20:05:33.433252 (+1132282us) ts_tablet_manager.cc:1145] Deleting tablet 22bc87d1a4804b6c80f77f9692500d63
0903 20:05:33.433259 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.465278 (+ 32019us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.489752 (+ 24474us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.491033 (+ 1281us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1131209}
I20250903 20:05:33.491609 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 22bc87d1a4804b6c80f77f9692500d63 (table default.Table_c967f8b0d1314f589eae529cc492493b [id=93b379ddcb834087a42ce197c318f764]) successfully deleted
I20250903 20:05:33.492126 27684 tablet_replica.cc:331] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
W20250903 20:05:33.492179 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 653eade3172344d7aa801edf906ae1d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 653eade3172344d7aa801edf906ae1d4 already in progress: creating tablet
I20250903 20:05:33.492316 27684 raft_consensus.cc:2241] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
W20250903 20:05:33.492398 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet e35574d360334150bb494e13f55d3656 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e35574d360334150bb494e13f55d3656 already in progress: creating tablet
W20250903 20:05:33.492585 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 8330ab3e4b6f4560b9e41a9df280c211 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8330ab3e4b6f4560b9e41a9df280c211 already in progress: creating tablet
I20250903 20:05:33.492611 27684 raft_consensus.cc:2270] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:33.492794 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet a09771c026a9474c9f61ed259d843cca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a09771c026a9474c9f61ed259d843cca already in progress: creating tablet
I20250903 20:05:33.493140 27684 ts_tablet_manager.cc:1905] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:33.493167 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 931404d9b3844d908baaed3190d57b56 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 931404d9b3844d908baaed3190d57b56 already in progress: creating tablet
I20250903 20:05:33.493638 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 978) took 2209 ms. Trace:
I20250903 20:05:33.493712 27687 rpcz_store.cc:276] 0903 20:05:31.283924 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.284792 (+ 868us) service_pool.cc:225] Handling call
0903 20:05:33.482772 (+2197980us) ts_tablet_manager.cc:1145] Deleting tablet 59a1dd48d76241d7ba817070fbffd853
0903 20:05:33.482781 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.493616 (+ 10835us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2197879}
I20250903 20:05:33.494024 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 993) took 1903 ms. Trace:
I20250903 20:05:33.494081 27687 rpcz_store.cc:276] 0903 20:05:31.590086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.590135 (+ 49us) service_pool.cc:225] Handling call
0903 20:05:33.493929 (+1903794us) ts_tablet_manager.cc:1145] Deleting tablet 17514a2af39b402abbe8263d7c1abc7f
0903 20:05:33.493939 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.494012 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1903572}
I20250903 20:05:33.494436 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 991) took 1906 ms. Trace:
I20250903 20:05:33.494527 27361 tablet_service.cc:1468] Processing CreateTablet for tablet ae2c8631271a4d859877ab8e175b5400 (DEFAULT_TABLE table=default.Table_93cd83ac3d974349b325ce4fa2fc27ac [id=cfa475daa78640a3a5c3564215c7aab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.494977 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae2c8631271a4d859877ab8e175b5400. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:33.495121 27184 leader_election.cc:343] T 20c25c84c6fb4609a1964a4ff3c526da P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 20c25c84c6fb4609a1964a4ff3c526da
I20250903 20:05:33.495234 27184 leader_election.cc:304] T 20c25c84c6fb4609a1964a4ff3c526da P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:33.495374 27184 raft_consensus.cc:2684] T 20c25c84c6fb4609a1964a4ff3c526da P 89f8f0ee68b940db9ce250c0604f934a: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250903 20:05:33.495952 27213 tablet_service.cc:1468] Processing CreateTablet for tablet 503dba32175c4459892c62d3f80693fb (DEFAULT_TABLE table=default.Table_5cbc8c64bdd34f028ab716135624ab49 [id=68b98ec051114850a10e37b28208f4fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.496086 27226 tablet_service.cc:1468] Processing CreateTablet for tablet 52ee6ba19a734329acde7608327f2a78 (DEFAULT_TABLE table=default.Table_10275f9babd149008387d2c114944d27 [id=523a2be18abd4730951758e0b8b91a2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.496299 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 503dba32175c4459892c62d3f80693fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.496598 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52ee6ba19a734329acde7608327f2a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.496759 27217 tablet_service.cc:1468] Processing CreateTablet for tablet ae2c8631271a4d859877ab8e175b5400 (DEFAULT_TABLE table=default.Table_93cd83ac3d974349b325ce4fa2fc27ac [id=cfa475daa78640a3a5c3564215c7aab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.497049 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae2c8631271a4d859877ab8e175b5400. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.505846 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 607c9a9cbc5540e5bd0fa436fcba564c (table default.Table_5b44669e91a8427c8e7d96b866fdb3b5 [id=96167b0d9f9b47168a9fc59635b7136d]) successfully deleted
I20250903 20:05:33.506642 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 1bf69315f07a470bb2b897ad7c590729 (table default.Table_5fb98b65f51b4d628de6bed5d36b53d1 [id=b82c8f8f0f6b415580c2b590f05c0f7a]) successfully deleted
W20250903 20:05:33.507200 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 59a1dd48d76241d7ba817070fbffd853 because the tablet was not found. No further retry: Not found: Tablet not found: 59a1dd48d76241d7ba817070fbffd853
I20250903 20:05:33.508571 27693 tablet_bootstrap.cc:492] T 4bcd65022b7b4502983b6b58809047e3 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:33.508705 27693 ts_tablet_manager.cc:1397] T 4bcd65022b7b4502983b6b58809047e3 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20250903 20:05:33.509490 27693 raft_consensus.cc:357] T 4bcd65022b7b4502983b6b58809047e3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.509661 27693 raft_consensus.cc:383] T 4bcd65022b7b4502983b6b58809047e3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.509712 27693 raft_consensus.cc:738] T 4bcd65022b7b4502983b6b58809047e3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.509907 27693 consensus_queue.cc:260] T 4bcd65022b7b4502983b6b58809047e3 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.494517 27687 rpcz_store.cc:276] 0903 20:05:31.588230 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.596687 (+ 8457us) service_pool.cc:225] Handling call
0903 20:05:33.494314 (+1897627us) ts_tablet_manager.cc:1145] Deleting tablet e64c4a9179b6491ba92df75e0cc1bc50
0903 20:05:33.494325 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.494417 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1896598}
W20250903 20:05:33.510424 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 17514a2af39b402abbe8263d7c1abc7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17514a2af39b402abbe8263d7c1abc7f already in progress: creating tablet
I20250903 20:05:33.510810 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 992) took 1922 ms. Trace:
I20250903 20:05:33.514154 28710 raft_consensus.cc:491] T 46f49fd718be4a2996d781ab31268217 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:33.514333 28710 raft_consensus.cc:513] T 46f49fd718be4a2996d781ab31268217 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.514894 28710 leader_election.cc:290] T 46f49fd718be4a2996d781ab31268217 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:33.515563 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46f49fd718be4a2996d781ab31268217" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:33.516075 27462 leader_election.cc:343] T 46f49fd718be4a2996d781ab31268217 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:33.516553 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46f49fd718be4a2996d781ab31268217" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:33.518436 27320 consensus_peers.cc:489] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22bc87d1a4804b6c80f77f9692500d63. This is attempt 1: this message will repeat every 5th retry.
W20250903 20:05:33.523213 27462 leader_election.cc:343] T 46f49fd718be4a2996d781ab31268217 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:33.523352 27462 leader_election.cc:304] T 46f49fd718be4a2996d781ab31268217 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:33.523602 28710 raft_consensus.cc:2747] T 46f49fd718be4a2996d781ab31268217 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:33.522799 27687 rpcz_store.cc:276] 0903 20:05:31.588377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.603021 (+ 14644us) service_pool.cc:225] Handling call
0903 20:05:33.510672 (+1907651us) ts_tablet_manager.cc:1145] Deleting tablet 3f303233e6c347289650c6cf83f8127f
0903 20:05:33.510682 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.510791 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1905621}
I20250903 20:05:33.524356 27693 ts_tablet_manager.cc:1428] T 4bcd65022b7b4502983b6b58809047e3 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.016s user 0.002s sys 0.001s
I20250903 20:05:33.530359 27693 tablet_bootstrap.cc:492] T 4c18a98d5c0f4a43ae91c9130047b681 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:33.530997 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1002) took 1656 ms. Trace:
I20250903 20:05:33.531085 27687 rpcz_store.cc:276] 0903 20:05:31.874407 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.874590 (+ 183us) service_pool.cc:225] Handling call
0903 20:05:33.524560 (+1649970us) ts_tablet_manager.cc:1145] Deleting tablet b5bff419ed31453e80ceec23ea05caad
0903 20:05:33.530840 (+ 6280us) spinlock_profiling.cc:243] Waited 6.01 ms on lock 0x61800000d518. stack: 00007efcbf8c4dc6 00007efcbf8c4b80 00007efcbe8ac3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007efcbf8ad6cd 00007efcbf8ae73e 00007efcd2f768ed 00007efcd2ef81d3 00007efccf142676 00007efccf1439f9 00007efccf176344 00007efccf157da0 00007efccf157b2d 0000000000a774dc
0903 20:05:33.530853 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.530977 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13811584,"tablet-delete.queue_time_us":1649765}
I20250903 20:05:33.533561 27687 tablet_replica.cc:331] T c8a293c6fa034e9fb29682439cb3bcc2 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:33.534125 27685 ts_tablet_manager.cc:1918] T 20c25c84c6fb4609a1964a4ff3c526da P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.534451 27687 raft_consensus.cc:2241] T c8a293c6fa034e9fb29682439cb3bcc2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:33.534678 27687 raft_consensus.cc:2270] T c8a293c6fa034e9fb29682439cb3bcc2 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.535277 27687 ts_tablet_manager.cc:1905] T c8a293c6fa034e9fb29682439cb3bcc2 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.535424 27685 log.cc:1199] T 20c25c84c6fb4609a1964a4ff3c526da P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/20c25c84c6fb4609a1964a4ff3c526da
I20250903 20:05:33.535887 27685 ts_tablet_manager.cc:1939] T 20c25c84c6fb4609a1964a4ff3c526da P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:33.532788 27693 tablet_bootstrap.cc:654] T 4c18a98d5c0f4a43ae91c9130047b681 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:33.537616 27184 leader_election.cc:343] T f9709dfd18654c5bb071e39da7c8889f P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: f9709dfd18654c5bb071e39da7c8889f
I20250903 20:05:33.537753 27184 leader_election.cc:304] T f9709dfd18654c5bb071e39da7c8889f P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:33.537997 28808 raft_consensus.cc:2747] T f9709dfd18654c5bb071e39da7c8889f P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250903 20:05:33.538519 27184 consensus_peers.cc:489] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 512908bc4d6941ec81b145ee5795405f. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:33.539075 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 870) took 5109 ms. Trace:
I20250903 20:05:33.539168 27685 rpcz_store.cc:276] 0903 20:05:28.429765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.429815 (+ 50us) service_pool.cc:225] Handling call
0903 20:05:33.487076 (+5057261us) ts_tablet_manager.cc:1145] Deleting tablet 20c25c84c6fb4609a1964a4ff3c526da
0903 20:05:33.487084 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.492815 (+ 5731us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.534011 (+ 41196us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.539058 (+ 5047us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5057055}
I20250903 20:05:33.540241 27685 tablet_replica.cc:331] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:33.540608 27685 raft_consensus.cc:2241] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:33.540983 27685 raft_consensus.cc:2270] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.542878 27685 ts_tablet_manager.cc:1905] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.547170 29108 raft_consensus.cc:491] T fc634116a78f45dd833034ddda90e9bf P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:33.547335 29108 raft_consensus.cc:513] T fc634116a78f45dd833034ddda90e9bf P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:33.547417 27684 ts_tablet_manager.cc:1918] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 20:05:33.547916 27684 log.cc:1199] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/9367b3a22f624c4087848099de32c080
I20250903 20:05:33.547984 29108 leader_election.cc:290] T fc634116a78f45dd833034ddda90e9bf P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:33.548274 27684 ts_tablet_manager.cc:1939] T 9367b3a22f624c4087848099de32c080 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:33.548717 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc634116a78f45dd833034ddda90e9bf" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:33.549033 27508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc634116a78f45dd833034ddda90e9bf" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:33.549247 27508 raft_consensus.cc:2391] T fc634116a78f45dd833034ddda90e9bf P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 51578fb7bceb4e119b3ab1e58c4f5528 in current term 1: Already voted for candidate 89f8f0ee68b940db9ce250c0604f934a in this term.
I20250903 20:05:33.549849 27321 leader_election.cc:304] T fc634116a78f45dd833034ddda90e9bf P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:33.550575 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 9367b3a22f624c4087848099de32c080 (table default.Table_08918e8b515e4487bf4aee0a33234bd5 [id=891b259c1c6b44e2a82dc982c24e6cb6]) successfully deleted
I20250903 20:05:33.550920 29108 raft_consensus.cc:3058] T fc634116a78f45dd833034ddda90e9bf P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:33.552098 27684 tablet_replica.cc:331] T a9f43781299e47a8aecc37a0b73d7a64 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:33.552285 27684 raft_consensus.cc:2241] T a9f43781299e47a8aecc37a0b73d7a64 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:33.552452 27684 raft_consensus.cc:2270] T a9f43781299e47a8aecc37a0b73d7a64 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.553038 27684 ts_tablet_manager.cc:1905] T a9f43781299e47a8aecc37a0b73d7a64 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.553309 27687 ts_tablet_manager.cc:1918] T c8a293c6fa034e9fb29682439cb3bcc2 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 20:05:33.553408 27687 log.cc:1199] T c8a293c6fa034e9fb29682439cb3bcc2 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/c8a293c6fa034e9fb29682439cb3bcc2
I20250903 20:05:33.553457 29108 raft_consensus.cc:2747] T fc634116a78f45dd833034ddda90e9bf P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:33.554134 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet df9d0f3662a245d7bc7cc184bc9ef3db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df9d0f3662a245d7bc7cc184bc9ef3db already in progress: creating tablet
W20250903 20:05:33.554489 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet df9d0f3662a245d7bc7cc184bc9ef3db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df9d0f3662a245d7bc7cc184bc9ef3db already in progress: creating tablet
I20250903 20:05:33.554881 27687 ts_tablet_manager.cc:1939] T c8a293c6fa034e9fb29682439cb3bcc2 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:33.555428 27693 tablet_bootstrap.cc:492] T 4c18a98d5c0f4a43ae91c9130047b681 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:33.555585 27693 ts_tablet_manager.cc:1397] T 4c18a98d5c0f4a43ae91c9130047b681 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20250903 20:05:33.555963 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 503dba32175c4459892c62d3f80693fb (DEFAULT_TABLE table=default.Table_5cbc8c64bdd34f028ab716135624ab49 [id=68b98ec051114850a10e37b28208f4fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:33.556386 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 503dba32175c4459892c62d3f80693fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:33.556349 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1003) took 1656 ms. Trace:
I20250903 20:05:33.556435 27693 raft_consensus.cc:357] T 4c18a98d5c0f4a43ae91c9130047b681 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.556641 27693 raft_consensus.cc:383] T 4c18a98d5c0f4a43ae91c9130047b681 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.556722 27693 raft_consensus.cc:738] T 4c18a98d5c0f4a43ae91c9130047b681 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.556995 27693 consensus_queue.cc:260] T 4c18a98d5c0f4a43ae91c9130047b681 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.557552 27693 ts_tablet_manager.cc:1428] T 4c18a98d5c0f4a43ae91c9130047b681 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:33.556529 27687 rpcz_store.cc:276] 0903 20:05:31.899586 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.899678 (+ 92us) service_pool.cc:225] Handling call
0903 20:05:33.533483 (+1633805us) ts_tablet_manager.cc:1145] Deleting tablet c8a293c6fa034e9fb29682439cb3bcc2
0903 20:05:33.533494 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.550272 (+ 16778us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.553225 (+ 2953us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.556330 (+ 3105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1633545}
I20250903 20:05:33.559538 27685 ts_tablet_manager.cc:1918] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.558085 27693 tablet_bootstrap.cc:492] T 45f7068bafaf48d290ded9d1272c0e46 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:33.561426 27693 tablet_bootstrap.cc:654] T 45f7068bafaf48d290ded9d1272c0e46 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:33.562542 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1007) took 1503 ms. Trace:
I20250903 20:05:33.562696 27687 rpcz_store.cc:276] 0903 20:05:32.058809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.058867 (+ 58us) service_pool.cc:225] Handling call
0903 20:05:33.562418 (+1503551us) ts_tablet_manager.cc:1145] Deleting tablet 2478fe89414b4845b0ba377c1cde98b7
0903 20:05:33.562427 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.562524 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1503360}
I20250903 20:05:33.562906 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 20c25c84c6fb4609a1964a4ff3c526da (table default.Table_6b435f3c61214c74a632c94e955abcdf [id=c2cf9ce651fc41a9b2b223b998e45d9b]) successfully deleted
I20250903 20:05:33.563349 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1008) took 1429 ms. Trace:
W20250903 20:05:33.563481 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet e64c4a9179b6491ba92df75e0cc1bc50 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e64c4a9179b6491ba92df75e0cc1bc50 already in progress: creating tablet
I20250903 20:05:33.563478 27687 rpcz_store.cc:276] 0903 20:05:32.133951 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.134040 (+ 89us) service_pool.cc:225] Handling call
0903 20:05:33.563241 (+1429201us) ts_tablet_manager.cc:1145] Deleting tablet 4e34d6d5d68e4395aed78821ea077c1b
0903 20:05:33.563250 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.563332 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1427530}
W20250903 20:05:33.563694 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 3f303233e6c347289650c6cf83f8127f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f303233e6c347289650c6cf83f8127f already in progress: creating tablet
I20250903 20:05:33.564109 27685 log.cc:1199] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/53881ad30bc843839008ace4012a3f70
I20250903 20:05:33.564554 27685 ts_tablet_manager.cc:1939] T 53881ad30bc843839008ace4012a3f70 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:33.564682 27687 tablet_replica.cc:331] T b0d79c78e0024b7ba3bb3356be7c8ceb P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:33.564885 27687 raft_consensus.cc:2241] T b0d79c78e0024b7ba3bb3356be7c8ceb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:33.565117 27687 raft_consensus.cc:2270] T b0d79c78e0024b7ba3bb3356be7c8ceb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.565711 27687 ts_tablet_manager.cc:1905] T b0d79c78e0024b7ba3bb3356be7c8ceb P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.566633 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 871) took 5136 ms. Trace:
I20250903 20:05:33.570679 27685 rpcz_store.cc:276] 0903 20:05:28.430170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.430231 (+ 61us) service_pool.cc:225] Handling call
0903 20:05:33.540192 (+5109961us) ts_tablet_manager.cc:1145] Deleting tablet 53881ad30bc843839008ace4012a3f70
0903 20:05:33.540202 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.546349 (+ 6147us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.559434 (+ 13085us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.566610 (+ 7176us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5109803}
I20250903 20:05:33.571527 27685 tablet_replica.cc:331] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:33.572018 27685 raft_consensus.cc:2241] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:33.572674 27685 raft_consensus.cc:2270] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.573710 27685 ts_tablet_manager.cc:1905] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.576602 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 59a1dd48d76241d7ba817070fbffd853 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:33.576828 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a223f75045345ae833f52bcc8f239f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:33.576886 27693 tablet_bootstrap.cc:492] T 45f7068bafaf48d290ded9d1272c0e46 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:33.577210 27693 ts_tablet_manager.cc:1397] T 45f7068bafaf48d290ded9d1272c0e46 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.001s
W20250903 20:05:33.577260 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet b5bff419ed31453e80ceec23ea05caad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5bff419ed31453e80ceec23ea05caad already in progress: creating tablet
I20250903 20:05:33.577736 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet c8a293c6fa034e9fb29682439cb3bcc2 (table default.Table_397211088a974a09bd7f031d5c7ca1cc [id=5cfc8827e4ca4f4a94424c48d6f71eb5]) successfully deleted
W20250903 20:05:33.578014 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 2478fe89414b4845b0ba377c1cde98b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2478fe89414b4845b0ba377c1cde98b7 already in progress: creating tablet
W20250903 20:05:33.578291 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 4e34d6d5d68e4395aed78821ea077c1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e34d6d5d68e4395aed78821ea077c1b already in progress: creating tablet
I20250903 20:05:33.578449 27693 raft_consensus.cc:357] T 45f7068bafaf48d290ded9d1272c0e46 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.578836 27693 raft_consensus.cc:383] T 45f7068bafaf48d290ded9d1272c0e46 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.578956 27693 raft_consensus.cc:738] T 45f7068bafaf48d290ded9d1272c0e46 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.579020 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 53881ad30bc843839008ace4012a3f70 (table default.Table_2b6469f261a04e5d9cb69bf836d97734 [id=cedbbfab7d0e41369cafd9ce516f2dba]) successfully deleted
I20250903 20:05:33.580368 27693 consensus_queue.cc:260] T 45f7068bafaf48d290ded9d1272c0e46 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.580962 27693 ts_tablet_manager.cc:1428] T 45f7068bafaf48d290ded9d1272c0e46 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:33.583029 27693 tablet_bootstrap.cc:492] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:33.584849 27693 tablet_bootstrap.cc:654] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:33.587620 27184 consensus_peers.cc:489] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:33.594861 27685 ts_tablet_manager.cc:1918] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.595986 27687 ts_tablet_manager.cc:1918] T b0d79c78e0024b7ba3bb3356be7c8ceb P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 20:05:33.596174 27687 log.cc:1199] T b0d79c78e0024b7ba3bb3356be7c8ceb P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/b0d79c78e0024b7ba3bb3356be7c8ceb
I20250903 20:05:33.596586 27687 ts_tablet_manager.cc:1939] T b0d79c78e0024b7ba3bb3356be7c8ceb P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:33.597304 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1012) took 1462 ms. Trace:
I20250903 20:05:33.597440 27687 rpcz_store.cc:276] 0903 20:05:32.134441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.134504 (+ 63us) service_pool.cc:225] Handling call
0903 20:05:33.564015 (+1429511us) ts_tablet_manager.cc:1145] Deleting tablet b0d79c78e0024b7ba3bb3356be7c8ceb
0903 20:05:33.564643 (+ 628us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.569762 (+ 5119us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.595897 (+ 26135us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.597287 (+ 1390us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1428020}
I20250903 20:05:33.598042 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1009) took 1463 ms. Trace:
I20250903 20:05:33.598397 27687 rpcz_store.cc:276] 0903 20:05:32.134096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.134512 (+ 416us) service_pool.cc:225] Handling call
0903 20:05:33.597917 (+1463405us) ts_tablet_manager.cc:1145] Deleting tablet 1ad2ca3e72434b69ad48e533da72e8fd
0903 20:05:33.597927 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.598025 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1461845}
I20250903 20:05:33.599059 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1010) took 1464 ms. Trace:
I20250903 20:05:33.599390 27687 rpcz_store.cc:276] 0903 20:05:32.134186 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.134281 (+ 95us) service_pool.cc:225] Handling call
0903 20:05:33.598967 (+1464686us) ts_tablet_manager.cc:1145] Deleting tablet 6d21f38759d743af972635f8b5b8d8d2
0903 20:05:33.598975 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.599045 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1460488}
I20250903 20:05:33.599684 27685 log.cc:1199] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/7e2f98f760cf4df4bc2a10530e2157c6
I20250903 20:05:33.600217 27685 ts_tablet_manager.cc:1939] T 7e2f98f760cf4df4bc2a10530e2157c6 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:33.600965 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1021) took 1307 ms. Trace:
I20250903 20:05:33.601087 27687 rpcz_store.cc:276] 0903 20:05:32.293489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.293556 (+ 67us) service_pool.cc:225] Handling call
0903 20:05:33.600549 (+1306993us) ts_tablet_manager.cc:1145] Deleting tablet f9709dfd18654c5bb071e39da7c8889f
0903 20:05:33.600558 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.600605 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1306778}
I20250903 20:05:33.601217 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet b0d79c78e0024b7ba3bb3356be7c8ceb (table default.Table_94e9e32ef1134f07ade1cdf8da69ead6 [id=e248f9c4b9c9406395aaea7ba3feb9ba]) successfully deleted
W20250903 20:05:33.601678 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 1ad2ca3e72434b69ad48e533da72e8fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ad2ca3e72434b69ad48e533da72e8fd already in progress: creating tablet
W20250903 20:05:33.601935 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 6d21f38759d743af972635f8b5b8d8d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d21f38759d743af972635f8b5b8d8d2 already in progress: creating tablet
W20250903 20:05:33.602283 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet f9709dfd18654c5bb071e39da7c8889f because the tablet was not found. No further retry: Not found: Tablet not found: f9709dfd18654c5bb071e39da7c8889f
I20250903 20:05:33.602762 27693 tablet_bootstrap.cc:492] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:33.602937 27693 ts_tablet_manager.cc:1397] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20250903 20:05:33.603725 27693 raft_consensus.cc:357] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.604048 27693 raft_consensus.cc:383] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.604202 27693 raft_consensus.cc:738] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.604451 27693 consensus_queue.cc:260] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.604950 27693 ts_tablet_manager.cc:1428] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:33.607864 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_60a3bd90021349e694df58967ede5cf5" } modify_external_catalogs: true
I20250903 20:05:33.607967 27693 tablet_bootstrap.cc:492] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:33.608122 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_60a3bd90021349e694df58967ede5cf5" } modify_external_catalogs: true
I20250903 20:05:33.609545 27687 tablet_replica.cc:331] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:33.610152 28710 raft_consensus.cc:491] T ed096b452ba747f89e91edc15e6e3ac1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:33.610358 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 872) took 5179 ms. Trace:
I20250903 20:05:33.610322 28710 raft_consensus.cc:513] T ed096b452ba747f89e91edc15e6e3ac1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.610448 27685 rpcz_store.cc:276] 0903 20:05:28.430769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:28.430816 (+ 47us) service_pool.cc:225] Handling call
0903 20:05:33.571469 (+5140653us) ts_tablet_manager.cc:1145] Deleting tablet 7e2f98f760cf4df4bc2a10530e2157c6
0903 20:05:33.571481 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.591818 (+ 20337us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.594765 (+ 2947us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.610334 (+ 15569us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5140447}
I20250903 20:05:33.611037 28710 leader_election.cc:290] T ed096b452ba747f89e91edc15e6e3ac1 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:33.612308 27687 raft_consensus.cc:2241] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:33.612445 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed096b452ba747f89e91edc15e6e3ac1" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:33.612677 27687 raft_consensus.cc:2270] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:33.613564 27462 leader_election.cc:343] T ed096b452ba747f89e91edc15e6e3ac1 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:33.616827 27693 tablet_bootstrap.cc:654] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:33.616881 27684 ts_tablet_manager.cc:1918] T a9f43781299e47a8aecc37a0b73d7a64 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 20:05:33.617862 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 880) took 4585 ms. Trace:
I20250903 20:05:33.619596 27685 rpcz_store.cc:276] 0903 20:05:29.032279 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:29.032346 (+ 67us) service_pool.cc:225] Handling call
0903 20:05:33.617714 (+4585368us) ts_tablet_manager.cc:1145] Deleting tablet 4739f10128d64e2c986257b1d9deb90f
0903 20:05:33.617726 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.617841 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4585105}
I20250903 20:05:33.617549 27687 ts_tablet_manager.cc:1905] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.620697 27684 log.cc:1199] T a9f43781299e47a8aecc37a0b73d7a64 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/a9f43781299e47a8aecc37a0b73d7a64
I20250903 20:05:33.621295 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 881) took 4588 ms. Trace:
I20250903 20:05:33.621531 27684 ts_tablet_manager.cc:1939] T a9f43781299e47a8aecc37a0b73d7a64 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:33.621796 27685 rpcz_store.cc:276] 0903 20:05:29.033304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:29.033353 (+ 49us) service_pool.cc:225] Handling call
0903 20:05:33.621161 (+4587808us) ts_tablet_manager.cc:1145] Deleting tablet b3bb55085af044ec8bb2368c845e7404
0903 20:05:33.621174 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.621280 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4587636}
I20250903 20:05:33.622537 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed096b452ba747f89e91edc15e6e3ac1" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:33.622952 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 882) took 4589 ms. Trace:
I20250903 20:05:33.623068 27685 rpcz_store.cc:276] 0903 20:05:29.033642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:29.033683 (+ 41us) service_pool.cc:225] Handling call
0903 20:05:33.622840 (+4589157us) ts_tablet_manager.cc:1145] Deleting tablet de20d18d7f264811a0631aa1144aef2c
0903 20:05:33.622849 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.622937 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4588994}
W20250903 20:05:33.623139 27462 leader_election.cc:343] T ed096b452ba747f89e91edc15e6e3ac1 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:33.623282 27462 leader_election.cc:304] T ed096b452ba747f89e91edc15e6e3ac1 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:33.623606 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet a9f43781299e47a8aecc37a0b73d7a64 (table default.Table_73a4ac2afbad4654ac623cd581a45eb2 [id=f70d7ea93e154e6199f6766316353862]) successfully deleted
I20250903 20:05:33.623683 28710 raft_consensus.cc:2747] T ed096b452ba747f89e91edc15e6e3ac1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:33.624121 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 883) took 4590 ms. Trace:
W20250903 20:05:33.625500 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 3c47c5fca0ee4349b2667494f45af5d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c47c5fca0ee4349b2667494f45af5d0 already in progress: creating tablet
I20250903 20:05:33.625569 27684 tablet_replica.cc:331] T 567630e2a93645e58f9bf05ffff87b32 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:33.625640 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 7e2f98f760cf4df4bc2a10530e2157c6 (table default.Table_3185c25ec3784288bd52ccf75830589d [id=4d1a27b6dfb74dd69ba27d0f1e767dac]) successfully deleted
I20250903 20:05:33.625752 27684 raft_consensus.cc:2241] T 567630e2a93645e58f9bf05ffff87b32 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:33.625691 27685 rpcz_store.cc:276] 0903 20:05:29.033953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:29.033990 (+ 37us) service_pool.cc:225] Handling call
0903 20:05:33.624024 (+4590034us) ts_tablet_manager.cc:1145] Deleting tablet 5962685d79044081a75532e19bfa2bce
0903 20:05:33.624033 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.624109 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4589904}
W20250903 20:05:33.625923 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 4739f10128d64e2c986257b1d9deb90f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4739f10128d64e2c986257b1d9deb90f already in progress: creating tablet
W20250903 20:05:33.626166 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet b3bb55085af044ec8bb2368c845e7404 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3bb55085af044ec8bb2368c845e7404 already in progress: creating tablet
W20250903 20:05:33.626297 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet b32c82c3d8c4420cb6e3a95129550076 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b32c82c3d8c4420cb6e3a95129550076 already in progress: creating tablet
W20250903 20:05:33.626435 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet de20d18d7f264811a0631aa1144aef2c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de20d18d7f264811a0631aa1144aef2c already in progress: creating tablet
I20250903 20:05:33.626471 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 886) took 4591 ms. Trace:
W20250903 20:05:33.626602 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 16549a6d9aa9432eb5cee19f69ce6957 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16549a6d9aa9432eb5cee19f69ce6957 already in progress: creating tablet
I20250903 20:05:33.626566 27685 rpcz_store.cc:276] 0903 20:05:29.034603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:29.034648 (+ 45us) service_pool.cc:225] Handling call
0903 20:05:33.626335 (+4591687us) ts_tablet_manager.cc:1145] Deleting tablet 86e33bd235c14042a578a4a1c70f11b7
0903 20:05:33.626346 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.626455 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4591455}
W20250903 20:05:33.626664 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 5962685d79044081a75532e19bfa2bce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5962685d79044081a75532e19bfa2bce already in progress: creating tablet
I20250903 20:05:33.627035 27684 raft_consensus.cc:2270] T 567630e2a93645e58f9bf05ffff87b32 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.627110 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 885) took 4592 ms. Trace:
I20250903 20:05:33.627254 27685 rpcz_store.cc:276] 0903 20:05:29.034497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:29.034745 (+ 248us) service_pool.cc:225] Handling call
0903 20:05:33.627011 (+4592266us) ts_tablet_manager.cc:1145] Deleting tablet 13496efe34944b7280e7a684f180e6e0
0903 20:05:33.627019 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.627095 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4592111}
I20250903 20:05:33.627751 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 887) took 4592 ms. Trace:
I20250903 20:05:33.627846 27684 ts_tablet_manager.cc:1905] T 567630e2a93645e58f9bf05ffff87b32 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.627965 27685 rpcz_store.cc:276] 0903 20:05:29.034988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:29.035578 (+ 590us) service_pool.cc:225] Handling call
0903 20:05:33.627662 (+4592084us) ts_tablet_manager.cc:1145] Deleting tablet 9766cea9d5bb407a9900c7e3bb4d2641
0903 20:05:33.627670 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.627737 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4591930}
W20250903 20:05:33.628409 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 38b912e24bf244759be8a6dc70cf9f52 consensus state has no leader
I20250903 20:05:33.628808 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 888) took 4505 ms. Trace:
I20250903 20:05:33.631721 27685 rpcz_store.cc:276] 0903 20:05:29.123175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:29.123636 (+ 461us) service_pool.cc:225] Handling call
0903 20:05:33.628716 (+4505080us) ts_tablet_manager.cc:1145] Deleting tablet bf459db3327640f88574fca7b5f2ca98
0903 20:05:33.628724 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.628795 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4504824}
I20250903 20:05:33.632339 27685 tablet_replica.cc:331] T 90baf83357494a0aae325908c118b471 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:33.632597 27685 raft_consensus.cc:2241] T 90baf83357494a0aae325908c118b471 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:33.632948 27685 raft_consensus.cc:2270] T 90baf83357494a0aae325908c118b471 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.633962 27685 ts_tablet_manager.cc:1905] T 90baf83357494a0aae325908c118b471 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:33.634232 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 86e33bd235c14042a578a4a1c70f11b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86e33bd235c14042a578a4a1c70f11b7 already in progress: creating tablet
W20250903 20:05:33.634819 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 13496efe34944b7280e7a684f180e6e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13496efe34944b7280e7a684f180e6e0 already in progress: creating tablet
W20250903 20:05:33.636294 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 9766cea9d5bb407a9900c7e3bb4d2641 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9766cea9d5bb407a9900c7e3bb4d2641 already in progress: creating tablet
W20250903 20:05:33.636601 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f1d22e76ea72441db51f7e9db26eb520 consensus state has no leader
I20250903 20:05:33.641587 27684 ts_tablet_manager.cc:1918] T 567630e2a93645e58f9bf05ffff87b32 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.641705 27684 log.cc:1199] T 567630e2a93645e58f9bf05ffff87b32 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/567630e2a93645e58f9bf05ffff87b32
I20250903 20:05:33.641947 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_53b5ec8ef7854180bc6c9d983cbf216d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:33.642750 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53b5ec8ef7854180bc6c9d983cbf216d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:33.643703 27684 ts_tablet_manager.cc:1939] T 567630e2a93645e58f9bf05ffff87b32 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
W20250903 20:05:33.637853 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d1e89ad94adb4de8b3738e30a84a8f79 consensus state has no leader
I20250903 20:05:33.639389 27062 master_service.cc:948] ReplaceTablet: received request to replace tablet 65d07ebe1ec94baabc6ce011e761b4d5 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:33.645054 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_8df248ec82924d66982d58ce170d32f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:33.645780 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8df248ec82924d66982d58ce170d32f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:33.647697 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 567630e2a93645e58f9bf05ffff87b32 (table default.Table_f2d0d1d8bddc492dbccd2c5c0a4e1a16 [id=3348ac5023a64fe3b19aed6c9471d580]) successfully deleted
W20250903 20:05:33.648828 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet bf459db3327640f88574fca7b5f2ca98 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf459db3327640f88574fca7b5f2ca98 already in progress: creating tablet
I20250903 20:05:33.649538 27695 tablet_bootstrap.cc:492] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:33.649682 27695 ts_tablet_manager.cc:1397] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.305s user 0.002s sys 0.001s
I20250903 20:05:33.650313 27684 tablet_replica.cc:331] T 8fff36d5a80e45af8a256c3908555e08 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:33.650548 27695 raft_consensus.cc:357] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.650722 27695 raft_consensus.cc:383] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.650775 27695 raft_consensus.cc:738] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.650956 27684 raft_consensus.cc:2241] T 8fff36d5a80e45af8a256c3908555e08 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:33.650997 27695 consensus_queue.cc:260] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.651507 27695 ts_tablet_manager.cc:1428] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:33.651775 27695 tablet_bootstrap.cc:492] T bf459db3327640f88574fca7b5f2ca98 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:33.651778 27693 tablet_bootstrap.cc:492] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:33.651952 27693 ts_tablet_manager.cc:1397] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.002s
I20250903 20:05:33.652861 27693 raft_consensus.cc:357] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.653043 27693 raft_consensus.cc:383] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.653092 27693 raft_consensus.cc:738] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.653463 27684 raft_consensus.cc:2270] T 8fff36d5a80e45af8a256c3908555e08 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.653507 27693 consensus_queue.cc:260] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.654134 27693 ts_tablet_manager.cc:1428] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:33.654512 27693 tablet_bootstrap.cc:492] T 548b50819f694fa3ad508da9f442a99c P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:33.655048 27684 ts_tablet_manager.cc:1905] T 8fff36d5a80e45af8a256c3908555e08 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.656749 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bf69315f07a470bb2b897ad7c590729 with delete_type TABLET_DATA_DELETED (replaced by tablet 17514a2af39b402abbe8263d7c1abc7f) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:33.657696 27687 ts_tablet_manager.cc:1918] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.657951 27687 log.cc:1199] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/120ca61b3da742dc9a2efa483ead7e97
I20250903 20:05:33.658496 27693 tablet_bootstrap.cc:654] T 548b50819f694fa3ad508da9f442a99c P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:33.658707 27687 ts_tablet_manager.cc:1939] T 120ca61b3da742dc9a2efa483ead7e97 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:33.661000 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1022) took 1367 ms. Trace:
I20250903 20:05:33.663348 27687 rpcz_store.cc:276] 0903 20:05:32.293642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.293739 (+ 97us) service_pool.cc:225] Handling call
0903 20:05:33.609490 (+1315751us) ts_tablet_manager.cc:1145] Deleting tablet 120ca61b3da742dc9a2efa483ead7e97
0903 20:05:33.609502 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.629045 (+ 19543us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.657569 (+ 28524us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.660979 (+ 3410us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1315562}
I20250903 20:05:33.664041 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 120ca61b3da742dc9a2efa483ead7e97 (table default.Table_47699f82f0074c7eac5b4b83296c71e5 [id=a10d0a5b11484d078240afb2ad212bff]) successfully deleted
I20250903 20:05:33.664518 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1023) took 1370 ms. Trace:
I20250903 20:05:33.664726 27684 ts_tablet_manager.cc:1918] T 8fff36d5a80e45af8a256c3908555e08 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.664678 27687 rpcz_store.cc:276] 0903 20:05:32.293799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.294260 (+ 461us) service_pool.cc:225] Handling call
0903 20:05:33.664403 (+1370143us) ts_tablet_manager.cc:1145] Deleting tablet 90baf83357494a0aae325908c118b471
0903 20:05:33.664414 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.664498 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1369988}
W20250903 20:05:33.666904 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 90baf83357494a0aae325908c118b471 because the tablet was not found. No further retry: Not found: Tablet not found: 90baf83357494a0aae325908c118b471
I20250903 20:05:33.667367 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1024) took 1373 ms. Trace:
I20250903 20:05:33.667528 27687 rpcz_store.cc:276] 0903 20:05:32.293899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.294444 (+ 545us) service_pool.cc:225] Handling call
0903 20:05:33.667246 (+1372802us) ts_tablet_manager.cc:1145] Deleting tablet f2629f46704e4a0fa7922533928ca2ed
0903 20:05:33.667258 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.667348 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1372672}
W20250903 20:05:33.668964 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet f2629f46704e4a0fa7922533928ca2ed because the tablet was not found. No further retry: Not found: Tablet not found: f2629f46704e4a0fa7922533928ca2ed
I20250903 20:05:33.669374 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1025) took 1375 ms. Trace:
I20250903 20:05:33.670501 27684 log.cc:1199] T 8fff36d5a80e45af8a256c3908555e08 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/8fff36d5a80e45af8a256c3908555e08
I20250903 20:05:33.670661 27687 rpcz_store.cc:276] 0903 20:05:32.294019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.294643 (+ 624us) service_pool.cc:225] Handling call
0903 20:05:33.669283 (+1374640us) ts_tablet_manager.cc:1145] Deleting tablet 0a9b7770432942da9373d5100c71bd6a
0903 20:05:33.669295 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.669357 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1374454}
I20250903 20:05:33.671110 27684 ts_tablet_manager.cc:1939] T 8fff36d5a80e45af8a256c3908555e08 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
W20250903 20:05:33.671284 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 0a9b7770432942da9373d5100c71bd6a because the tablet was not found. No further retry: Not found: Tablet not found: 0a9b7770432942da9373d5100c71bd6a
I20250903 20:05:33.671538 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1026) took 1375 ms. Trace:
I20250903 20:05:33.671658 27687 rpcz_store.cc:276] 0903 20:05:32.295530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.295848 (+ 318us) service_pool.cc:225] Handling call
0903 20:05:33.671043 (+1375195us) ts_tablet_manager.cc:1145] Deleting tablet 4af8c03c9d834405b4e25d35441db961
0903 20:05:33.671054 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.671121 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1375023}
W20250903 20:05:33.672152 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 4af8c03c9d834405b4e25d35441db961 because the tablet was not found. No further retry: Not found: Tablet not found: 4af8c03c9d834405b4e25d35441db961
I20250903 20:05:33.672415 27687 tablet_replica.cc:331] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:33.672669 27687 raft_consensus.cc:2241] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:33.672740 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 8fff36d5a80e45af8a256c3908555e08 (table default.Table_644507bb34724729854f122d8191cf82 [id=5616ab4744594f75b25cc4b7db7c3797]) successfully deleted
I20250903 20:05:33.673341 27685 ts_tablet_manager.cc:1918] T 90baf83357494a0aae325908c118b471 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.673753 27687 raft_consensus.cc:2270] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.673923 27684 tablet_replica.cc:331] T 120ca61b3da742dc9a2efa483ead7e97 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:33.674098 27685 log.cc:1199] T 90baf83357494a0aae325908c118b471 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/90baf83357494a0aae325908c118b471
I20250903 20:05:33.674119 27684 raft_consensus.cc:2241] T 120ca61b3da742dc9a2efa483ead7e97 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:33.674327 27684 raft_consensus.cc:2270] T 120ca61b3da742dc9a2efa483ead7e97 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.675163 27685 ts_tablet_manager.cc:1939] T 90baf83357494a0aae325908c118b471 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:33.675570 27684 ts_tablet_manager.cc:1905] T 120ca61b3da742dc9a2efa483ead7e97 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.676126 27687 ts_tablet_manager.cc:1905] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.682425 26885 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskH7eH49/build/asan/bin/kudu
/tmp/dist-test-taskH7eH49/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.65.125:44059
--webserver_interface=127.26.65.125
--webserver_port=41613
--builtin_ntp_servers=127.26.65.84:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.65.126:39015,127.26.65.125:44059,127.26.65.124:40911
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250903 20:05:33.682868 27695 tablet_bootstrap.cc:654] T bf459db3327640f88574fca7b5f2ca98 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:33.682917 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 898) took 4286 ms. Trace:
I20250903 20:05:33.682983 27685 rpcz_store.cc:276] 0903 20:05:29.396226 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:29.400500 (+ 4274us) service_pool.cc:225] Handling call
0903 20:05:33.632277 (+4231777us) ts_tablet_manager.cc:1145] Deleting tablet 90baf83357494a0aae325908c118b471
0903 20:05:33.632288 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.661338 (+ 29050us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.673213 (+ 11875us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.682895 (+ 9682us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4231575}
I20250903 20:05:33.683516 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 90baf83357494a0aae325908c118b471 (table default.Table_75f638a7c40e48e584a02949e6f864fd [id=7e9a08baa86e406c85192816b4a5d26d]) successfully deleted
I20250903 20:05:33.684504 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 896) took 4288 ms. Trace:
I20250903 20:05:33.684576 27685 rpcz_store.cc:276] 0903 20:05:29.395949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:29.402262 (+ 6313us) service_pool.cc:225] Handling call
0903 20:05:33.684376 (+4282114us) ts_tablet_manager.cc:1145] Deleting tablet 53b00774b745462091321ea47fcc2114
0903 20:05:33.684388 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.684488 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4281954}
W20250903 20:05:33.685019 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 53b00774b745462091321ea47fcc2114 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53b00774b745462091321ea47fcc2114 already in progress: creating tablet
I20250903 20:05:33.685388 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 897) took 4289 ms. Trace:
I20250903 20:05:33.686033 27693 tablet_bootstrap.cc:492] T 548b50819f694fa3ad508da9f442a99c P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:33.686156 27693 ts_tablet_manager.cc:1397] T 548b50819f694fa3ad508da9f442a99c P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.000s
I20250903 20:05:33.687289 27693 raft_consensus.cc:357] T 548b50819f694fa3ad508da9f442a99c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.687536 27693 raft_consensus.cc:383] T 548b50819f694fa3ad508da9f442a99c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.687623 27693 raft_consensus.cc:738] T 548b50819f694fa3ad508da9f442a99c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.687887 27693 consensus_queue.cc:260] T 548b50819f694fa3ad508da9f442a99c P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.688439 27693 ts_tablet_manager.cc:1428] T 548b50819f694fa3ad508da9f442a99c P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:33.685451 27685 rpcz_store.cc:276] 0903 20:05:29.396109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:29.402446 (+ 6337us) service_pool.cc:225] Handling call
0903 20:05:33.685296 (+4282850us) ts_tablet_manager.cc:1145] Deleting tablet 31186292b25f4c3cb23597f66dcbd282
0903 20:05:33.685304 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.685374 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4282741}
W20250903 20:05:33.689095 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 31186292b25f4c3cb23597f66dcbd282 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31186292b25f4c3cb23597f66dcbd282 already in progress: creating tablet
I20250903 20:05:33.689666 27693 tablet_bootstrap.cc:492] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:33.691720 27693 tablet_bootstrap.cc:654] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:33.692068 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 899) took 3443 ms. Trace:
I20250903 20:05:33.692215 27685 rpcz_store.cc:276] 0903 20:05:30.248720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.248942 (+ 222us) service_pool.cc:225] Handling call
0903 20:05:33.691908 (+3442966us) ts_tablet_manager.cc:1145] Deleting tablet cc1bc205bf39447bb9c3c7863a0959cf
0903 20:05:33.691920 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.692045 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3442770}
W20250903 20:05:33.692768 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet cc1bc205bf39447bb9c3c7863a0959cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc1bc205bf39447bb9c3c7863a0959cf already in progress: creating tablet
I20250903 20:05:33.693156 27685 tablet_replica.cc:331] T d1c96296a579440c887ba6c2a93f7204 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:33.693392 27685 raft_consensus.cc:2241] T d1c96296a579440c887ba6c2a93f7204 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:33.693620 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 3590506b3fd44a6c84aa469c14ed3e9f
I20250903 20:05:33.694459 27692 tablet_bootstrap.cc:492] T 950a6fcd35f64f2d889831fe9cdf503a P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:33.694597 27685 raft_consensus.cc:2270] T d1c96296a579440c887ba6c2a93f7204 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.694701 27692 ts_tablet_manager.cc:1397] T 950a6fcd35f64f2d889831fe9cdf503a P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.260s user 0.029s sys 0.026s
I20250903 20:05:33.695283 27685 ts_tablet_manager.cc:1905] T d1c96296a579440c887ba6c2a93f7204 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.695643 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet 3590506b3fd44a6c84aa469c14ed3e9f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:33 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:33.696842 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet 3590506b3fd44a6c84aa469c14ed3e9f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:33 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:33.697394 27692 raft_consensus.cc:357] T 950a6fcd35f64f2d889831fe9cdf503a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.697520 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c9acca1de7954498ab2dcc3ee1ff2f45" } modify_external_catalogs: true
I20250903 20:05:33.697608 27692 raft_consensus.cc:383] T 950a6fcd35f64f2d889831fe9cdf503a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.697678 27692 raft_consensus.cc:738] T 950a6fcd35f64f2d889831fe9cdf503a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.697736 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c9acca1de7954498ab2dcc3ee1ff2f45" } modify_external_catalogs: true
I20250903 20:05:33.697906 27692 consensus_queue.cc:260] T 950a6fcd35f64f2d889831fe9cdf503a P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.698670 27692 ts_tablet_manager.cc:1428] T 950a6fcd35f64f2d889831fe9cdf503a P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:33.698745 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet 3590506b3fd44a6c84aa469c14ed3e9f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:33 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:33.699018 27692 tablet_bootstrap.cc:492] T 1223743918814d88b401782ca4b521dd P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:33.699882 27695 tablet_bootstrap.cc:492] T bf459db3327640f88574fca7b5f2ca98 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:33.700073 27695 ts_tablet_manager.cc:1397] T bf459db3327640f88574fca7b5f2ca98 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.001s
I20250903 20:05:33.700882 27695 raft_consensus.cc:357] T bf459db3327640f88574fca7b5f2ca98 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.701462 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet d1a254d560904b9aa31b8ee7de65ffa5
I20250903 20:05:33.702610 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet dd27901224d6442389e8fc09c6b498ab
I20250903 20:05:33.701079 27695 raft_consensus.cc:383] T bf459db3327640f88574fca7b5f2ca98 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.702913 27695 raft_consensus.cc:738] T bf459db3327640f88574fca7b5f2ca98 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.705098 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet d1a254d560904b9aa31b8ee7de65ffa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:33 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:33.705860 27500 tablet_service.cc:1515] Processing DeleteTablet for tablet dd27901224d6442389e8fc09c6b498ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:33 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:33.707095 27347 tablet_service.cc:1515] Processing DeleteTablet for tablet dd27901224d6442389e8fc09c6b498ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:33 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:33.707322 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet d1a254d560904b9aa31b8ee7de65ffa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:33 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:33.707330 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet dd27901224d6442389e8fc09c6b498ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:33 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:33.708317 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet d1a254d560904b9aa31b8ee7de65ffa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:33 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:33.708305 27695 consensus_queue.cc:260] T bf459db3327640f88574fca7b5f2ca98 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.709183 27695 ts_tablet_manager.cc:1428] T bf459db3327640f88574fca7b5f2ca98 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250903 20:05:33.709662 27695 tablet_bootstrap.cc:492] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:33.713260 27692 tablet_bootstrap.cc:654] T 1223743918814d88b401782ca4b521dd P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:33.715368 27695 tablet_bootstrap.cc:654] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:33.715802 27687 ts_tablet_manager.cc:1918] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.715979 27687 log.cc:1199] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/8fff36d5a80e45af8a256c3908555e08
I20250903 20:05:33.717072 27687 ts_tablet_manager.cc:1939] T 8fff36d5a80e45af8a256c3908555e08 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:33.721551 27684 ts_tablet_manager.cc:1918] T 120ca61b3da742dc9a2efa483ead7e97 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.721719 27684 log.cc:1199] T 120ca61b3da742dc9a2efa483ead7e97 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/120ca61b3da742dc9a2efa483ead7e97
I20250903 20:05:33.722126 27684 ts_tablet_manager.cc:1939] T 120ca61b3da742dc9a2efa483ead7e97 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:33.722304 27064 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 67ca1dcf6fc7435db555392235f9900f (table default.Table_d3945d5a36d84041b35394333dc704f9 [id=5568f788d1414c83b4359b5adf28632a]). Expected version 2 got 0
I20250903 20:05:33.722599 27064 catalog_manager.cc:5582] T fc634116a78f45dd833034ddda90e9bf P 89f8f0ee68b940db9ce250c0604f934a reported cstate change: term changed from 0 to 1, leader changed from <none> to 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65). New cstate: current_term: 1 leader_uuid: "89f8f0ee68b940db9ce250c0604f934a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:33.723038 27064 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet fc634116a78f45dd833034ddda90e9bf (table default.Table_78e7a5b76ebc469396d8a3d670ab63ee [id=fb43441ca8f2415d99a628e81ef1642f]). Expected version 2 got 0
I20250903 20:05:33.723161 27684 tablet_replica.cc:331] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:33.723403 27684 raft_consensus.cc:2241] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:33.723524 27064 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 4bcd65022b7b4502983b6b58809047e3 (table default.Table_df7e7287722a45988929dfd3f0ff3eb6 [id=7ee5ea1a352941d7ba044bf48f21848a]). Expected version 2 got 0
I20250903 20:05:33.723760 27693 tablet_bootstrap.cc:492] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:33.723785 27684 raft_consensus.cc:2270] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.723951 27693 ts_tablet_manager.cc:1397] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20250903 20:05:33.725924 27692 tablet_bootstrap.cc:492] T 1223743918814d88b401782ca4b521dd P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:33.726076 27692 ts_tablet_manager.cc:1397] T 1223743918814d88b401782ca4b521dd P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20250903 20:05:33.726647 27693 raft_consensus.cc:357] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.726852 27693 raft_consensus.cc:383] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.726933 27693 raft_consensus.cc:738] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.727206 27693 consensus_queue.cc:260] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.727756 27693 ts_tablet_manager.cc:1428] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:33.728077 27693 tablet_bootstrap.cc:492] T aa9f5e0964b348b7af184f60cd145666 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:33.729040 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1027) took 1424 ms. Trace:
W20250903 20:05:33.729425 27183 consensus_peers.cc:489] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 726f4d920f3e47fea41b4e12c021c988. This is attempt 1: this message will repeat every 5th retry.
W20250903 20:05:33.729648 27183 consensus_peers.cc:489] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 567630e2a93645e58f9bf05ffff87b32. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:33.730377 27500 tablet_service.cc:1515] Processing DeleteTablet for tablet b7a6d5d8a97a4769a81ec7224d591f8c with delete_type TABLET_DATA_DELETED (replaced by tablet 79c0cc0c30ab42eab0be415e3db9589f) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:33.731277 27685 ts_tablet_manager.cc:1918] T d1c96296a579440c887ba6c2a93f7204 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:33.731761 27685 log.cc:1199] T d1c96296a579440c887ba6c2a93f7204 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/d1c96296a579440c887ba6c2a93f7204
I20250903 20:05:33.732271 27685 ts_tablet_manager.cc:1939] T d1c96296a579440c887ba6c2a93f7204 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:33.733384 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 901) took 3484 ms. Trace:
I20250903 20:05:33.734113 27685 rpcz_store.cc:276] 0903 20:05:30.248899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.249627 (+ 728us) service_pool.cc:225] Handling call
0903 20:05:33.693105 (+3443478us) ts_tablet_manager.cc:1145] Deleting tablet d1c96296a579440c887ba6c2a93f7204
0903 20:05:33.693115 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.712023 (+ 18908us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.731159 (+ 19136us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.733363 (+ 2204us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3443287}
I20250903 20:05:33.730098 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 120ca61b3da742dc9a2efa483ead7e97 (table default.Table_47699f82f0074c7eac5b4b83296c71e5 [id=a10d0a5b11484d078240afb2ad212bff]) successfully deleted
I20250903 20:05:33.735008 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet d1c96296a579440c887ba6c2a93f7204 (table default.Table_66babebee1c9462ea057c8489d43846d [id=a4d648e314504b68bc297abdbccad46f]) successfully deleted
I20250903 20:05:33.730832 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet df5f439f5aca4231af229159811b7a75
I20250903 20:05:33.735306 27692 raft_consensus.cc:357] T 1223743918814d88b401782ca4b521dd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.735531 27692 raft_consensus.cc:383] T 1223743918814d88b401782ca4b521dd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.735627 27692 raft_consensus.cc:738] T 1223743918814d88b401782ca4b521dd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.735872 27692 consensus_queue.cc:260] T 1223743918814d88b401782ca4b521dd P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.736460 27692 ts_tablet_manager.cc:1428] T 1223743918814d88b401782ca4b521dd P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20250903 20:05:33.736553 27500 tablet_service.cc:1515] Processing DeleteTablet for tablet df5f439f5aca4231af229159811b7a75 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:33 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:33.736585 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet df5f439f5aca4231af229159811b7a75 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:33 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:33.736377 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet df5f439f5aca4231af229159811b7a75 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:33 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:33.735491 27685 tablet_replica.cc:331] T 5f27f525fdab407280aa23df1ae71b43 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:33.737399 27685 raft_consensus.cc:2241] T 5f27f525fdab407280aa23df1ae71b43 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:33.737883 27685 raft_consensus.cc:2270] T 5f27f525fdab407280aa23df1ae71b43 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.738758 27685 ts_tablet_manager.cc:1905] T 5f27f525fdab407280aa23df1ae71b43 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.738790 27692 tablet_bootstrap.cc:492] T 548b50819f694fa3ad508da9f442a99c P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:33.739136 29108 raft_consensus.cc:491] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:33.739349 29108 raft_consensus.cc:513] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.740059 29108 leader_election.cc:290] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:33.740572 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 512908bc4d6941ec81b145ee5795405f with delete_type TABLET_DATA_DELETED (replaced by tablet 1598824a733a4a93bed043e81b86abf3) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:33.740749 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet a253e3399622409e92299069b19d3698 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:33.740931 27489 tablet_service.cc:1515] Processing DeleteTablet for tablet 120ca61b3da742dc9a2efa483ead7e97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:33.741019 27503 tablet_service.cc:1515] Processing DeleteTablet for tablet e877d56bbbfa4c5a92d025d0311ce5c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:33.741186 27488 tablet_service.cc:1515] Processing DeleteTablet for tablet 5002b7e8118c49b6a247cb8dc4ca1aea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:33.741367 27496 tablet_service.cc:1515] Processing DeleteTablet for tablet 200c06f296b94eab94f35f312c5a443b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:33.741456 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet a0f79831866b40c49c5b27ce2014c3f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:33.741609 27498 tablet_service.cc:1515] Processing DeleteTablet for tablet 45f7068bafaf48d290ded9d1272c0e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:33.741691 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e877d56bbbfa4c5a92d025d0311ce5c6" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:33.741838 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 567630e2a93645e58f9bf05ffff87b32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:33.741933 27511 raft_consensus.cc:2466] T e877d56bbbfa4c5a92d025d0311ce5c6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:33.742041 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet 22bc87d1a4804b6c80f77f9692500d63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:33.742267 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c18a98d5c0f4a43ae91c9130047b681 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:33.742571 27692 tablet_bootstrap.cc:654] T 548b50819f694fa3ad508da9f442a99c P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:33.742528 27320 leader_election.cc:304] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:33.742668 27500 tablet_service.cc:1515] Processing DeleteTablet for tablet 541278ca9ffb4e41b341139c2faa337f with delete_type TABLET_DATA_DELETED (replaced by tablet e9126e4a18c546c28927f22c9f16426e) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:33.742893 27684 ts_tablet_manager.cc:1905] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.742960 29108 raft_consensus.cc:2802] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:33.743067 29108 raft_consensus.cc:491] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:33.743152 29108 raft_consensus.cc:3058] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:33.747073 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_923868d510f34889be990c9cd4180adc" } new_table_name: "default.Table_63d8bb9ad5a14a7d81fb6f15d20e6ff0" modify_external_catalogs: true
I20250903 20:05:33.748638 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_d5ec146fd09a42cca06f2e0665bc3441"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:33.749197 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5ec146fd09a42cca06f2e0665bc3441 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:33.746058 29108 raft_consensus.cc:513] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.754352 29108 leader_election.cc:290] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:33.751143 27320 consensus_peers.cc:489] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 541278ca9ffb4e41b341139c2faa337f. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:33.754637 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_fd55f02a3038460fb305e900be86ef28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:33.755146 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e877d56bbbfa4c5a92d025d0311ce5c6" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
W20250903 20:05:33.755280 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd55f02a3038460fb305e900be86ef28 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:33.755322 27511 raft_consensus.cc:3058] T e877d56bbbfa4c5a92d025d0311ce5c6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:33.755743 27684 ts_tablet_manager.cc:1918] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.755827 27684 log.cc:1199] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/02b3e262dd7841dfbb0520377eb94743
I20250903 20:05:33.756170 27684 ts_tablet_manager.cc:1939] T 02b3e262dd7841dfbb0520377eb94743 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:33.757934 27511 raft_consensus.cc:2466] T e877d56bbbfa4c5a92d025d0311ce5c6 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:33.758462 27320 leader_election.cc:304] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:33.758867 29108 raft_consensus.cc:2802] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:33.759260 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 02b3e262dd7841dfbb0520377eb94743 (table default.Table_3f1f99f4ab084617b3a72447d6e84209 [id=a30b474b4c8b494db74d62e576e509a7]) successfully deleted
W20250903 20:05:33.759567 27051 catalog_manager.cc:4895] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 4af8c03c9d834405b4e25d35441db961 because the tablet was not found. No further retry: Not found: Tablet not found: 4af8c03c9d834405b4e25d35441db961
W20250903 20:05:33.759775 27051 catalog_manager.cc:4895] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 607c9a9cbc5540e5bd0fa436fcba564c because the tablet was not found. No further retry: Not found: Tablet not found: 607c9a9cbc5540e5bd0fa436fcba564c
W20250903 20:05:33.759980 27051 catalog_manager.cc:4895] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 63a1f26fac3b4b939ebb79c6a07d8f8e because the tablet was not found. No further retry: Not found: Tablet not found: 63a1f26fac3b4b939ebb79c6a07d8f8e
W20250903 20:05:33.760166 27051 catalog_manager.cc:4895] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 5f27f525fdab407280aa23df1ae71b43 because the tablet was not found. No further retry: Not found: Tablet not found: 5f27f525fdab407280aa23df1ae71b43
I20250903 20:05:33.760231 29108 raft_consensus.cc:695] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
W20250903 20:05:33.760397 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 3590506b3fd44a6c84aa469c14ed3e9f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3590506b3fd44a6c84aa469c14ed3e9f already in progress: creating tablet
W20250903 20:05:33.760581 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet d1a254d560904b9aa31b8ee7de65ffa5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1a254d560904b9aa31b8ee7de65ffa5 already in progress: creating tablet
I20250903 20:05:33.760533 29108 consensus_queue.cc:237] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:33.760771 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet dd27901224d6442389e8fc09c6b498ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dd27901224d6442389e8fc09c6b498ab already in progress: creating tablet
W20250903 20:05:33.760949 27051 catalog_manager.cc:4895] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet b7a6d5d8a97a4769a81ec7224d591f8c because the tablet was not found. No further retry: Not found: Tablet not found: b7a6d5d8a97a4769a81ec7224d591f8c
W20250903 20:05:33.761157 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet df5f439f5aca4231af229159811b7a75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df5f439f5aca4231af229159811b7a75 already in progress: creating tablet
I20250903 20:05:33.758169 27684 tablet_replica.cc:331] T a253e3399622409e92299069b19d3698 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:33.761677 27684 raft_consensus.cc:2241] T a253e3399622409e92299069b19d3698 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
W20250903 20:05:33.762023 27051 catalog_manager.cc:4895] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 512908bc4d6941ec81b145ee5795405f because the tablet was not found. No further retry: Not found: Tablet not found: 512908bc4d6941ec81b145ee5795405f
I20250903 20:05:33.762423 27684 raft_consensus.cc:2270] T a253e3399622409e92299069b19d3698 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.763003 27684 ts_tablet_manager.cc:1905] T a253e3399622409e92299069b19d3698 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.763633 28710 consensus_queue.cc:1035] T 7345305820be4d32a3ceb227f045f009 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:33.768247 28710 consensus_queue.cc:1035] T 7345305820be4d32a3ceb227f045f009 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:33.769305 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b99ab7e3978a411cb629b6a3d0f4ccd9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:33.770107 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b99ab7e3978a411cb629b6a3d0f4ccd9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:33.775724 27693 tablet_bootstrap.cc:654] T aa9f5e0964b348b7af184f60cd145666 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:33.778671 27687 rpcz_store.cc:276] 0903 20:05:32.295695 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.296058 (+ 363us) service_pool.cc:225] Handling call
0903 20:05:33.671971 (+1375913us) ts_tablet_manager.cc:1145] Deleting tablet 8fff36d5a80e45af8a256c3908555e08
0903 20:05:33.671982 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.681032 (+ 9050us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.704962 (+ 23930us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.719775 (+ 14813us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1375778}
I20250903 20:05:33.787967 27687 tablet_replica.cc:331] T 567630e2a93645e58f9bf05ffff87b32 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:33.787907 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 8fff36d5a80e45af8a256c3908555e08 (table default.Table_644507bb34724729854f122d8191cf82 [id=5616ab4744594f75b25cc4b7db7c3797]) successfully deleted
I20250903 20:05:33.741209 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e877d56bbbfa4c5a92d025d0311ce5c6" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:33.788264 27687 raft_consensus.cc:2241] T 567630e2a93645e58f9bf05ffff87b32 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:33.754751 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e877d56bbbfa4c5a92d025d0311ce5c6" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:33.788456 27687 raft_consensus.cc:2270] T 567630e2a93645e58f9bf05ffff87b32 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.788460 27240 raft_consensus.cc:2466] T e877d56bbbfa4c5a92d025d0311ce5c6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:33.788992 27687 ts_tablet_manager.cc:1905] T 567630e2a93645e58f9bf05ffff87b32 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.769654 27695 tablet_bootstrap.cc:492] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:33.789469 27695 ts_tablet_manager.cc:1397] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.080s user 0.005s sys 0.000s
I20250903 20:05:33.790323 27695 raft_consensus.cc:357] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.790562 27695 raft_consensus.cc:383] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.790652 27695 raft_consensus.cc:738] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.790913 27695 consensus_queue.cc:260] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.793920 27695 ts_tablet_manager.cc:1428] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:33.795864 27695 tablet_bootstrap.cc:492] T 7a9e55f0811345f7afa71645f38ebdaf P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:33.797751 27695 tablet_bootstrap.cc:654] T 7a9e55f0811345f7afa71645f38ebdaf P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:33.802670 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ddf4574624894772bd7c0aa8c5e177cf" } new_table_name: "default.Table_734675d85e3b44a1b5880e64e6494525" modify_external_catalogs: true
I20250903 20:05:33.806324 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 65d07ebe1ec94baabc6ce011e761b4d5
I20250903 20:05:33.806794 27062 catalog_manager.cc:6652] ReplaceTablet: tablet 65d07ebe1ec94baabc6ce011e761b4d5 deleted and replaced by tablet 515db6402ced455ea4bb9a0d2e89382f
I20250903 20:05:33.807096 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_cedba29d5f42445cba418e98e554bb61"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:33.807539 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 65d07ebe1ec94baabc6ce011e761b4d5 with delete_type TABLET_DATA_DELETED (replaced by tablet 515db6402ced455ea4bb9a0d2e89382f) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:33.807610 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cedba29d5f42445cba418e98e554bb61 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:33.807998 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet 65d07ebe1ec94baabc6ce011e761b4d5 with delete_type TABLET_DATA_DELETED (replaced by tablet 515db6402ced455ea4bb9a0d2e89382f) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:33.811277 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet e35ae50252734a099ba5ce2d04e49e1f
I20250903 20:05:33.811924 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_16c1f5a369af4b17a52df8480211496c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:33.812465 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_16c1f5a369af4b17a52df8480211496c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:33.812664 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet e35ae50252734a099ba5ce2d04e49e1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:33 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:33.812896 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet e35ae50252734a099ba5ce2d04e49e1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:33 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:33.813084 29108 raft_consensus.cc:491] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:33.813194 29108 raft_consensus.cc:513] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.813766 29108 leader_election.cc:290] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:33.815781 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf459db3327640f88574fca7b5f2ca98" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:33.811157 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_4bcfc1c4f42c4c179852756650e5a925"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:33.815987 27248 raft_consensus.cc:2466] T bf459db3327640f88574fca7b5f2ca98 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:33.816321 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf459db3327640f88574fca7b5f2ca98" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:33.816444 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4bcfc1c4f42c4c179852756650e5a925 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:33.816504 27511 raft_consensus.cc:2466] T bf459db3327640f88574fca7b5f2ca98 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:33.816642 27500 tablet_service.cc:1515] Processing DeleteTablet for tablet 65d07ebe1ec94baabc6ce011e761b4d5 with delete_type TABLET_DATA_DELETED (replaced by tablet 515db6402ced455ea4bb9a0d2e89382f) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:33.816891 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet e35ae50252734a099ba5ce2d04e49e1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:33 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:33.817014 27321 leader_election.cc:304] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:33.817627 27684 ts_tablet_manager.cc:1918] T a253e3399622409e92299069b19d3698 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:33.817759 29108 raft_consensus.cc:2802] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:33.817847 29108 raft_consensus.cc:491] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:33.817968 29108 raft_consensus.cc:3058] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:33.821028 29108 raft_consensus.cc:513] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.823812 27684 log.cc:1199] T a253e3399622409e92299069b19d3698 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/a253e3399622409e92299069b19d3698
I20250903 20:05:33.824190 27684 ts_tablet_manager.cc:1939] T a253e3399622409e92299069b19d3698 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:33.823702 27692 tablet_bootstrap.cc:492] T 548b50819f694fa3ad508da9f442a99c P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:33.826436 27692 ts_tablet_manager.cc:1397] T 548b50819f694fa3ad508da9f442a99c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.088s user 0.004s sys 0.013s
I20250903 20:05:33.821990 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf459db3327640f88574fca7b5f2ca98" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:33.826927 27511 raft_consensus.cc:3058] T bf459db3327640f88574fca7b5f2ca98 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:33.827438 27692 raft_consensus.cc:357] T 548b50819f694fa3ad508da9f442a99c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.827662 27692 raft_consensus.cc:383] T 548b50819f694fa3ad508da9f442a99c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.827737 27692 raft_consensus.cc:738] T 548b50819f694fa3ad508da9f442a99c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.827935 27692 consensus_queue.cc:260] T 548b50819f694fa3ad508da9f442a99c P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.828411 27692 ts_tablet_manager.cc:1428] T 548b50819f694fa3ad508da9f442a99c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:33.828806 27692 tablet_bootstrap.cc:492] T 7a9e55f0811345f7afa71645f38ebdaf P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:33.829655 27511 raft_consensus.cc:2466] T bf459db3327640f88574fca7b5f2ca98 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:33.830745 27692 tablet_bootstrap.cc:654] T 7a9e55f0811345f7afa71645f38ebdaf P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:33.831106 27320 leader_election.cc:304] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:33.831458 28567 raft_consensus.cc:2802] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:33.831609 28567 raft_consensus.cc:695] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:33.831887 28567 consensus_queue.cc:237] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.835371 27684 tablet_replica.cc:331] T e877d56bbbfa4c5a92d025d0311ce5c6 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:33.835789 27684 raft_consensus.cc:2241] T e877d56bbbfa4c5a92d025d0311ce5c6 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:33.835999 27684 raft_consensus.cc:2270] T e877d56bbbfa4c5a92d025d0311ce5c6 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.836782 27684 ts_tablet_manager.cc:1905] T e877d56bbbfa4c5a92d025d0311ce5c6 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.837134 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf459db3327640f88574fca7b5f2ca98" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:33.837385 27248 raft_consensus.cc:3058] T bf459db3327640f88574fca7b5f2ca98 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:33.839087 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet a253e3399622409e92299069b19d3698 (table default.Table_8bbb5561118c4a68af214f8d319d2caf [id=2218f11e07d243adaa3a4d24e521b1b7]) successfully deleted
I20250903 20:05:33.839179 27695 tablet_bootstrap.cc:492] T 7a9e55f0811345f7afa71645f38ebdaf P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:33.839314 27695 ts_tablet_manager.cc:1397] T 7a9e55f0811345f7afa71645f38ebdaf P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.001s
W20250903 20:05:33.839529 27051 catalog_manager.cc:4895] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 120ca61b3da742dc9a2efa483ead7e97 because the tablet was not found. No further retry: Not found: Tablet not found: 120ca61b3da742dc9a2efa483ead7e97
I20250903 20:05:33.840245 27248 raft_consensus.cc:2466] T bf459db3327640f88574fca7b5f2ca98 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
W20250903 20:05:33.844099 27184 consensus_peers.cc:489] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1bf69315f07a470bb2b897ad7c590729. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:33.844825 28874 consensus_queue.cc:1035] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:33.845042 29108 leader_election.cc:290] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:33.847640 27321 consensus_peers.cc:489] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Couldn't send request to peer 89f8f0ee68b940db9ce250c0604f934a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1c96296a579440c887ba6c2a93f7204. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:33.848786 27693 tablet_bootstrap.cc:492] T aa9f5e0964b348b7af184f60cd145666 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:33.848932 27693 ts_tablet_manager.cc:1397] T aa9f5e0964b348b7af184f60cd145666 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.121s user 0.004s sys 0.000s
I20250903 20:05:33.849843 27693 raft_consensus.cc:357] T aa9f5e0964b348b7af184f60cd145666 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.850037 27693 raft_consensus.cc:383] T aa9f5e0964b348b7af184f60cd145666 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.850106 27693 raft_consensus.cc:738] T aa9f5e0964b348b7af184f60cd145666 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.850348 27693 consensus_queue.cc:260] T aa9f5e0964b348b7af184f60cd145666 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.850646 27695 raft_consensus.cc:357] T 7a9e55f0811345f7afa71645f38ebdaf P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:33.850813 27695 raft_consensus.cc:383] T 7a9e55f0811345f7afa71645f38ebdaf P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.850864 27693 ts_tablet_manager.cc:1428] T aa9f5e0964b348b7af184f60cd145666 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:33.850864 27695 raft_consensus.cc:738] T 7a9e55f0811345f7afa71645f38ebdaf P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.852098 27695 consensus_queue.cc:260] T 7a9e55f0811345f7afa71645f38ebdaf P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:33.853796 27695 ts_tablet_manager.cc:1428] T 7a9e55f0811345f7afa71645f38ebdaf P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250903 20:05:33.855019 27695 tablet_bootstrap.cc:492] T a38f65e992884635aae9942681c0dc4f P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:33.856863 27693 tablet_bootstrap.cc:492] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:33.861385 27685 ts_tablet_manager.cc:1918] T 5f27f525fdab407280aa23df1ae71b43 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.868292 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_92312eea209d4177bae77b8fa25fb7a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:33.868598 27183 consensus_peers.cc:489] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a253e3399622409e92299069b19d3698. This is attempt 1: this message will repeat every 5th retry.
W20250903 20:05:33.868963 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_92312eea209d4177bae77b8fa25fb7a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:33.868677 27685 log.cc:1199] T 5f27f525fdab407280aa23df1ae71b43 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/5f27f525fdab407280aa23df1ae71b43
I20250903 20:05:33.870543 27685 ts_tablet_manager.cc:1939] T 5f27f525fdab407280aa23df1ae71b43 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:33.870580 27061 master_service.cc:948] ReplaceTablet: received request to replace tablet ef02f91c0964446daf62ac1dde0330ff from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:33.871089 27061 master_service.cc:948] ReplaceTablet: received request to replace tablet 5fb2426aa01e47598f268d0a90475271 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:33.871433 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 907) took 3621 ms. Trace:
I20250903 20:05:33.871471 27061 master_service.cc:948] ReplaceTablet: received request to replace tablet cc1bc205bf39447bb9c3c7863a0959cf from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:33.871517 27685 rpcz_store.cc:276] 0903 20:05:30.249507 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.252720 (+ 3213us) service_pool.cc:225] Handling call
0903 20:05:33.735436 (+3482716us) ts_tablet_manager.cc:1145] Deleting tablet 5f27f525fdab407280aa23df1ae71b43
0903 20:05:33.735448 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.808340 (+ 72892us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.861211 (+ 52871us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.871412 (+ 10201us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3482543}
I20250903 20:05:33.871807 27061 master_service.cc:948] ReplaceTablet: received request to replace tablet f1d22e76ea72441db51f7e9db26eb520 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:33.862464 27695 tablet_bootstrap.cc:654] T a38f65e992884635aae9942681c0dc4f P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:33.872208 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 5f27f525fdab407280aa23df1ae71b43 (table default.Table_1c679659094d4666aaaf68f76f667890 [id=61a25739b03042f7bd5bb693b68656d5]) successfully deleted
I20250903 20:05:33.872745 29108 raft_consensus.cc:491] T 4bcd65022b7b4502983b6b58809047e3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:33.872912 29108 raft_consensus.cc:513] T 4bcd65022b7b4502983b6b58809047e3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.873749 29108 leader_election.cc:290] T 4bcd65022b7b4502983b6b58809047e3 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:33.874549 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4bcd65022b7b4502983b6b58809047e3" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:33.874786 27248 raft_consensus.cc:2466] T 4bcd65022b7b4502983b6b58809047e3 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:33.875315 27321 leader_election.cc:304] T 4bcd65022b7b4502983b6b58809047e3 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:33.876094 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4bcd65022b7b4502983b6b58809047e3" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:33.876298 27511 raft_consensus.cc:2466] T 4bcd65022b7b4502983b6b58809047e3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:33.875486 27687 ts_tablet_manager.cc:1918] T 567630e2a93645e58f9bf05ffff87b32 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.876049 29108 raft_consensus.cc:2802] T 4bcd65022b7b4502983b6b58809047e3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:33.882393 29108 raft_consensus.cc:491] T 4bcd65022b7b4502983b6b58809047e3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:33.882499 29108 raft_consensus.cc:3058] T 4bcd65022b7b4502983b6b58809047e3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:33.884908 29108 raft_consensus.cc:513] T 4bcd65022b7b4502983b6b58809047e3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.886581 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4bcd65022b7b4502983b6b58809047e3" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:33.886813 27248 raft_consensus.cc:3058] T 4bcd65022b7b4502983b6b58809047e3 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:33.887832 27685 tablet_replica.cc:331] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:33.888006 27685 raft_consensus.cc:2241] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:33.888307 27685 raft_consensus.cc:2270] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.881614 27684 ts_tablet_manager.cc:1918] T e877d56bbbfa4c5a92d025d0311ce5c6 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:33.887929 27692 tablet_bootstrap.cc:492] T 7a9e55f0811345f7afa71645f38ebdaf P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:33.888844 27684 log.cc:1199] T e877d56bbbfa4c5a92d025d0311ce5c6 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/e877d56bbbfa4c5a92d025d0311ce5c6
I20250903 20:05:33.885947 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4bcd65022b7b4502983b6b58809047e3" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:33.888938 27692 ts_tablet_manager.cc:1397] T 7a9e55f0811345f7afa71645f38ebdaf P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.060s user 0.005s sys 0.001s
I20250903 20:05:33.888947 27511 raft_consensus.cc:3058] T 4bcd65022b7b4502983b6b58809047e3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:33.889544 27248 raft_consensus.cc:2466] T 4bcd65022b7b4502983b6b58809047e3 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:33.889786 27692 raft_consensus.cc:357] T 7a9e55f0811345f7afa71645f38ebdaf P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:33.889935 27692 raft_consensus.cc:383] T 7a9e55f0811345f7afa71645f38ebdaf P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.889981 27692 raft_consensus.cc:738] T 7a9e55f0811345f7afa71645f38ebdaf P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.890018 27321 leader_election.cc:304] T 4bcd65022b7b4502983b6b58809047e3 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:33.890333 28567 raft_consensus.cc:2802] T 4bcd65022b7b4502983b6b58809047e3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:33.890437 28567 raft_consensus.cc:695] T 4bcd65022b7b4502983b6b58809047e3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:33.890260 27692 consensus_queue.cc:260] T 7a9e55f0811345f7afa71645f38ebdaf P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:33.890657 28567 consensus_queue.cc:237] T 4bcd65022b7b4502983b6b58809047e3 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.891062 27692 ts_tablet_manager.cc:1428] T 7a9e55f0811345f7afa71645f38ebdaf P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:33.892838 29108 leader_election.cc:290] T 4bcd65022b7b4502983b6b58809047e3 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:33.893162 27684 ts_tablet_manager.cc:1939] T e877d56bbbfa4c5a92d025d0311ce5c6 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:33.893860 27511 raft_consensus.cc:2466] T 4bcd65022b7b4502983b6b58809047e3 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:33.894589 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet e877d56bbbfa4c5a92d025d0311ce5c6 (table default.Table_d9105c35c28247f1a71866f2096d93f2 [id=33b22e921daa4ccf9406155bac438d08]) successfully deleted
I20250903 20:05:33.894979 27684 tablet_replica.cc:331] T 5002b7e8118c49b6a247cb8dc4ca1aea P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:33.897404 27684 raft_consensus.cc:2241] T 5002b7e8118c49b6a247cb8dc4ca1aea P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:33.897673 27684 raft_consensus.cc:2270] T 5002b7e8118c49b6a247cb8dc4ca1aea P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.898358 27684 ts_tablet_manager.cc:1905] T 5002b7e8118c49b6a247cb8dc4ca1aea P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.898548 27687 log.cc:1199] T 567630e2a93645e58f9bf05ffff87b32 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/567630e2a93645e58f9bf05ffff87b32
I20250903 20:05:33.898985 27687 ts_tablet_manager.cc:1939] T 567630e2a93645e58f9bf05ffff87b32 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:33.899822 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1028) took 1604 ms. Trace:
I20250903 20:05:33.900197 27692 tablet_bootstrap.cc:492] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:33.900487 27693 tablet_bootstrap.cc:654] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:33.901587 27685 ts_tablet_manager.cc:1905] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.901937 27692 tablet_bootstrap.cc:654] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:33.899940 27687 rpcz_store.cc:276] 0903 20:05:32.295810 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.296215 (+ 405us) service_pool.cc:225] Handling call
0903 20:05:33.787908 (+1491693us) ts_tablet_manager.cc:1145] Deleting tablet 567630e2a93645e58f9bf05ffff87b32
0903 20:05:33.787919 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.792078 (+ 4159us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.875353 (+ 83275us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.899799 (+ 24446us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1491564}
I20250903 20:05:33.913353 28808 consensus_queue.cc:1035] T fc634116a78f45dd833034ddda90e9bf P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:33.914278 28710 raft_consensus.cc:491] T 1223743918814d88b401782ca4b521dd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:33.914446 28710 raft_consensus.cc:513] T 1223743918814d88b401782ca4b521dd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.915171 28710 leader_election.cc:290] T 1223743918814d88b401782ca4b521dd P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:33.915763 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1223743918814d88b401782ca4b521dd" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:33.916476 27462 leader_election.cc:343] T 1223743918814d88b401782ca4b521dd P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:33.918864 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_120fea21398e41afaa4f552bbd492f38" } new_table_name: "default.Table_804156147a704fa781400755e58b999b" modify_external_catalogs: true
I20250903 20:05:33.915671 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1223743918814d88b401782ca4b521dd" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:33.916692 27320 consensus_peers.cc:489] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1c96296a579440c887ba6c2a93f7204. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:33.922365 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 567630e2a93645e58f9bf05ffff87b32 (table default.Table_f2d0d1d8bddc492dbccd2c5c0a4e1a16 [id=3348ac5023a64fe3b19aed6c9471d580]) successfully deleted
W20250903 20:05:33.922892 27462 leader_election.cc:343] T 1223743918814d88b401782ca4b521dd P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:33.923034 27462 leader_election.cc:304] T 1223743918814d88b401782ca4b521dd P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:33.923298 28710 raft_consensus.cc:2747] T 1223743918814d88b401782ca4b521dd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:33.924261 27687 tablet_replica.cc:331] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:33.924634 27687 raft_consensus.cc:2241] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:33.925127 27687 raft_consensus.cc:2270] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.929679 27684 ts_tablet_manager.cc:1918] T 5002b7e8118c49b6a247cb8dc4ca1aea P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:33.929903 27687 ts_tablet_manager.cc:1905] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.930285 27684 log.cc:1199] T 5002b7e8118c49b6a247cb8dc4ca1aea P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/5002b7e8118c49b6a247cb8dc4ca1aea
I20250903 20:05:33.930732 27684 ts_tablet_manager.cc:1939] T 5002b7e8118c49b6a247cb8dc4ca1aea P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:33.931886 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 5002b7e8118c49b6a247cb8dc4ca1aea (table default.Table_b20633c3b14e463aac0b7bd917c3ab9b [id=c6a148f02792496e8d8f2bb2dfc5e69b]) successfully deleted
W20250903 20:05:33.932085 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e12222227f7f4000b4601d80888e1b89 consensus state has no leader
I20250903 20:05:33.932296 27684 tablet_replica.cc:331] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:33.932469 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d40b242552784e45a5de2341a25174fc" } new_table_name: "default.Table_c72975c3981d4bb5b21256544717b9ed" modify_external_catalogs: true
I20250903 20:05:33.932492 27684 raft_consensus.cc:2241] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:33.935688 27684 pending_rounds.cc:70] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f: Trying to abort 1 pending ops.
I20250903 20:05:33.935740 27684 pending_rounds.cc:77] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196385006775152640 op_type: NO_OP noop_request { }
I20250903 20:05:33.935820 27684 raft_consensus.cc:2887] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:33.935906 27684 raft_consensus.cc:2270] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:33.935889 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d6bdbaa18dba41bc997a529a1d2d322d" } new_table_name: "default.Table_32aedf31a0514e6f90ec9dc9f0831e13" modify_external_catalogs: true
I20250903 20:05:33.936080 28874 consensus_queue.cc:1035] T fc634116a78f45dd833034ddda90e9bf P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:33.938421 27684 ts_tablet_manager.cc:1905] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:33.944904 27693 tablet_bootstrap.cc:492] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:33.945186 27693 ts_tablet_manager.cc:1397] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.088s user 0.004s sys 0.000s
I20250903 20:05:33.946096 27693 raft_consensus.cc:357] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.946424 27693 raft_consensus.cc:383] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.946542 27693 raft_consensus.cc:738] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.946857 27693 consensus_queue.cc:260] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.947489 27693 ts_tablet_manager.cc:1428] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:33.947863 27693 tablet_bootstrap.cc:492] T 886b8783aa9d4618aad498d4a1115843 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
W20250903 20:05:33.951234 27053 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet f1d22e76ea72441db51f7e9db26eb520 (table default.Table_258f358903dd4f6fa7e310d72562a570 [id=ae5e27c159a9462482d09a79e148f795], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f1d22e76ea72441db51f7e9db26eb520 consensus state has no leader
I20250903 20:05:33.953200 27693 tablet_bootstrap.cc:654] T 886b8783aa9d4618aad498d4a1115843 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:33.954232 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_6225fbd3b8554129a7d622f5bd325fc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:33.954841 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6225fbd3b8554129a7d622f5bd325fc9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:33.955206 27692 tablet_bootstrap.cc:492] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:33.955391 27692 ts_tablet_manager.cc:1397] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.003s
I20250903 20:05:33.956158 27692 raft_consensus.cc:357] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:33.956341 27692 raft_consensus.cc:383] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:33.956394 27692 raft_consensus.cc:738] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:33.956729 27695 tablet_bootstrap.cc:492] T a38f65e992884635aae9942681c0dc4f P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:33.956772 27692 consensus_queue.cc:260] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:33.956883 27695 ts_tablet_manager.cc:1397] T a38f65e992884635aae9942681c0dc4f P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.102s user 0.001s sys 0.004s
I20250903 20:05:33.957911 27692 ts_tablet_manager.cc:1428] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:33.958191 27692 tablet_bootstrap.cc:492] T 238f2f30377b42c2852b016018aa3245 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:33.958457 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_77f24f93527744b5b55d7c2243ef8774" } modify_external_catalogs: true
I20250903 20:05:33.958628 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_77f24f93527744b5b55d7c2243ef8774" } modify_external_catalogs: true
I20250903 20:05:33.959949 27692 tablet_bootstrap.cc:654] T 238f2f30377b42c2852b016018aa3245 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:33.963753 27684 ts_tablet_manager.cc:1918] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.964424 28808 raft_consensus.cc:491] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:33.964567 28808 raft_consensus.cc:513] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.965178 28808 leader_election.cc:290] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:33.965829 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c18a98d5c0f4a43ae91c9130047b681" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:33.966097 27379 raft_consensus.cc:2466] T 4c18a98d5c0f4a43ae91c9130047b681 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:33.966732 27184 leader_election.cc:304] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:33.967087 28808 raft_consensus.cc:2802] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:33.967177 28808 raft_consensus.cc:491] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:33.967244 28808 raft_consensus.cc:3058] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:33.969913 28808 raft_consensus.cc:513] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:33.971470 28808 leader_election.cc:290] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:33.971855 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c18a98d5c0f4a43ae91c9130047b681" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:33.972090 27379 raft_consensus.cc:3058] T 4c18a98d5c0f4a43ae91c9130047b681 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:33.974900 27379 raft_consensus.cc:2466] T 4c18a98d5c0f4a43ae91c9130047b681 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:33.975628 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_765e9646c7d54d3c8be7a5dd6703f47c" } modify_external_catalogs: true
I20250903 20:05:33.975807 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_765e9646c7d54d3c8be7a5dd6703f47c" } modify_external_catalogs: true
I20250903 20:05:33.976557 27184 leader_election.cc:304] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:33.976920 28947 raft_consensus.cc:2802] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:33.977052 28947 raft_consensus.cc:695] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:33.977286 28947 consensus_queue.cc:237] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:33.978757 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fb9e90d6e97d4ab4b27bd5efb2ee4b55 consensus state has no leader
I20250903 20:05:33.979104 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_63d8bb9ad5a14a7d81fb6f15d20e6ff0" } modify_external_catalogs: true
I20250903 20:05:33.979303 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_63d8bb9ad5a14a7d81fb6f15d20e6ff0" } modify_external_catalogs: true
I20250903 20:05:33.992625 27687 ts_tablet_manager.cc:1918] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:33.965581 27517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c18a98d5c0f4a43ae91c9130047b681" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:33.998409 27517 raft_consensus.cc:2466] T 4c18a98d5c0f4a43ae91c9130047b681 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
W20250903 20:05:33.998586 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fab456384e7242779690eb19bf41afac consensus state has no leader
I20250903 20:05:33.971073 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c18a98d5c0f4a43ae91c9130047b681" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:33.998911 27516 raft_consensus.cc:3058] T 4c18a98d5c0f4a43ae91c9130047b681 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:33.998997 27065 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_672b60689df64b1d8c9ebd27468d68d4" } modify_external_catalogs: true
I20250903 20:05:33.999167 27065 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_672b60689df64b1d8c9ebd27468d68d4" } modify_external_catalogs: true
I20250903 20:05:34.001905 27516 raft_consensus.cc:2466] T 4c18a98d5c0f4a43ae91c9130047b681 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:34.001950 27695 raft_consensus.cc:357] T a38f65e992884635aae9942681c0dc4f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:34.002277 27695 raft_consensus.cc:383] T a38f65e992884635aae9942681c0dc4f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:34.002344 27695 raft_consensus.cc:738] T a38f65e992884635aae9942681c0dc4f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:34.002635 27695 consensus_queue.cc:260] T a38f65e992884635aae9942681c0dc4f P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:34.003348 27684 log.cc:1199] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/200c06f296b94eab94f35f312c5a443b
I20250903 20:05:34.003764 27695 ts_tablet_manager.cc:1428] T a38f65e992884635aae9942681c0dc4f P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.047s user 0.000s sys 0.003s
W20250903 20:05:34.004094 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e7fd7cbf5a3048558720c7e258be308c consensus state has no leader
I20250903 20:05:34.004159 27695 tablet_bootstrap.cc:492] T 886b8783aa9d4618aad498d4a1115843 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:34.006508 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_3a10826c979f434fa1b1a6a3dbae518b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:34.006669 27684 ts_tablet_manager.cc:1939] T 200c06f296b94eab94f35f312c5a443b P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
W20250903 20:05:34.007172 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a10826c979f434fa1b1a6a3dbae518b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:34.008600 27684 tablet_replica.cc:331] T 45f7068bafaf48d290ded9d1272c0e46 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:34.008715 27692 tablet_bootstrap.cc:492] T 238f2f30377b42c2852b016018aa3245 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:34.008718 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 200c06f296b94eab94f35f312c5a443b (table default.Table_c5abd2620f3f45628ce2531cbcb81d9d [id=37960c5e67734caa951c9a151b0ece51]) successfully deleted
I20250903 20:05:34.008841 27692 ts_tablet_manager.cc:1397] T 238f2f30377b42c2852b016018aa3245 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.051s user 0.001s sys 0.017s
I20250903 20:05:34.008903 27684 raft_consensus.cc:2241] T 45f7068bafaf48d290ded9d1272c0e46 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
W20250903 20:05:34.009018 27051 catalog_manager.cc:4895] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet a0f79831866b40c49c5b27ce2014c3f6 because the tablet was not found. No further retry: Not found: Tablet not found: a0f79831866b40c49c5b27ce2014c3f6
I20250903 20:05:34.009073 27684 raft_consensus.cc:2270] T 45f7068bafaf48d290ded9d1272c0e46 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:34.009541 27684 ts_tablet_manager.cc:1905] T 45f7068bafaf48d290ded9d1272c0e46 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:34.010617 27693 tablet_bootstrap.cc:492] T 886b8783aa9d4618aad498d4a1115843 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:34.011361 27693 ts_tablet_manager.cc:1397] T 886b8783aa9d4618aad498d4a1115843 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.064s user 0.006s sys 0.000s
I20250903 20:05:34.013146 27693 raft_consensus.cc:357] T 886b8783aa9d4618aad498d4a1115843 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:34.013399 27693 raft_consensus.cc:383] T 886b8783aa9d4618aad498d4a1115843 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:34.013485 27693 raft_consensus.cc:738] T 886b8783aa9d4618aad498d4a1115843 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:34.013762 27693 consensus_queue.cc:260] T 886b8783aa9d4618aad498d4a1115843 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:34.014338 27693 ts_tablet_manager.cc:1428] T 886b8783aa9d4618aad498d4a1115843 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:34.016074 27693 tablet_bootstrap.cc:492] T 7a9e55f0811345f7afa71645f38ebdaf P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:34.017630 27693 tablet_bootstrap.cc:654] T 7a9e55f0811345f7afa71645f38ebdaf P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:34.009661 27692 raft_consensus.cc:357] T 238f2f30377b42c2852b016018aa3245 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.030846 27692 raft_consensus.cc:383] T 238f2f30377b42c2852b016018aa3245 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:34.031038 27692 raft_consensus.cc:738] T 238f2f30377b42c2852b016018aa3245 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:34.031488 27692 consensus_queue.cc:260] T 238f2f30377b42c2852b016018aa3245 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.033017 27059 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 67ca1dcf6fc7435db555392235f9900f (table default.Table_d3945d5a36d84041b35394333dc704f9 [id=5568f788d1414c83b4359b5adf28632a]). Expected version 2 got 0
I20250903 20:05:34.033521 27059 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 7345305820be4d32a3ceb227f045f009 (table default.Table_5f0199a660844c15a254a062c1540498 [id=52f969e50b9d466cbd14de6f252b7706]). Expected version 1 got 0
I20250903 20:05:34.033716 27692 ts_tablet_manager.cc:1428] T 238f2f30377b42c2852b016018aa3245 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
I20250903 20:05:34.034076 27692 tablet_bootstrap.cc:492] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:34.035943 27692 tablet_bootstrap.cc:654] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:34.040686 27057 catalog_manager.cc:5582] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66). New cstate: current_term: 1 leader_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:34.041188 27057 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 4bcd65022b7b4502983b6b58809047e3 (table default.Table_df7e7287722a45988929dfd3f0ff3eb6 [id=7ee5ea1a352941d7ba044bf48f21848a]). Expected version 2 got 0
I20250903 20:05:34.041366 27057 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 7345305820be4d32a3ceb227f045f009 (table default.Table_5f0199a660844c15a254a062c1540498 [id=52f969e50b9d466cbd14de6f252b7706]). Expected version 1 got 0
I20250903 20:05:34.041504 27057 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 67ca1dcf6fc7435db555392235f9900f (table default.Table_d3945d5a36d84041b35394333dc704f9 [id=5568f788d1414c83b4359b5adf28632a]). Expected version 2 got 0
I20250903 20:05:34.041782 27057 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet fc634116a78f45dd833034ddda90e9bf (table default.Table_78e7a5b76ebc469396d8a3d670ab63ee [id=fb43441ca8f2415d99a628e81ef1642f]). Expected version 2 got 0
I20250903 20:05:34.042363 27064 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_1ce23fe2aa6d4aceb99b7f02a2e6d04b [id=4e05a1e86a4d4319b51f2ff36f3db76b] alter complete (version 1)
I20250903 20:05:34.011137 27687 log.cc:1199] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/541278ca9ffb4e41b341139c2faa337f
I20250903 20:05:34.037194 28567 raft_consensus.cc:491] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:34.042716 28567 raft_consensus.cc:513] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.043351 28567 leader_election.cc:290] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:34.043959 27687 ts_tablet_manager.cc:1939] T 541278ca9ffb4e41b341139c2faa337f P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:34.043991 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed096b452ba747f89e91edc15e6e3ac1" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:34.044179 27516 raft_consensus.cc:2466] T ed096b452ba747f89e91edc15e6e3ac1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:34.044476 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed096b452ba747f89e91edc15e6e3ac1" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:34.044456 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_0a09cc2674c449fda58086db253699a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:34.044750 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1029) took 1748 ms. Trace:
I20250903 20:05:34.044828 27687 rpcz_store.cc:276] 0903 20:05:32.296477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.296520 (+ 43us) service_pool.cc:225] Handling call
0903 20:05:33.924183 (+1627663us) ts_tablet_manager.cc:1145] Deleting tablet 541278ca9ffb4e41b341139c2faa337f
0903 20:05:33.924200 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.967922 (+ 43722us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:33.992490 (+ 24568us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:34.044730 (+ 52240us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1627524}
W20250903 20:05:34.045003 27321 leader_election.cc:343] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:34.045136 27687 tablet_replica.cc:331] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
W20250903 20:05:34.045143 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a09cc2674c449fda58086db253699a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:34.045336 27320 leader_election.cc:304] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:34.045825 28567 raft_consensus.cc:2802] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:34.045886 28567 raft_consensus.cc:491] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:34.045939 28567 raft_consensus.cc:3058] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:34.048488 28567 raft_consensus.cc:513] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.049242 28567 leader_election.cc:290] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:34.050148 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed096b452ba747f89e91edc15e6e3ac1" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:34.050376 27516 raft_consensus.cc:3058] T ed096b452ba747f89e91edc15e6e3ac1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:34.053040 27516 raft_consensus.cc:2466] T ed096b452ba747f89e91edc15e6e3ac1 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:34.045338 27687 raft_consensus.cc:2241] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:34.054793 27687 raft_consensus.cc:2270] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:34.055349 27687 ts_tablet_manager.cc:1905] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:34.056161 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 2d9c1642fec841d497ff3496dd4564db
I20250903 20:05:34.058595 27320 leader_election.cc:304] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:34.058893 28567 raft_consensus.cc:2802] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:34.059340 28567 raft_consensus.cc:695] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:34.059672 28567 consensus_queue.cc:237] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.061219 27360 tablet_service.cc:1468] Processing CreateTablet for tablet c0a9434dc2cb444c9337aa5e2b39e86a (DEFAULT_TABLE table=default.Table_ba9ea3be44c64888a491bcc40e505a32 [id=756be0d2a564487ca6e2a1c2f5005ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.061394 27347 tablet_service.cc:1468] Processing CreateTablet for tablet 217fbfedfbf0417fb210fddc56a46066 (DEFAULT_TABLE table=default.Table_334d7a39ba4b4c2babe0c815dec6a6c8 [id=48fd4ec5e92b4680a3263508257f0fc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.061617 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0a9434dc2cb444c9337aa5e2b39e86a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.061805 27350 tablet_service.cc:1468] Processing CreateTablet for tablet 28baeae269224536a88e4cf064bab40f (DEFAULT_TABLE table=default.Table_ac73bfe5b3b342f6ad66cd0c724af31a [id=df9bd1ec985845f7803bf7a7d0261cf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.062258 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28baeae269224536a88e4cf064bab40f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.064915 27352 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d9c1642fec841d497ff3496dd4564db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:33 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.065147 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 217fbfedfbf0417fb210fddc56a46066. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.079272 27693 tablet_bootstrap.cc:492] T 7a9e55f0811345f7afa71645f38ebdaf P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:34.079476 27693 ts_tablet_manager.cc:1397] T 7a9e55f0811345f7afa71645f38ebdaf P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.064s user 0.004s sys 0.000s
I20250903 20:05:34.080399 27693 raft_consensus.cc:357] T 7a9e55f0811345f7afa71645f38ebdaf P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:34.080590 27693 raft_consensus.cc:383] T 7a9e55f0811345f7afa71645f38ebdaf P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:34.080642 27693 raft_consensus.cc:738] T 7a9e55f0811345f7afa71645f38ebdaf P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:34.080871 27693 consensus_queue.cc:260] T 7a9e55f0811345f7afa71645f38ebdaf P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:34.058111 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed096b452ba747f89e91edc15e6e3ac1" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:34.071336 27217 tablet_service.cc:1468] Processing CreateTablet for tablet c0a9434dc2cb444c9337aa5e2b39e86a (DEFAULT_TABLE table=default.Table_ba9ea3be44c64888a491bcc40e505a32 [id=756be0d2a564487ca6e2a1c2f5005ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.071959 27227 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d9c1642fec841d497ff3496dd4564db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:33 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:34.072131 27222 tablet_service.cc:1468] Processing CreateTablet for tablet 217fbfedfbf0417fb210fddc56a46066 (DEFAULT_TABLE table=default.Table_334d7a39ba4b4c2babe0c815dec6a6c8 [id=48fd4ec5e92b4680a3263508257f0fc4]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:34.083324 27321 leader_election.cc:343] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:34.083441 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 217fbfedfbf0417fb210fddc56a46066. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.083703 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0a9434dc2cb444c9337aa5e2b39e86a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.072337 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 28baeae269224536a88e4cf064bab40f (DEFAULT_TABLE table=default.Table_ac73bfe5b3b342f6ad66cd0c724af31a [id=df9bd1ec985845f7803bf7a7d0261cf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.084116 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28baeae269224536a88e4cf064bab40f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.072412 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet e877d56bbbfa4c5a92d025d0311ce5c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:34.072464 27216 tablet_service.cc:1515] Processing DeleteTablet for tablet 200c06f296b94eab94f35f312c5a443b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:34.072513 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e2f98f760cf4df4bc2a10530e2157c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:13 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:34.072566 27212 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c18a98d5c0f4a43ae91c9130047b681 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:34.072628 27221 tablet_service.cc:1515] Processing DeleteTablet for tablet 53881ad30bc843839008ace4012a3f70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:34.088563 27692 tablet_bootstrap.cc:492] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:34.089577 27692 ts_tablet_manager.cc:1397] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.056s user 0.007s sys 0.003s
I20250903 20:05:34.074724 29198 tablet.cc:1722] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:34.090467 27692 raft_consensus.cc:357] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:34.090673 27692 raft_consensus.cc:383] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:34.090739 27692 raft_consensus.cc:738] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:34.091009 27692 consensus_queue.cc:260] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:34.091708 27493 tablet_service.cc:1468] Processing CreateTablet for tablet c0a9434dc2cb444c9337aa5e2b39e86a (DEFAULT_TABLE table=default.Table_ba9ea3be44c64888a491bcc40e505a32 [id=756be0d2a564487ca6e2a1c2f5005ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.092094 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0a9434dc2cb444c9337aa5e2b39e86a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.080761 27695 tablet_bootstrap.cc:654] T 886b8783aa9d4618aad498d4a1115843 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:34.094516 27500 tablet_service.cc:1468] Processing CreateTablet for tablet 217fbfedfbf0417fb210fddc56a46066 (DEFAULT_TABLE table=default.Table_334d7a39ba4b4c2babe0c815dec6a6c8 [id=48fd4ec5e92b4680a3263508257f0fc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.057590 27685 ts_tablet_manager.cc:1918] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:34.094779 27685 log.cc:1199] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/f2629f46704e4a0fa7922533928ca2ed
I20250903 20:05:34.094794 27693 ts_tablet_manager.cc:1428] T 7a9e55f0811345f7afa71645f38ebdaf P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20250903 20:05:34.094923 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 217fbfedfbf0417fb210fddc56a46066. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.095177 27685 ts_tablet_manager.cc:1939] T f2629f46704e4a0fa7922533928ca2ed P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:34.095508 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 905) took 3846 ms. Trace:
I20250903 20:05:34.095783 27685 rpcz_store.cc:276] 0903 20:05:30.249307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.254710 (+ 5403us) service_pool.cc:225] Handling call
0903 20:05:33.887784 (+3633074us) ts_tablet_manager.cc:1145] Deleting tablet f2629f46704e4a0fa7922533928ca2ed
0903 20:05:33.887794 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:33.969878 (+ 82084us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:34.057469 (+ 87591us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:34.095493 (+ 38024us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3632932}
I20250903 20:05:34.096148 27685 tablet_replica.cc:331] T f9709dfd18654c5bb071e39da7c8889f P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:34.096422 27685 raft_consensus.cc:2241] T f9709dfd18654c5bb071e39da7c8889f P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:34.096602 27685 raft_consensus.cc:2270] T f9709dfd18654c5bb071e39da7c8889f P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:34.097160 27685 ts_tablet_manager.cc:1905] T f9709dfd18654c5bb071e39da7c8889f P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:34.098500 27693 tablet_bootstrap.cc:492] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:34.098618 27497 tablet_service.cc:1468] Processing CreateTablet for tablet 28baeae269224536a88e4cf064bab40f (DEFAULT_TABLE table=default.Table_ac73bfe5b3b342f6ad66cd0c724af31a [id=df9bd1ec985845f7803bf7a7d0261cf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.099057 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28baeae269224536a88e4cf064bab40f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.099349 27692 ts_tablet_manager.cc:1428] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20250903 20:05:34.099376 27684 ts_tablet_manager.cc:1918] T 45f7068bafaf48d290ded9d1272c0e46 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:34.099460 27684 log.cc:1199] T 45f7068bafaf48d290ded9d1272c0e46 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/45f7068bafaf48d290ded9d1272c0e46
I20250903 20:05:34.099860 27684 ts_tablet_manager.cc:1939] T 45f7068bafaf48d290ded9d1272c0e46 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:34.100411 27692 tablet_bootstrap.cc:492] T 24eadc70e10d4d02bcb643d45c9e0e36 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:34.102077 27692 tablet_bootstrap.cc:654] T 24eadc70e10d4d02bcb643d45c9e0e36 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:34.102386 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d9c1642fec841d497ff3496dd4564db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:33 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:34.103427 27687 ts_tablet_manager.cc:1918] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:34.103547 27687 log.cc:1199] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/63a1f26fac3b4b939ebb79c6a07d8f8e
I20250903 20:05:34.103893 27687 ts_tablet_manager.cc:1939] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:34.103988 27695 tablet_bootstrap.cc:492] T 886b8783aa9d4618aad498d4a1115843 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:34.104115 27695 ts_tablet_manager.cc:1397] T 886b8783aa9d4618aad498d4a1115843 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.100s user 0.001s sys 0.002s
I20250903 20:05:34.104319 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 541278ca9ffb4e41b341139c2faa337f (table default.Table_01215977307c453e9236e5bd72bc8427 [id=7ae71054c4d34d66a210e7c8f7460d4b]) successfully deleted
I20250903 20:05:34.104640 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1030) took 1807 ms. Trace:
I20250903 20:05:34.104852 27695 raft_consensus.cc:357] T 886b8783aa9d4618aad498d4a1115843 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:34.105057 27695 raft_consensus.cc:383] T 886b8783aa9d4618aad498d4a1115843 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:34.105125 27695 raft_consensus.cc:738] T 886b8783aa9d4618aad498d4a1115843 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:34.107631 27695 consensus_queue.cc:260] T 886b8783aa9d4618aad498d4a1115843 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:34.108863 28947 raft_consensus.cc:491] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:34.109048 28947 raft_consensus.cc:513] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.110343 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet f2629f46704e4a0fa7922533928ca2ed (table default.Table_ae457e422e0c4314aeaecd7755413366 [id=2051ecb140f24088911aad60fdb9082b]) successfully deleted
I20250903 20:05:34.111055 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "548b50819f694fa3ad508da9f442a99c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:34.111258 27516 raft_consensus.cc:2466] T 548b50819f694fa3ad508da9f442a99c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:34.109704 28947 leader_election.cc:290] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:34.111827 27183 leader_election.cc:304] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:34.112196 28947 raft_consensus.cc:2802] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:34.112433 28947 raft_consensus.cc:491] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:34.112581 28947 raft_consensus.cc:3058] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:34.116092 28947 raft_consensus.cc:513] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.117286 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "548b50819f694fa3ad508da9f442a99c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:34.117489 27516 raft_consensus.cc:3058] T 548b50819f694fa3ad508da9f442a99c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:34.118507 27684 tablet_replica.cc:331] T 4c18a98d5c0f4a43ae91c9130047b681 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:34.118765 27684 raft_consensus.cc:2241] T 4c18a98d5c0f4a43ae91c9130047b681 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:34.118929 27684 raft_consensus.cc:2270] T 4c18a98d5c0f4a43ae91c9130047b681 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:34.119527 27684 ts_tablet_manager.cc:1905] T 4c18a98d5c0f4a43ae91c9130047b681 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:34.120333 27516 raft_consensus.cc:2466] T 548b50819f694fa3ad508da9f442a99c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:34.120963 27183 leader_election.cc:304] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:34.121354 29196 raft_consensus.cc:2802] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:34.121484 29196 raft_consensus.cc:695] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:34.121747 29196 consensus_queue.cc:237] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.130532 27687 rpcz_store.cc:276] 0903 20:05:32.296797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.296841 (+ 44us) service_pool.cc:225] Handling call
0903 20:05:34.045091 (+1748250us) ts_tablet_manager.cc:1145] Deleting tablet 63a1f26fac3b4b939ebb79c6a07d8f8e
0903 20:05:34.045100 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.084780 (+ 39680us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:34.103321 (+ 18541us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:34.104619 (+ 1298us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1748101}
I20250903 20:05:34.114936 27693 tablet_bootstrap.cc:654] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:34.135142 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 63a1f26fac3b4b939ebb79c6a07d8f8e (table default.Table_362ac8cb413e426ebf154eae4b6b5c7e [id=10ef25498f7448dbb19cd66d4c6ba713]) successfully deleted
I20250903 20:05:34.136248 27693 tablet_bootstrap.cc:492] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:34.136435 27693 ts_tablet_manager.cc:1397] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20250903 20:05:34.137298 27693 raft_consensus.cc:357] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.137501 27693 raft_consensus.cc:383] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:34.137579 27693 raft_consensus.cc:738] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:34.137897 27687 tablet_replica.cc:331] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:34.138085 27687 raft_consensus.cc:2241] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:34.138923 27687 raft_consensus.cc:2270] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:34.139757 27687 ts_tablet_manager.cc:1905] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:34.141563 29196 raft_consensus.cc:491] T 7a9e55f0811345f7afa71645f38ebdaf P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:34.141737 29196 raft_consensus.cc:513] T 7a9e55f0811345f7afa71645f38ebdaf P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:34.144831 29196 leader_election.cc:290] T 7a9e55f0811345f7afa71645f38ebdaf P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:34.145466 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a9e55f0811345f7afa71645f38ebdaf" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:34.145641 27516 raft_consensus.cc:2466] T 7a9e55f0811345f7afa71645f38ebdaf P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:34.146083 27183 leader_election.cc:304] T 7a9e55f0811345f7afa71645f38ebdaf P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:34.142947 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a9e55f0811345f7afa71645f38ebdaf" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:34.146512 27383 raft_consensus.cc:2466] T 7a9e55f0811345f7afa71645f38ebdaf P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:34.147137 29196 raft_consensus.cc:2802] T 7a9e55f0811345f7afa71645f38ebdaf P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:34.147279 29196 raft_consensus.cc:491] T 7a9e55f0811345f7afa71645f38ebdaf P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:34.147405 29196 raft_consensus.cc:3058] T 7a9e55f0811345f7afa71645f38ebdaf P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:34.148312 28567 raft_consensus.cc:491] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:34.148495 28567 raft_consensus.cc:513] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.149360 28567 leader_election.cc:290] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:34.149827 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "950a6fcd35f64f2d889831fe9cdf503a" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:34.150321 27693 consensus_queue.cc:260] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:34.150417 27321 leader_election.cc:343] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:34.150466 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "950a6fcd35f64f2d889831fe9cdf503a" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:34.150627 27516 raft_consensus.cc:2466] T 950a6fcd35f64f2d889831fe9cdf503a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:34.150848 27693 ts_tablet_manager.cc:1428] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250903 20:05:34.151003 27320 leader_election.cc:304] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:34.152077 29108 raft_consensus.cc:2802] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:34.152156 29108 raft_consensus.cc:491] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:34.152233 29108 raft_consensus.cc:3058] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:34.153998 27360 tablet_service.cc:1468] Processing CreateTablet for tablet ffc1cf1b5d59485bad240bceb0d37804 (DEFAULT_TABLE table=default.Table_b1d886984546451495e327fac36f5e49 [id=86455fa9f50241b8a5b14836cbe5a867]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.154404 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffc1cf1b5d59485bad240bceb0d37804. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.154716 29108 raft_consensus.cc:513] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.155802 29108 leader_election.cc:290] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:34.156345 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "950a6fcd35f64f2d889831fe9cdf503a" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:34.156510 27516 raft_consensus.cc:3058] T 950a6fcd35f64f2d889831fe9cdf503a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:34.157028 27693 tablet_bootstrap.cc:492] T a38f65e992884635aae9942681c0dc4f P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:34.158433 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "548b50819f694fa3ad508da9f442a99c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:34.158614 27381 raft_consensus.cc:2466] T 548b50819f694fa3ad508da9f442a99c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:34.159154 27516 raft_consensus.cc:2466] T 950a6fcd35f64f2d889831fe9cdf503a P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:34.159595 27320 leader_election.cc:304] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:34.160131 28567 raft_consensus.cc:2802] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:34.160287 28567 raft_consensus.cc:695] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:34.160545 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "950a6fcd35f64f2d889831fe9cdf503a" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:34.160949 28567 consensus_queue.cc:237] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.164672 27684 ts_tablet_manager.cc:1918] T 4c18a98d5c0f4a43ae91c9130047b681 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250903 20:05:34.165711 27321 leader_election.cc:343] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:34.168174 27352 tablet_service.cc:1468] Processing CreateTablet for tablet 2d4df4cc00c9433bb120566a797be986 (DEFAULT_TABLE table=default.Table_8851a4dad8da4bb8907db1ff956dd28f [id=27dbd6d3bb504a82a84b666698c93299]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.168359 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 88f74467a168473ab81d6fad2bddacab (DEFAULT_TABLE table=default.Table_e027444cbd5545a38f6e4f35e1ed90c3 [id=e36fab9d943649d9bbc994d3d490c545]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.168485 27359 tablet_service.cc:1468] Processing CreateTablet for tablet 6d8673f26fe748c7aa5c15f0112730b2 (DEFAULT_TABLE table=default.Table_23a41a80bee1454e983b6dff5b8e1c8b [id=998235969e704094952d213816af798b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.168633 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d4df4cc00c9433bb120566a797be986. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.168825 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88f74467a168473ab81d6fad2bddacab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.168908 27349 tablet_service.cc:1468] Processing CreateTablet for tablet 1e3ee607233b43539ecbcff732d641d4 (DEFAULT_TABLE table=default.Table_473b25f4188347ffba83a46006e66cf1 [id=244396a4b04a4a8ea9728362d526a609]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.169092 27357 tablet_service.cc:1468] Processing CreateTablet for tablet 0aab16e37f6e433aa4834dd6606f78a1 (DEFAULT_TABLE table=default.Table_762b9caaaec44868b1816efca7d186cf [id=d061ae74dd2c4db0af3f19caa59a0fad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.169198 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e3ee607233b43539ecbcff732d641d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.169354 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0aab16e37f6e433aa4834dd6606f78a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.170365 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d8673f26fe748c7aa5c15f0112730b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.174672 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 88f74467a168473ab81d6fad2bddacab (DEFAULT_TABLE table=default.Table_e027444cbd5545a38f6e4f35e1ed90c3 [id=e36fab9d943649d9bbc994d3d490c545]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.175257 27216 tablet_service.cc:1468] Processing CreateTablet for tablet 6d8673f26fe748c7aa5c15f0112730b2 (DEFAULT_TABLE table=default.Table_23a41a80bee1454e983b6dff5b8e1c8b [id=998235969e704094952d213816af798b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.175683 27212 tablet_service.cc:1468] Processing CreateTablet for tablet 0aab16e37f6e433aa4834dd6606f78a1 (DEFAULT_TABLE table=default.Table_762b9caaaec44868b1816efca7d186cf [id=d061ae74dd2c4db0af3f19caa59a0fad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.176148 27221 tablet_service.cc:1468] Processing CreateTablet for tablet 2d4df4cc00c9433bb120566a797be986 (DEFAULT_TABLE table=default.Table_8851a4dad8da4bb8907db1ff956dd28f [id=27dbd6d3bb504a82a84b666698c93299]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.176561 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 1e3ee607233b43539ecbcff732d641d4 (DEFAULT_TABLE table=default.Table_473b25f4188347ffba83a46006e66cf1 [id=244396a4b04a4a8ea9728362d526a609]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.177457 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88f74467a168473ab81d6fad2bddacab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.177348 29108 consensus_queue.cc:1035] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:34.178962 27693 tablet_bootstrap.cc:654] T a38f65e992884635aae9942681c0dc4f P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:34.180807 27217 tablet_service.cc:1468] Processing CreateTablet for tablet e65eac0cf8fa4a29927db5149d39a3f6 (DEFAULT_TABLE table=default.Table_66ee6081d8504c119f365516fc8be9f5 [id=a4286d274c8f4ff5ba944f677288099a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.181136 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e65eac0cf8fa4a29927db5149d39a3f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.182389 27212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0aab16e37f6e433aa4834dd6606f78a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.183740 29108 consensus_queue.cc:1035] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:34.184597 27687 ts_tablet_manager.cc:1918] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:34.186251 29108 raft_consensus.cc:491] T 200c06f296b94eab94f35f312c5a443b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:34.186384 29108 raft_consensus.cc:513] T 200c06f296b94eab94f35f312c5a443b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.187143 29108 leader_election.cc:290] T 200c06f296b94eab94f35f312c5a443b P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:34.187532 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "200c06f296b94eab94f35f312c5a443b" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:34.187683 27320 leader_election.cc:343] T 200c06f296b94eab94f35f312c5a443b P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 200c06f296b94eab94f35f312c5a443b
I20250903 20:05:34.187734 27249 raft_consensus.cc:2391] T 200c06f296b94eab94f35f312c5a443b P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 51578fb7bceb4e119b3ab1e58c4f5528 in current term 1: Already voted for candidate e72ef1dbcc5143cf9f04a808f101c80f in this term.
I20250903 20:05:34.188365 27321 leader_election.cc:304] T 200c06f296b94eab94f35f312c5a443b P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:34.191684 27216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d8673f26fe748c7aa5c15f0112730b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.193527 27684 log.cc:1199] T 4c18a98d5c0f4a43ae91c9130047b681 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/4c18a98d5c0f4a43ae91c9130047b681
I20250903 20:05:34.194155 27684 ts_tablet_manager.cc:1939] T 4c18a98d5c0f4a43ae91c9130047b681 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:34.194589 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d4df4cc00c9433bb120566a797be986. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.195520 29108 raft_consensus.cc:3058] T 200c06f296b94eab94f35f312c5a443b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:34.196138 27684 tablet_replica.cc:331] T 2d9c1642fec841d497ff3496dd4564db P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:34.198040 29108 raft_consensus.cc:2747] T 200c06f296b94eab94f35f312c5a443b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:34.198427 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e3ee607233b43539ecbcff732d641d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.205678 28916 consensus_queue.cc:1035] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:34.206380 27687 log.cc:1199] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/22bc87d1a4804b6c80f77f9692500d63
I20250903 20:05:34.207118 27687 ts_tablet_manager.cc:1939] T 22bc87d1a4804b6c80f77f9692500d63 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:34.209702 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1031) took 1912 ms. Trace:
I20250903 20:05:34.221889 27214 tablet_service.cc:1468] Processing CreateTablet for tablet ffc1cf1b5d59485bad240bceb0d37804 (DEFAULT_TABLE table=default.Table_b1d886984546451495e327fac36f5e49 [id=86455fa9f50241b8a5b14836cbe5a867]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.222791 27222 tablet_service.cc:1468] Processing CreateTablet for tablet f77b44eacc7f4a978c022ef296eda9ab (DEFAULT_TABLE table=default.Table_12a587f34ab643568acbad040e1ce139 [id=cf0b4416a7e744649da5f411ac48b975]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.223330 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f77b44eacc7f4a978c022ef296eda9ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.228652 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffc1cf1b5d59485bad240bceb0d37804. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.229775 28874 raft_consensus.cc:491] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51578fb7bceb4e119b3ab1e58c4f5528)
I20250903 20:05:34.229971 28874 raft_consensus.cc:513] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.230751 28874 leader_election.cc:290] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
W20250903 20:05:34.231658 27183 leader_election.cc:343] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: e8bc99de4ffe43d58351d170b7c61d8a
W20250903 20:05:34.231854 27184 leader_election.cc:343] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: e8bc99de4ffe43d58351d170b7c61d8a
I20250903 20:05:34.231985 27184 leader_election.cc:304] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:34.232275 28874 raft_consensus.cc:2747] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:34.233896 29196 raft_consensus.cc:513] T 7a9e55f0811345f7afa71645f38ebdaf P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:34.218149 27350 tablet_service.cc:1468] Processing CreateTablet for tablet f77b44eacc7f4a978c022ef296eda9ab (DEFAULT_TABLE table=default.Table_12a587f34ab643568acbad040e1ce139 [id=cf0b4416a7e744649da5f411ac48b975]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.235167 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f77b44eacc7f4a978c022ef296eda9ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.191238 27347 tablet_service.cc:1468] Processing CreateTablet for tablet e65eac0cf8fa4a29927db5149d39a3f6 (DEFAULT_TABLE table=default.Table_66ee6081d8504c119f365516fc8be9f5 [id=a4286d274c8f4ff5ba944f677288099a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.238797 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e65eac0cf8fa4a29927db5149d39a3f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.241834 28874 raft_consensus.cc:491] T 4af8c03c9d834405b4e25d35441db961 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51578fb7bceb4e119b3ab1e58c4f5528)
I20250903 20:05:34.242017 28874 raft_consensus.cc:513] T 4af8c03c9d834405b4e25d35441db961 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:34.242763 28874 leader_election.cc:290] T 4af8c03c9d834405b4e25d35441db961 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
W20250903 20:05:34.244347 27183 leader_election.cc:343] T 4af8c03c9d834405b4e25d35441db961 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 4af8c03c9d834405b4e25d35441db961
I20250903 20:05:34.246074 28947 leader_election.cc:290] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:34.246346 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "548b50819f694fa3ad508da9f442a99c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:34.246551 27383 raft_consensus.cc:3058] T 548b50819f694fa3ad508da9f442a99c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:34.249295 27383 raft_consensus.cc:2466] T 548b50819f694fa3ad508da9f442a99c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:34.250128 27508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a9e55f0811345f7afa71645f38ebdaf" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:34.250272 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a9e55f0811345f7afa71645f38ebdaf" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:34.250365 27508 raft_consensus.cc:3058] T 7a9e55f0811345f7afa71645f38ebdaf P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:34.250459 27383 raft_consensus.cc:3058] T 7a9e55f0811345f7afa71645f38ebdaf P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:34.253239 27508 raft_consensus.cc:2466] T 7a9e55f0811345f7afa71645f38ebdaf P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:34.253263 27383 raft_consensus.cc:2466] T 7a9e55f0811345f7afa71645f38ebdaf P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:34.253928 27183 leader_election.cc:304] T 7a9e55f0811345f7afa71645f38ebdaf P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
W20250903 20:05:34.254281 27184 leader_election.cc:343] T 4af8c03c9d834405b4e25d35441db961 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 4af8c03c9d834405b4e25d35441db961
I20250903 20:05:34.254506 28947 raft_consensus.cc:2802] T 7a9e55f0811345f7afa71645f38ebdaf P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:34.254500 27184 leader_election.cc:304] T 4af8c03c9d834405b4e25d35441db961 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:34.254812 28874 raft_consensus.cc:2747] T 4af8c03c9d834405b4e25d35441db961 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:34.255367 27695 ts_tablet_manager.cc:1428] T 886b8783aa9d4618aad498d4a1115843 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.151s user 0.001s sys 0.001s
I20250903 20:05:34.257144 28947 raft_consensus.cc:695] T 7a9e55f0811345f7afa71645f38ebdaf P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:34.257230 27687 rpcz_store.cc:276] 0903 20:05:32.297140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.297180 (+ 40us) service_pool.cc:225] Handling call
0903 20:05:34.137842 (+1840662us) ts_tablet_manager.cc:1145] Deleting tablet 22bc87d1a4804b6c80f77f9692500d63
0903 20:05:34.137852 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.181222 (+ 43370us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:34.184491 (+ 3269us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:34.209672 (+ 25181us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1840523}
I20250903 20:05:34.257486 28947 consensus_queue.cc:237] T 7a9e55f0811345f7afa71645f38ebdaf P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:34.258410 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 22bc87d1a4804b6c80f77f9692500d63 (table default.Table_c967f8b0d1314f589eae529cc492493b [id=93b379ddcb834087a42ce197c318f764]) successfully deleted
I20250903 20:05:34.259135 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1032) took 1961 ms. Trace:
I20250903 20:05:34.260224 29196 leader_election.cc:290] T 7a9e55f0811345f7afa71645f38ebdaf P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:34.263180 27687 rpcz_store.cc:276] 0903 20:05:32.297445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.297486 (+ 41us) service_pool.cc:225] Handling call
0903 20:05:34.259013 (+1961527us) ts_tablet_manager.cc:1145] Deleting tablet 726f4d920f3e47fea41b4e12c021c988
0903 20:05:34.259027 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.259114 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1961382}
W20250903 20:05:34.264206 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 726f4d920f3e47fea41b4e12c021c988 because the tablet was not found. No further retry: Not found: Tablet not found: 726f4d920f3e47fea41b4e12c021c988
I20250903 20:05:34.264909 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1036) took 1766 ms. Trace:
I20250903 20:05:34.266618 28710 raft_consensus.cc:491] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:34.266752 27687 rpcz_store.cc:276] 0903 20:05:32.498723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.508039 (+ 9316us) service_pool.cc:225] Handling call
0903 20:05:34.264731 (+1756692us) ts_tablet_manager.cc:1145] Deleting tablet e35574d360334150bb494e13f55d3656
0903 20:05:34.264747 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.264883 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1756408}
I20250903 20:05:34.266793 28710 raft_consensus.cc:513] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:34.267261 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet e35574d360334150bb494e13f55d3656 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e35574d360334150bb494e13f55d3656 already in progress: creating tablet
I20250903 20:05:34.267467 28710 leader_election.cc:290] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:34.268689 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1035) took 1773 ms. Trace:
I20250903 20:05:34.270817 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a38f65e992884635aae9942681c0dc4f" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:34.270993 27695 tablet_bootstrap.cc:492] T 950a6fcd35f64f2d889831fe9cdf503a P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:34.271035 27239 raft_consensus.cc:2466] T a38f65e992884635aae9942681c0dc4f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:34.269049 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a38f65e992884635aae9942681c0dc4f" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:34.271595 27462 leader_election.cc:304] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
W20250903 20:05:34.269891 27320 consensus_peers.cc:489] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5002b7e8118c49b6a247cb8dc4ca1aea. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:34.271834 27687 rpcz_store.cc:276] 0903 20:05:32.495328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.508667 (+ 13339us) service_pool.cc:225] Handling call
0903 20:05:34.268530 (+1759863us) ts_tablet_manager.cc:1145] Deleting tablet 653eade3172344d7aa801edf906ae1d4
0903 20:05:34.268544 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.268667 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1759669}
W20250903 20:05:34.272233 27462 leader_election.cc:343] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:34.272305 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 653eade3172344d7aa801edf906ae1d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 653eade3172344d7aa801edf906ae1d4 already in progress: creating tablet
I20250903 20:05:34.272583 28710 raft_consensus.cc:2802] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:34.272619 27695 tablet_bootstrap.cc:654] T 950a6fcd35f64f2d889831fe9cdf503a P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:34.272673 28710 raft_consensus.cc:491] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:34.272749 28710 raft_consensus.cc:3058] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:34.274550 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1037) took 1706 ms. Trace:
I20250903 20:05:34.274631 27687 rpcz_store.cc:276] 0903 20:05:32.567698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.567767 (+ 69us) service_pool.cc:225] Handling call
0903 20:05:34.274319 (+1706552us) ts_tablet_manager.cc:1145] Deleting tablet a09771c026a9474c9f61ed259d843cca
0903 20:05:34.274338 (+ 19us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.274530 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1706009}
W20250903 20:05:34.275056 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet a09771c026a9474c9f61ed259d843cca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a09771c026a9474c9f61ed259d843cca already in progress: creating tablet
I20250903 20:05:34.276314 28710 raft_consensus.cc:513] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:34.277515 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a38f65e992884635aae9942681c0dc4f" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:34.277710 27249 raft_consensus.cc:3058] T a38f65e992884635aae9942681c0dc4f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:34.278074 28710 leader_election.cc:290] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:34.278141 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a38f65e992884635aae9942681c0dc4f" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
W20250903 20:05:34.278848 27462 leader_election.cc:343] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:34.278887 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1038) took 1710 ms. Trace:
I20250903 20:05:34.278949 27687 rpcz_store.cc:276] 0903 20:05:32.568245 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.568305 (+ 60us) service_pool.cc:225] Handling call
0903 20:05:34.278767 (+1710462us) ts_tablet_manager.cc:1145] Deleting tablet 8330ab3e4b6f4560b9e41a9df280c211
0903 20:05:34.278779 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.278872 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1710289}
W20250903 20:05:34.279364 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 8330ab3e4b6f4560b9e41a9df280c211 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8330ab3e4b6f4560b9e41a9df280c211 already in progress: creating tablet
I20250903 20:05:34.280544 27249 raft_consensus.cc:2466] T a38f65e992884635aae9942681c0dc4f P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:34.280606 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1039) took 1672 ms. Trace:
I20250903 20:05:34.280673 27687 rpcz_store.cc:276] 0903 20:05:32.608177 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.608241 (+ 64us) service_pool.cc:225] Handling call
0903 20:05:34.280500 (+1672259us) ts_tablet_manager.cc:1145] Deleting tablet 931404d9b3844d908baaed3190d57b56
0903 20:05:34.280509 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.280591 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1672032}
W20250903 20:05:34.281056 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 931404d9b3844d908baaed3190d57b56 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 931404d9b3844d908baaed3190d57b56 already in progress: creating tablet
I20250903 20:05:34.281152 27462 leader_election.cc:304] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:34.282168 28710 raft_consensus.cc:2802] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:34.283416 27692 tablet_bootstrap.cc:492] T 24eadc70e10d4d02bcb643d45c9e0e36 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:34.283607 27692 ts_tablet_manager.cc:1397] T 24eadc70e10d4d02bcb643d45c9e0e36 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.183s user 0.018s sys 0.008s
I20250903 20:05:34.284373 27692 raft_consensus.cc:357] T 24eadc70e10d4d02bcb643d45c9e0e36 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.284667 27692 raft_consensus.cc:383] T 24eadc70e10d4d02bcb643d45c9e0e36 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:34.284775 27692 raft_consensus.cc:738] T 24eadc70e10d4d02bcb643d45c9e0e36 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:34.285089 27692 consensus_queue.cc:260] T 24eadc70e10d4d02bcb643d45c9e0e36 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.286947 29200 tablet.cc:1722] T 2d9c1642fec841d497ff3496dd4564db P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:34.287657 27687 tablet_replica.cc:331] T 9367b3a22f624c4087848099de32c080 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:34.289844 27687 raft_consensus.cc:2241] T 9367b3a22f624c4087848099de32c080 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:34.291395 29202 tablet.cc:1722] T 2d9c1642fec841d497ff3496dd4564db P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:34.292367 27684 raft_consensus.cc:2241] T 2d9c1642fec841d497ff3496dd4564db P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:34.292807 27684 raft_consensus.cc:2270] T 2d9c1642fec841d497ff3496dd4564db P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:34.291086 27687 raft_consensus.cc:2270] T 9367b3a22f624c4087848099de32c080 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:34.295233 27051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.125:44059, user_credentials={real_user=slave}} blocked reactor thread for 189954us
I20250903 20:05:34.297782 27687 ts_tablet_manager.cc:1905] T 9367b3a22f624c4087848099de32c080 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:34.297892 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet ed096b452ba747f89e91edc15e6e3ac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:34.299260 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5f424141e5ff4643903a98bfeb011982" } modify_external_catalogs: true
I20250903 20:05:34.299494 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5f424141e5ff4643903a98bfeb011982" } modify_external_catalogs: true
I20250903 20:05:34.306532 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_258d201c94be4e2aa3dce9385816da45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:34.307138 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_258d201c94be4e2aa3dce9385816da45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:34.308774 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0fef1d019eff4323b7ba56368614aadc" } new_table_name: "default.Table_94f3e8f67f00459caaeb1430fa5203d5" modify_external_catalogs: true
I20250903 20:05:34.309655 27693 tablet_bootstrap.cc:492] T a38f65e992884635aae9942681c0dc4f P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:34.309787 27693 ts_tablet_manager.cc:1397] T a38f65e992884635aae9942681c0dc4f P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.153s user 0.008s sys 0.004s
I20250903 20:05:34.311085 27692 ts_tablet_manager.cc:1428] T 24eadc70e10d4d02bcb643d45c9e0e36 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.027s user 0.002s sys 0.002s
I20250903 20:05:34.310652 27693 raft_consensus.cc:357] T a38f65e992884635aae9942681c0dc4f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:34.311492 27692 tablet_bootstrap.cc:492] T 5557dfdd953a41e9bd3a8be4cd689604 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:34.312531 27693 raft_consensus.cc:383] T a38f65e992884635aae9942681c0dc4f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:34.312654 27693 raft_consensus.cc:738] T a38f65e992884635aae9942681c0dc4f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:34.312930 27693 consensus_queue.cc:260] T a38f65e992884635aae9942681c0dc4f P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:34.313678 27693 ts_tablet_manager.cc:1428] T a38f65e992884635aae9942681c0dc4f P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:34.320152 27692 tablet_bootstrap.cc:654] T 5557dfdd953a41e9bd3a8be4cd689604 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:34.321974 27687 ts_tablet_manager.cc:1918] T 9367b3a22f624c4087848099de32c080 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 20:05:34.322106 27687 log.cc:1199] T 9367b3a22f624c4087848099de32c080 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/9367b3a22f624c4087848099de32c080
I20250903 20:05:34.324594 27695 tablet_bootstrap.cc:492] T 950a6fcd35f64f2d889831fe9cdf503a P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:34.325006 27695 ts_tablet_manager.cc:1397] T 950a6fcd35f64f2d889831fe9cdf503a P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.054s user 0.000s sys 0.003s
I20250903 20:05:34.327426 27687 ts_tablet_manager.cc:1939] T 9367b3a22f624c4087848099de32c080 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:34.331600 28710 raft_consensus.cc:695] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:34.332098 28710 consensus_queue.cc:237] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:34.336107 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1040) took 1622 ms. Trace:
I20250903 20:05:34.336233 27687 rpcz_store.cc:276] 0903 20:05:32.713837 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.714180 (+ 343us) service_pool.cc:225] Handling call
0903 20:05:34.287593 (+1573413us) ts_tablet_manager.cc:1145] Deleting tablet 9367b3a22f624c4087848099de32c080
0903 20:05:34.287608 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.317897 (+ 30289us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:34.321841 (+ 3944us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:34.336079 (+ 14238us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1535526}
I20250903 20:05:34.336692 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 9367b3a22f624c4087848099de32c080 (table default.Table_08918e8b515e4487bf4aee0a33234bd5 [id=891b259c1c6b44e2a82dc982c24e6cb6]) successfully deleted
I20250903 20:05:34.341164 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1041) took 1604 ms. Trace:
I20250903 20:05:34.341334 27687 rpcz_store.cc:276] 0903 20:05:32.736675 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.737219 (+ 544us) service_pool.cc:225] Handling call
0903 20:05:34.341002 (+1603783us) ts_tablet_manager.cc:1145] Deleting tablet df9d0f3662a245d7bc7cc184bc9ef3db
0903 20:05:34.341015 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.341141 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1588692}
W20250903 20:05:34.342377 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet df9d0f3662a245d7bc7cc184bc9ef3db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df9d0f3662a245d7bc7cc184bc9ef3db already in progress: creating tablet
I20250903 20:05:34.342831 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1042) took 1596 ms. Trace:
I20250903 20:05:34.342949 27687 rpcz_store.cc:276] 0903 20:05:32.745901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.746104 (+ 203us) service_pool.cc:225] Handling call
0903 20:05:34.342723 (+1596619us) ts_tablet_manager.cc:1145] Deleting tablet df9d0f3662a245d7bc7cc184bc9ef3db
0903 20:05:34.342733 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.342817 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1584414}
W20250903 20:05:34.350718 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet df9d0f3662a245d7bc7cc184bc9ef3db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df9d0f3662a245d7bc7cc184bc9ef3db already in progress: creating tablet
I20250903 20:05:34.351291 27687 tablet_replica.cc:331] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:34.351583 27687 raft_consensus.cc:2241] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:34.352030 27692 tablet_bootstrap.cc:492] T 5557dfdd953a41e9bd3a8be4cd689604 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:34.351948 27687 raft_consensus.cc:2270] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:34.352272 27692 ts_tablet_manager.cc:1397] T 5557dfdd953a41e9bd3a8be4cd689604 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.002s
I20250903 20:05:34.353575 27692 raft_consensus.cc:357] T 5557dfdd953a41e9bd3a8be4cd689604 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.353823 27692 raft_consensus.cc:383] T 5557dfdd953a41e9bd3a8be4cd689604 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:34.353925 27692 raft_consensus.cc:738] T 5557dfdd953a41e9bd3a8be4cd689604 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:34.354344 27692 consensus_queue.cc:260] T 5557dfdd953a41e9bd3a8be4cd689604 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.354471 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet 950a6fcd35f64f2d889831fe9cdf503a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:34.352802 27687 ts_tablet_manager.cc:1905] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:34.354969 27692 ts_tablet_manager.cc:1428] T 5557dfdd953a41e9bd3a8be4cd689604 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:34.355461 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet 1223743918814d88b401782ca4b521dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:34.355512 27692 tablet_bootstrap.cc:492] T f326ec776ba446be8d5dda5c1dc5b6ec P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:34.359012 27692 tablet_bootstrap.cc:654] T f326ec776ba446be8d5dda5c1dc5b6ec P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:34.361935 27350 tablet_service.cc:1515] Processing DeleteTablet for tablet 22bc87d1a4804b6c80f77f9692500d63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.362293 27347 tablet_service.cc:1515] Processing DeleteTablet for tablet 45f7068bafaf48d290ded9d1272c0e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.362437 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 8fff36d5a80e45af8a256c3908555e08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.362553 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet 5002b7e8118c49b6a247cb8dc4ca1aea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.363138 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet 200c06f296b94eab94f35f312c5a443b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.363374 27352 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c18a98d5c0f4a43ae91c9130047b681 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.363867 27062 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet fc634116a78f45dd833034ddda90e9bf (table default.Table_78e7a5b76ebc469396d8a3d670ab63ee [id=fb43441ca8f2415d99a628e81ef1642f]). Expected version 2 got 0
I20250903 20:05:34.364276 27062 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 7a9e55f0811345f7afa71645f38ebdaf (table default.Table_adf05e59a7be478b89666c3162ac83f1 [id=e69321e60a3a4e71a463a0011c0c6d65]). Expected version 2 got 0
I20250903 20:05:34.364475 27062 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 7345305820be4d32a3ceb227f045f009 (table default.Table_5f0199a660844c15a254a062c1540498 [id=52f969e50b9d466cbd14de6f252b7706]). Expected version 1 got 0
I20250903 20:05:34.366973 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet e877d56bbbfa4c5a92d025d0311ce5c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.378235 27692 tablet_bootstrap.cc:492] T f326ec776ba446be8d5dda5c1dc5b6ec P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:34.378407 27692 ts_tablet_manager.cc:1397] T f326ec776ba446be8d5dda5c1dc5b6ec P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.001s
I20250903 20:05:34.379482 27692 raft_consensus.cc:357] T f326ec776ba446be8d5dda5c1dc5b6ec P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:34.379534 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet bf459db3327640f88574fca7b5f2ca98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:29 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:34.379724 27692 raft_consensus.cc:383] T f326ec776ba446be8d5dda5c1dc5b6ec P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:34.379848 27692 raft_consensus.cc:738] T f326ec776ba446be8d5dda5c1dc5b6ec P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:34.379586 27216 tablet_service.cc:1515] Processing DeleteTablet for tablet f2629f46704e4a0fa7922533928ca2ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:34.380214 27692 consensus_queue.cc:260] T f326ec776ba446be8d5dda5c1dc5b6ec P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:34.380544 27687 ts_tablet_manager.cc:1918] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 20:05:34.380609 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet a38f65e992884635aae9942681c0dc4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:34.381062 27692 ts_tablet_manager.cc:1428] T f326ec776ba446be8d5dda5c1dc5b6ec P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:34.381767 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet aa9f5e0964b348b7af184f60cd145666 with delete_type TABLET_DATA_DELETED (replaced by tablet 26decef5841a4a3b85f2a5f578ae95dc) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:34.382977 27216 tablet_service.cc:1515] Processing DeleteTablet for tablet 548b50819f694fa3ad508da9f442a99c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:34.386579 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet 548b50819f694fa3ad508da9f442a99c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:34.388070 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet 46f49fd718be4a2996d781ab31268217 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:23 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:34.388765 27692 tablet_bootstrap.cc:492] T a3103e1595ec4c38ba65dca26fe1167c P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:34.394107 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_69b1032a1e7f4f4d972120dca05a0e38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:34.395005 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69b1032a1e7f4f4d972120dca05a0e38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:34.395584 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_021085b0fd254dbf961aa3d9aeadd6d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:34.396505 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_021085b0fd254dbf961aa3d9aeadd6d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:34.397388 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet bf459db3327640f88574fca7b5f2ca98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:29 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:34.398937 27687 log.cc:1199] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/a9f43781299e47a8aecc37a0b73d7a64
I20250903 20:05:34.399542 27687 ts_tablet_manager.cc:1939] T a9f43781299e47a8aecc37a0b73d7a64 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:34.403335 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1053) took 1582 ms. Trace:
I20250903 20:05:34.406286 27687 rpcz_store.cc:276] 0903 20:05:32.821302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.824496 (+ 3194us) service_pool.cc:225] Handling call
0903 20:05:34.351224 (+1526728us) ts_tablet_manager.cc:1145] Deleting tablet a9f43781299e47a8aecc37a0b73d7a64
0903 20:05:34.351239 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.365283 (+ 14044us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:34.380414 (+ 15131us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:34.403310 (+ 22896us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1526519}
I20250903 20:05:34.407047 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet a9f43781299e47a8aecc37a0b73d7a64 (table default.Table_73a4ac2afbad4654ac623cd581a45eb2 [id=f70d7ea93e154e6199f6766316353862]) successfully deleted
I20250903 20:05:34.408567 27687 tablet_replica.cc:331] T a253e3399622409e92299069b19d3698 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:34.408823 27687 raft_consensus.cc:2241] T a253e3399622409e92299069b19d3698 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:34.409569 27687 raft_consensus.cc:2270] T a253e3399622409e92299069b19d3698 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:34.411135 27687 ts_tablet_manager.cc:1905] T a253e3399622409e92299069b19d3698 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:34.415586 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet 200c06f296b94eab94f35f312c5a443b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:34.415519 27692 tablet_bootstrap.cc:654] T a3103e1595ec4c38ba65dca26fe1167c P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:34.417608 28710 raft_consensus.cc:491] T 238f2f30377b42c2852b016018aa3245 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:34.417761 28710 raft_consensus.cc:513] T 238f2f30377b42c2852b016018aa3245 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.418473 28710 leader_election.cc:290] T 238f2f30377b42c2852b016018aa3245 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:34.419538 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "238f2f30377b42c2852b016018aa3245" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:34.420023 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "238f2f30377b42c2852b016018aa3245" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:34.420807 27462 leader_election.cc:343] T 238f2f30377b42c2852b016018aa3245 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:34.420979 27462 leader_election.cc:343] T 238f2f30377b42c2852b016018aa3245 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:34.421043 27462 leader_election.cc:304] T 238f2f30377b42c2852b016018aa3245 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:34.421424 28710 raft_consensus.cc:2747] T 238f2f30377b42c2852b016018aa3245 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:34.423691 27493 tablet_service.cc:1468] Processing CreateTablet for tablet ffc1cf1b5d59485bad240bceb0d37804 (DEFAULT_TABLE table=default.Table_b1d886984546451495e327fac36f5e49 [id=86455fa9f50241b8a5b14836cbe5a867]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.425673 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffc1cf1b5d59485bad240bceb0d37804. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.434407 27695 raft_consensus.cc:357] T 950a6fcd35f64f2d889831fe9cdf503a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.434892 27695 raft_consensus.cc:383] T 950a6fcd35f64f2d889831fe9cdf503a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:34.435029 27695 raft_consensus.cc:738] T 950a6fcd35f64f2d889831fe9cdf503a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:34.435513 27695 consensus_queue.cc:260] T 950a6fcd35f64f2d889831fe9cdf503a P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.436652 27695 ts_tablet_manager.cc:1428] T 950a6fcd35f64f2d889831fe9cdf503a P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.111s user 0.001s sys 0.002s
I20250903 20:05:34.437862 27695 tablet_bootstrap.cc:492] T 1223743918814d88b401782ca4b521dd P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:34.441009 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 13137053365a428e9d1e8d8852a53b6a
I20250903 20:05:34.442183 27352 tablet_service.cc:1515] Processing DeleteTablet for tablet 13137053365a428e9d1e8d8852a53b6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:33 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.443665 27687 ts_tablet_manager.cc:1918] T a253e3399622409e92299069b19d3698 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:34.443930 27687 log.cc:1199] T a253e3399622409e92299069b19d3698 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/a253e3399622409e92299069b19d3698
I20250903 20:05:34.444749 27687 ts_tablet_manager.cc:1939] T a253e3399622409e92299069b19d3698 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:34.445108 27060 catalog_manager.cc:5582] T 7a9e55f0811345f7afa71645f38ebdaf P 89f8f0ee68b940db9ce250c0604f934a reported cstate change: term changed from 0 to 1, leader changed from <none> to 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65). New cstate: current_term: 1 leader_uuid: "89f8f0ee68b940db9ce250c0604f934a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:34.445631 27060 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 7a9e55f0811345f7afa71645f38ebdaf (table default.Table_adf05e59a7be478b89666c3162ac83f1 [id=e69321e60a3a4e71a463a0011c0c6d65]). Expected version 2 got 0
I20250903 20:05:34.446435 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 5557dfdd953a41e9bd3a8be4cd689604
I20250903 20:05:34.447144 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7039490d1b01480f9a14cdf90ce4643f" } new_table_name: "default.Table_2c2de9890e8a4b6eb41987afd2881351" modify_external_catalogs: true
I20250903 20:05:34.448642 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 2035344c7f6d478b89a89981d89f22ca
I20250903 20:05:34.449327 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_903b10284919440483b8f23fb840c6dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:34.449991 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_903b10284919440483b8f23fb840c6dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:34.452864 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_6b71f7836da34d58a17dacbfb7bcac07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:34.453104 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1060) took 1607 ms. Trace:
I20250903 20:05:34.453198 27687 rpcz_store.cc:276] 0903 20:05:32.846056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.846107 (+ 51us) service_pool.cc:225] Handling call
0903 20:05:34.407410 (+1561303us) ts_tablet_manager.cc:1145] Deleting tablet a253e3399622409e92299069b19d3698
0903 20:05:34.408513 (+ 1103us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.419737 (+ 11224us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:34.435235 (+ 15498us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:34.453078 (+ 17843us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1560508}
W20250903 20:05:34.453399 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b71f7836da34d58a17dacbfb7bcac07 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:34.453644 27687 tablet_replica.cc:331] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:34.453831 27687 raft_consensus.cc:2241] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:34.454120 27687 pending_rounds.cc:70] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528: Trying to abort 1 pending ops.
I20250903 20:05:34.454164 27687 pending_rounds.cc:77] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196385005621329920 op_type: NO_OP noop_request { }
I20250903 20:05:34.454277 27687 raft_consensus.cc:2887] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:34.454361 27687 raft_consensus.cc:2270] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:34.454612 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet f1d22e76ea72441db51f7e9db26eb520
I20250903 20:05:34.454916 27687 ts_tablet_manager.cc:1905] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:34.455068 27061 catalog_manager.cc:6652] ReplaceTablet: tablet f1d22e76ea72441db51f7e9db26eb520 deleted and replaced by tablet 069a9e21c417467fa68bac210dd64e4f
I20250903 20:05:34.456061 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet a2358eac86914f3ba51bfc6c299838c3
I20250903 20:05:34.456789 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_aab525602fac468990c81cb00e05f1ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:34.457278 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aab525602fac468990c81cb00e05f1ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:34.461043 27687 ts_tablet_manager.cc:1918] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:34.461180 27687 log.cc:1199] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/d1c96296a579440c887ba6c2a93f7204
I20250903 20:05:34.461423 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_62256f24380d4b02949f23a0c3c07d0a" } new_table_name: "default.Table_cbc1343da06443418cb594f7720331d8" modify_external_catalogs: true
I20250903 20:05:34.461568 27687 ts_tablet_manager.cc:1939] T d1c96296a579440c887ba6c2a93f7204 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:34.463521 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet a253e3399622409e92299069b19d3698 (table default.Table_8bbb5561118c4a68af214f8d319d2caf [id=2218f11e07d243adaa3a4d24e521b1b7]) successfully deleted
I20250903 20:05:34.463650 27352 tablet_service.cc:1515] Processing DeleteTablet for tablet 2035344c7f6d478b89a89981d89f22ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.463853 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet f1d22e76ea72441db51f7e9db26eb520 with delete_type TABLET_DATA_DELETED (replaced by tablet 069a9e21c417467fa68bac210dd64e4f) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.463995 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet a2358eac86914f3ba51bfc6c299838c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:33 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.464522 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet a2358eac86914f3ba51bfc6c299838c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:33 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:34.464789 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet 5557dfdd953a41e9bd3a8be4cd689604 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:33 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.465008 27221 tablet_service.cc:1515] Processing DeleteTablet for tablet f1d22e76ea72441db51f7e9db26eb520 with delete_type TABLET_DATA_DELETED (replaced by tablet 069a9e21c417467fa68bac210dd64e4f) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:34.465000 27057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d3945d5a36d84041b35394333dc704f9" } new_table_name: "default.Table_27479e1172ec4dcf9b4919e521916766" modify_external_catalogs: true
I20250903 20:05:34.465196 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet 2035344c7f6d478b89a89981d89f22ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:34.465551 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4fbd2782bdef4fc3bb40abb04472553d consensus state has no leader
I20250903 20:05:34.465883 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_3838c77991ba4638b506d7cc7fc43fe9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:34.466826 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_9d4f2aec0e3f4f858affaa9a001f0b88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:34.467247 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1061) took 1620 ms. Trace:
W20250903 20:05:34.467293 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3838c77991ba4638b506d7cc7fc43fe9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:34.467331 27687 rpcz_store.cc:276] 0903 20:05:32.847044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.847089 (+ 45us) service_pool.cc:225] Handling call
0903 20:05:34.453590 (+1606501us) ts_tablet_manager.cc:1145] Deleting tablet d1c96296a579440c887ba6c2a93f7204
0903 20:05:34.453602 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.457981 (+ 4379us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:34.460915 (+ 2934us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:34.467224 (+ 6309us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1606345}
I20250903 20:05:34.467626 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_67b0c9f0d9164596a34c1f4a5a8cfe2f" } new_table_name: "default.Table_68b075a000dd40478007dee2aa4bd1f6" modify_external_catalogs: true
W20250903 20:05:34.467293 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d4f2aec0e3f4f858affaa9a001f0b88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:34.473503 27064 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7849ab155085483f9e0ba20afc6f34a9" } modify_external_catalogs: true
I20250903 20:05:34.473744 27064 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7849ab155085483f9e0ba20afc6f34a9" } modify_external_catalogs: true
I20250903 20:05:34.474610 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet d1c96296a579440c887ba6c2a93f7204 (table default.Table_66babebee1c9462ea057c8489d43846d [id=a4d648e314504b68bc297abdbccad46f]) successfully deleted
I20250903 20:05:34.474821 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_fbe1657165754851afaaadbf3028cc72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:34.475386 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1062) took 1628 ms. Trace:
W20250903 20:05:34.475441 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fbe1657165754851afaaadbf3028cc72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:34.475488 27687 rpcz_store.cc:276] 0903 20:05:32.847366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.847409 (+ 43us) service_pool.cc:225] Handling call
0903 20:05:34.475187 (+1627778us) ts_tablet_manager.cc:1145] Deleting tablet 512908bc4d6941ec81b145ee5795405f
0903 20:05:34.475204 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.475362 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1627601}
W20250903 20:05:34.475924 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 512908bc4d6941ec81b145ee5795405f because the tablet was not found. No further retry: Not found: Tablet not found: 512908bc4d6941ec81b145ee5795405f
I20250903 20:05:34.476222 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1057) took 1630 ms. Trace:
I20250903 20:05:34.476413 27687 rpcz_store.cc:276] 0903 20:05:32.845734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.862289 (+ 16555us) service_pool.cc:225] Handling call
0903 20:05:34.476148 (+1613859us) ts_tablet_manager.cc:1145] Deleting tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8
0903 20:05:34.476156 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.476208 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1613299}
I20250903 20:05:34.476850 27687 tablet_replica.cc:331] T 02b3e262dd7841dfbb0520377eb94743 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:34.477154 27687 raft_consensus.cc:2241] T 02b3e262dd7841dfbb0520377eb94743 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250903 20:05:34.477229 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8 because the tablet was not found. No further retry: Not found: Tablet not found: 1f6cd4d475f147fe8dfcc7e3a73dedf8
I20250903 20:05:34.477437 27687 raft_consensus.cc:2270] T 02b3e262dd7841dfbb0520377eb94743 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:34.477867 27216 tablet_service.cc:1515] Processing DeleteTablet for tablet 5557dfdd953a41e9bd3a8be4cd689604 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:33 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:34.478487 27493 tablet_service.cc:1468] Processing CreateTablet for tablet f77b44eacc7f4a978c022ef296eda9ab (DEFAULT_TABLE table=default.Table_12a587f34ab643568acbad040e1ce139 [id=cf0b4416a7e744649da5f411ac48b975]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.479025 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f77b44eacc7f4a978c022ef296eda9ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.479045 27685 ts_tablet_manager.cc:1918] T f9709dfd18654c5bb071e39da7c8889f P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:34.479168 27685 log.cc:1199] T f9709dfd18654c5bb071e39da7c8889f P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/f9709dfd18654c5bb071e39da7c8889f
I20250903 20:05:34.479525 27685 ts_tablet_manager.cc:1939] T f9709dfd18654c5bb071e39da7c8889f P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:34.479839 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 909) took 4230 ms. Trace:
I20250903 20:05:34.479913 27685 rpcz_store.cc:276] 0903 20:05:30.249689 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.255245 (+ 5556us) service_pool.cc:225] Handling call
0903 20:05:34.096103 (+3840858us) ts_tablet_manager.cc:1145] Deleting tablet f9709dfd18654c5bb071e39da7c8889f
0903 20:05:34.096112 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.268641 (+172529us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:34.478917 (+210276us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:34.479823 (+ 906us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3840711}
I20250903 20:05:34.481148 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet f9709dfd18654c5bb071e39da7c8889f (table default.Table_876161e56d1d4c4c9d6726471530d965 [id=39cbb7f0d8e145c6b5f283133017b80f]) successfully deleted
I20250903 20:05:34.481230 27500 tablet_service.cc:1468] Processing CreateTablet for tablet e65eac0cf8fa4a29927db5149d39a3f6 (DEFAULT_TABLE table=default.Table_66ee6081d8504c119f365516fc8be9f5 [id=a4286d274c8f4ff5ba944f677288099a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.481669 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e65eac0cf8fa4a29927db5149d39a3f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.485988 27687 ts_tablet_manager.cc:1905] T 02b3e262dd7841dfbb0520377eb94743 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:34.490113 27500 tablet_service.cc:1468] Processing CreateTablet for tablet 2d4df4cc00c9433bb120566a797be986 (DEFAULT_TABLE table=default.Table_8851a4dad8da4bb8907db1ff956dd28f [id=27dbd6d3bb504a82a84b666698c93299]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.490672 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d4df4cc00c9433bb120566a797be986. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:34.494182 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d9daec5ba727475bb4ce7eb531424421 consensus state has no leader
I20250903 20:05:34.494495 27058 master_service.cc:948] ReplaceTablet: received request to replace tablet a785adcd0bd943fc97b201b6d29073f4 from {username='slave'} at 127.0.0.1:45936
W20250903 20:05:34.498483 27059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a9adbac18a97487b93009784023348ad consensus state has no leader
I20250903 20:05:34.499701 27493 tablet_service.cc:1468] Processing CreateTablet for tablet 0aab16e37f6e433aa4834dd6606f78a1 (DEFAULT_TABLE table=default.Table_762b9caaaec44868b1816efca7d186cf [id=d061ae74dd2c4db0af3f19caa59a0fad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.500038 27693 tablet_bootstrap.cc:492] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:34.500167 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0aab16e37f6e433aa4834dd6606f78a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.501721 27693 tablet_bootstrap.cc:654] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:34.502053 27497 tablet_service.cc:1468] Processing CreateTablet for tablet 88f74467a168473ab81d6fad2bddacab (DEFAULT_TABLE table=default.Table_e027444cbd5545a38f6e4f35e1ed90c3 [id=e36fab9d943649d9bbc994d3d490c545]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.502472 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88f74467a168473ab81d6fad2bddacab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.503047 29220 raft_consensus.cc:491] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:34.503183 29220 raft_consensus.cc:513] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:34.503739 29220 leader_election.cc:290] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:34.504839 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "886b8783aa9d4618aad498d4a1115843" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:34.505030 27379 raft_consensus.cc:2466] T 886b8783aa9d4618aad498d4a1115843 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:34.505432 27462 leader_election.cc:304] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:34.505642 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "886b8783aa9d4618aad498d4a1115843" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:34.505795 29220 raft_consensus.cc:2802] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:34.505862 29220 raft_consensus.cc:491] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:34.505841 27240 raft_consensus.cc:2466] T 886b8783aa9d4618aad498d4a1115843 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:34.505919 29220 raft_consensus.cc:3058] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:34.506371 29221 raft_consensus.cc:491] T 5557dfdd953a41e9bd3a8be4cd689604 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:34.506490 29221 raft_consensus.cc:513] T 5557dfdd953a41e9bd3a8be4cd689604 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.507014 29221 leader_election.cc:290] T 5557dfdd953a41e9bd3a8be4cd689604 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:34.499334 27687 ts_tablet_manager.cc:1918] T 02b3e262dd7841dfbb0520377eb94743 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:34.507651 27687 log.cc:1199] T 02b3e262dd7841dfbb0520377eb94743 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/02b3e262dd7841dfbb0520377eb94743
I20250903 20:05:34.507736 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5557dfdd953a41e9bd3a8be4cd689604" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:34.508252 27462 leader_election.cc:343] T 5557dfdd953a41e9bd3a8be4cd689604 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:34.508307 29220 raft_consensus.cc:513] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:34.508641 27687 ts_tablet_manager.cc:1939] T 02b3e262dd7841dfbb0520377eb94743 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:34.508811 29220 leader_election.cc:290] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:34.509289 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5557dfdd953a41e9bd3a8be4cd689604" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:34.509627 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "886b8783aa9d4618aad498d4a1115843" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:34.509690 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1056) took 1664 ms. Trace:
I20250903 20:05:34.509778 27240 raft_consensus.cc:3058] T 886b8783aa9d4618aad498d4a1115843 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:34.509824 27687 rpcz_store.cc:276] 0903 20:05:32.845554 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.862893 (+ 17339us) service_pool.cc:225] Handling call
0903 20:05:34.476788 (+1613895us) ts_tablet_manager.cc:1145] Deleting tablet 02b3e262dd7841dfbb0520377eb94743
0903 20:05:34.476796 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.489283 (+ 12487us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:34.499218 (+ 9935us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:34.509669 (+ 10451us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1612613}
I20250903 20:05:34.510787 27687 tablet_replica.cc:331] T a0f79831866b40c49c5b27ce2014c3f6 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:34.511035 27687 raft_consensus.cc:2241] T a0f79831866b40c49c5b27ce2014c3f6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:34.511345 27687 raft_consensus.cc:2270] T a0f79831866b40c49c5b27ce2014c3f6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:34.512359 27497 tablet_service.cc:1468] Processing CreateTablet for tablet 6d8673f26fe748c7aa5c15f0112730b2 (DEFAULT_TABLE table=default.Table_23a41a80bee1454e983b6dff5b8e1c8b [id=998235969e704094952d213816af798b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.512426 27240 raft_consensus.cc:2466] T 886b8783aa9d4618aad498d4a1115843 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:34.512759 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d8673f26fe748c7aa5c15f0112730b2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:34.513111 27462 leader_election.cc:343] T 5557dfdd953a41e9bd3a8be4cd689604 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:34.513103 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "886b8783aa9d4618aad498d4a1115843" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:34.513232 27462 leader_election.cc:304] T 5557dfdd953a41e9bd3a8be4cd689604 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:34.513283 27379 raft_consensus.cc:3058] T 886b8783aa9d4618aad498d4a1115843 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:34.513406 27687 ts_tablet_manager.cc:1905] T a0f79831866b40c49c5b27ce2014c3f6 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:34.513607 27462 leader_election.cc:304] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:34.513808 29221 raft_consensus.cc:2747] T 5557dfdd953a41e9bd3a8be4cd689604 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:34.513929 29221 raft_consensus.cc:2802] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:34.514007 29221 raft_consensus.cc:695] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:34.514259 29221 consensus_queue.cc:237] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:34.515256 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 02b3e262dd7841dfbb0520377eb94743 (table default.Table_3f1f99f4ab084617b3a72447d6e84209 [id=a30b474b4c8b494db74d62e576e509a7]) successfully deleted
I20250903 20:05:34.515771 27379 raft_consensus.cc:2466] T 886b8783aa9d4618aad498d4a1115843 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:34.518908 27493 tablet_service.cc:1468] Processing CreateTablet for tablet 1e3ee607233b43539ecbcff732d641d4 (DEFAULT_TABLE table=default.Table_473b25f4188347ffba83a46006e66cf1 [id=244396a4b04a4a8ea9728362d526a609]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.519337 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e3ee607233b43539ecbcff732d641d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.521375 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet 13137053365a428e9d1e8d8852a53b6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:33 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:34.522420 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 45f7068bafaf48d290ded9d1272c0e46 (table default.Table_7179a3a012834e34b703156b9c834722 [id=e660a48b356a4b0784016e527303f1f6]) successfully deleted
W20250903 20:05:34.525667 27051 catalog_manager.cc:4895] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 567630e2a93645e58f9bf05ffff87b32 because the tablet was not found. No further retry: Not found: Tablet not found: 567630e2a93645e58f9bf05ffff87b32
I20250903 20:05:34.526403 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 54f1ce5192f547f595dea23a9eabf8e6
W20250903 20:05:34.526526 27051 catalog_manager.cc:4895] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 22bc87d1a4804b6c80f77f9692500d63 because the tablet was not found. No further retry: Not found: Tablet not found: 22bc87d1a4804b6c80f77f9692500d63
I20250903 20:05:34.527786 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet 54f1ce5192f547f595dea23a9eabf8e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.528275 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet 54f1ce5192f547f595dea23a9eabf8e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:34.528942 27693 tablet_bootstrap.cc:492] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:34.529207 27693 ts_tablet_manager.cc:1397] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.000s
I20250903 20:05:34.529943 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 922) took 4005 ms. Trace:
I20250903 20:05:34.530141 27685 rpcz_store.cc:276] 0903 20:05:30.524518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.524573 (+ 55us) service_pool.cc:225] Handling call
0903 20:05:34.529780 (+4005207us) ts_tablet_manager.cc:1145] Deleting tablet 07439db2c3b8470eb2b797cc68376aa7
0903 20:05:34.529792 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.529921 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4005028}
I20250903 20:05:34.530197 27693 raft_consensus.cc:357] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.530423 27693 raft_consensus.cc:383] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:34.530510 27693 raft_consensus.cc:738] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:34.530860 27693 consensus_queue.cc:260] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.533066 27693 ts_tablet_manager.cc:1428] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:34.536875 27693 tablet_bootstrap.cc:492] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:34.541296 27687 ts_tablet_manager.cc:1918] T a0f79831866b40c49c5b27ce2014c3f6 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:34.541453 27687 log.cc:1199] T a0f79831866b40c49c5b27ce2014c3f6 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/a0f79831866b40c49c5b27ce2014c3f6
I20250903 20:05:34.541918 27687 ts_tablet_manager.cc:1939] T a0f79831866b40c49c5b27ce2014c3f6 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:34.543519 27693 tablet_bootstrap.cc:654] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:34.530867 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 07439db2c3b8470eb2b797cc68376aa7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07439db2c3b8470eb2b797cc68376aa7 already in progress: creating tablet
I20250903 20:05:34.544183 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 4c18a98d5c0f4a43ae91c9130047b681 (table default.Table_16b62656e76142339c0a42ca5a7f5e6b [id=189c3642e92c43759dee94f23ef1aec3]) successfully deleted
W20250903 20:05:34.544659 27051 catalog_manager.cc:4895] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 541278ca9ffb4e41b341139c2faa337f because the tablet was not found. No further retry: Not found: Tablet not found: 541278ca9ffb4e41b341139c2faa337f
W20250903 20:05:34.544955 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 65d07ebe1ec94baabc6ce011e761b4d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65d07ebe1ec94baabc6ce011e761b4d5 already in progress: creating tablet
W20250903 20:05:34.545226 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet e35ae50252734a099ba5ce2d04e49e1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e35ae50252734a099ba5ce2d04e49e1f already in progress: creating tablet
I20250903 20:05:34.547619 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet 13137053365a428e9d1e8d8852a53b6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:33 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:34.547958 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 5557dfdd953a41e9bd3a8be4cd689604 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:33 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:34.550185 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 2035344c7f6d478b89a89981d89f22ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:34.550935 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1058) took 1705 ms. Trace:
I20250903 20:05:34.551115 27687 rpcz_store.cc:276] 0903 20:05:32.845840 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.893366 (+ 47526us) service_pool.cc:225] Handling call
0903 20:05:34.510740 (+1617374us) ts_tablet_manager.cc:1145] Deleting tablet a0f79831866b40c49c5b27ce2014c3f6
0903 20:05:34.510750 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.527199 (+ 16449us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:34.541165 (+ 13966us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:34.550906 (+ 9741us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1611833}
I20250903 20:05:34.551790 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1068) took 1574 ms. Trace:
I20250903 20:05:34.551923 27687 rpcz_store.cc:276] 0903 20:05:32.977713 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.977829 (+ 116us) service_pool.cc:225] Handling call
0903 20:05:34.551653 (+1573824us) ts_tablet_manager.cc:1145] Deleting tablet 3c47c5fca0ee4349b2667494f45af5d0
0903 20:05:34.551664 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.551776 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1557342}
I20250903 20:05:34.552594 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1069) took 1574 ms. Trace:
I20250903 20:05:34.552724 27687 rpcz_store.cc:276] 0903 20:05:32.977860 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.978064 (+ 204us) service_pool.cc:225] Handling call
0903 20:05:34.552468 (+1574404us) ts_tablet_manager.cc:1145] Deleting tablet b32c82c3d8c4420cb6e3a95129550076
0903 20:05:34.552475 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.552580 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1557715}
I20250903 20:05:34.553330 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet a0f79831866b40c49c5b27ce2014c3f6 (table default.Table_3d20da97fa86497e9376f8aa69f1c3fd [id=35b0c6c60b9b43b5ad6c376284f45da5]) successfully deleted
I20250903 20:05:34.553452 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1070) took 1485 ms. Trace:
I20250903 20:05:34.553587 27687 rpcz_store.cc:276] 0903 20:05:33.067629 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:33.077769 (+ 10140us) service_pool.cc:225] Handling call
0903 20:05:34.553332 (+1475563us) ts_tablet_manager.cc:1145] Deleting tablet 16549a6d9aa9432eb5cee19f69ce6957
0903 20:05:34.553342 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.553437 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1475333}
W20250903 20:05:34.553654 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 3c47c5fca0ee4349b2667494f45af5d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c47c5fca0ee4349b2667494f45af5d0 already in progress: creating tablet
W20250903 20:05:34.553871 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet b32c82c3d8c4420cb6e3a95129550076 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b32c82c3d8c4420cb6e3a95129550076 already in progress: creating tablet
W20250903 20:05:34.554138 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 16549a6d9aa9432eb5cee19f69ce6957 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16549a6d9aa9432eb5cee19f69ce6957 already in progress: creating tablet
W20250903 20:05:34.554553 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 1bf69315f07a470bb2b897ad7c590729 because the tablet was not found. No further retry: Not found: Tablet not found: 1bf69315f07a470bb2b897ad7c590729
W20250903 20:05:34.554817 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 3590506b3fd44a6c84aa469c14ed3e9f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3590506b3fd44a6c84aa469c14ed3e9f already in progress: creating tablet
I20250903 20:05:34.555476 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet 886b8783aa9d4618aad498d4a1115843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:34.556121 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet a785adcd0bd943fc97b201b6d29073f4
I20250903 20:05:34.556556 27058 catalog_manager.cc:6652] ReplaceTablet: tablet a785adcd0bd943fc97b201b6d29073f4 deleted and replaced by tablet efa7f4417a6d4917a9c805c8018bbfdf
I20250903 20:05:34.558329 27687 tablet_replica.cc:331] T 2d9c1642fec841d497ff3496dd4564db P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:34.558595 27693 tablet_bootstrap.cc:492] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:34.558754 27693 ts_tablet_manager.cc:1397] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.003s
I20250903 20:05:34.558597 27687 raft_consensus.cc:2241] T 2d9c1642fec841d497ff3496dd4564db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:34.559062 27687 raft_consensus.cc:2270] T 2d9c1642fec841d497ff3496dd4564db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:34.559573 27692 tablet_bootstrap.cc:492] T a3103e1595ec4c38ba65dca26fe1167c P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:34.559737 27692 ts_tablet_manager.cc:1397] T a3103e1595ec4c38ba65dca26fe1167c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.172s user 0.006s sys 0.013s
I20250903 20:05:34.559703 27693 raft_consensus.cc:357] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.559911 27693 raft_consensus.cc:383] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:34.559973 27693 raft_consensus.cc:738] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:34.560222 27693 consensus_queue.cc:260] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.560586 27692 raft_consensus.cc:357] T a3103e1595ec4c38ba65dca26fe1167c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.560760 27692 raft_consensus.cc:383] T a3103e1595ec4c38ba65dca26fe1167c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:34.560819 27692 raft_consensus.cc:738] T a3103e1595ec4c38ba65dca26fe1167c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:34.561023 27692 consensus_queue.cc:260] T a3103e1595ec4c38ba65dca26fe1167c P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.561678 27684 ts_tablet_manager.cc:1905] T 2d9c1642fec841d497ff3496dd4564db P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:34.561694 27692 ts_tablet_manager.cc:1428] T a3103e1595ec4c38ba65dca26fe1167c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:34.562045 27692 tablet_bootstrap.cc:492] T 472f0fb01263468a828ec13795656624 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:34.560719 27693 ts_tablet_manager.cc:1428] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:34.562348 27216 tablet_service.cc:1515] Processing DeleteTablet for tablet 950a6fcd35f64f2d889831fe9cdf503a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:34.562489 27693 tablet_bootstrap.cc:492] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:34.561527 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet a785adcd0bd943fc97b201b6d29073f4 with delete_type TABLET_DATA_DELETED (replaced by tablet efa7f4417a6d4917a9c805c8018bbfdf) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:34.562799 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet dd27901224d6442389e8fc09c6b498ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dd27901224d6442389e8fc09c6b498ab already in progress: creating tablet
W20250903 20:05:34.563056 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet d1a254d560904b9aa31b8ee7de65ffa5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1a254d560904b9aa31b8ee7de65ffa5 already in progress: creating tablet
W20250903 20:05:34.563226 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet df5f439f5aca4231af229159811b7a75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df5f439f5aca4231af229159811b7a75 already in progress: creating tablet
W20250903 20:05:34.563385 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 65d07ebe1ec94baabc6ce011e761b4d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65d07ebe1ec94baabc6ce011e761b4d5 already in progress: creating tablet
W20250903 20:05:34.563537 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet e35ae50252734a099ba5ce2d04e49e1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e35ae50252734a099ba5ce2d04e49e1f already in progress: creating tablet
I20250903 20:05:34.563750 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet a785adcd0bd943fc97b201b6d29073f4 with delete_type TABLET_DATA_DELETED (replaced by tablet efa7f4417a6d4917a9c805c8018bbfdf) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:34.563840 27692 tablet_bootstrap.cc:654] T 472f0fb01263468a828ec13795656624 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:34.564034 27693 tablet_bootstrap.cc:654] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:34.572475 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 923) took 4047 ms. Trace:
I20250903 20:05:34.572784 27685 rpcz_store.cc:276] 0903 20:05:30.524611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.525303 (+ 692us) service_pool.cc:225] Handling call
0903 20:05:34.572303 (+4047000us) ts_tablet_manager.cc:1145] Deleting tablet 167b359047ac457c9d165ec199cd9163
0903 20:05:34.572326 (+ 23us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.572446 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4046822}
W20250903 20:05:34.573381 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 167b359047ac457c9d165ec199cd9163 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 167b359047ac457c9d165ec199cd9163 already in progress: creating tablet
I20250903 20:05:34.573836 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 924) took 3997 ms. Trace:
I20250903 20:05:34.573966 27685 rpcz_store.cc:276] 0903 20:05:30.576079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.635169 (+ 59090us) service_pool.cc:225] Handling call
0903 20:05:34.573726 (+3938557us) ts_tablet_manager.cc:1145] Deleting tablet 1f2124e9baae4c819aeb51cef79c9ff3
0903 20:05:34.573735 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.573819 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3938116}
W20250903 20:05:34.574553 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 1f2124e9baae4c819aeb51cef79c9ff3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f2124e9baae4c819aeb51cef79c9ff3 already in progress: creating tablet
I20250903 20:05:34.574909 27685 tablet_replica.cc:331] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:34.574898 29108 raft_consensus.cc:491] T 45f7068bafaf48d290ded9d1272c0e46 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:34.575204 27685 raft_consensus.cc:2241] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:34.575114 29108 raft_consensus.cc:513] T 45f7068bafaf48d290ded9d1272c0e46 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.575518 27685 raft_consensus.cc:2270] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:34.575976 29108 leader_election.cc:290] T 45f7068bafaf48d290ded9d1272c0e46 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:34.576714 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45f7068bafaf48d290ded9d1272c0e46" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:34.577246 27321 leader_election.cc:343] T 45f7068bafaf48d290ded9d1272c0e46 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:34.578473 27320 leader_election.cc:343] T 45f7068bafaf48d290ded9d1272c0e46 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 45f7068bafaf48d290ded9d1272c0e46
I20250903 20:05:34.578576 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet f1d22e76ea72441db51f7e9db26eb520 with delete_type TABLET_DATA_DELETED (replaced by tablet 069a9e21c417467fa68bac210dd64e4f) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:34.578622 27320 leader_election.cc:304] T 45f7068bafaf48d290ded9d1272c0e46 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:34.578879 29108 raft_consensus.cc:2747] T 45f7068bafaf48d290ded9d1272c0e46 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:34.578934 27685 ts_tablet_manager.cc:1905] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:34.580451 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet a2358eac86914f3ba51bfc6c299838c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:33 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:34.582629 29108 raft_consensus.cc:491] T a38f65e992884635aae9942681c0dc4f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:34.582782 29108 raft_consensus.cc:513] T a38f65e992884635aae9942681c0dc4f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:34.583513 29108 leader_election.cc:290] T a38f65e992884635aae9942681c0dc4f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:34.584471 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a38f65e992884635aae9942681c0dc4f" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:34.584785 27248 raft_consensus.cc:2391] T a38f65e992884635aae9942681c0dc4f P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 51578fb7bceb4e119b3ab1e58c4f5528 in current term 1: Already voted for candidate e72ef1dbcc5143cf9f04a808f101c80f in this term.
I20250903 20:05:34.586894 27248 raft_consensus.cc:3058] T e877d56bbbfa4c5a92d025d0311ce5c6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:34.587930 27695 tablet_bootstrap.cc:654] T 1223743918814d88b401782ca4b521dd P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:34.591006 27508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a38f65e992884635aae9942681c0dc4f" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:34.591302 29108 consensus_queue.cc:1035] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:34.591711 27320 leader_election.cc:304] T a38f65e992884635aae9942681c0dc4f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:34.592267 29108 raft_consensus.cc:3058] T a38f65e992884635aae9942681c0dc4f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:34.595800 29200 tablet.cc:1722] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250903 20:05:34.598599 29108 raft_consensus.cc:2747] T a38f65e992884635aae9942681c0dc4f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:34.599256 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet 54f1ce5192f547f595dea23a9eabf8e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:34.600724 27692 tablet_bootstrap.cc:492] T 472f0fb01263468a828ec13795656624 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:34.600927 27692 ts_tablet_manager.cc:1397] T 472f0fb01263468a828ec13795656624 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20250903 20:05:34.601404 29202 tablet.cc:1722] T 6238740ce1894e899be9a40321f0ab73 P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250903 20:05:34.602432 27692 raft_consensus.cc:357] T 472f0fb01263468a828ec13795656624 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.602741 27692 raft_consensus.cc:383] T 472f0fb01263468a828ec13795656624 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:34.602862 27692 raft_consensus.cc:738] T 472f0fb01263468a828ec13795656624 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:34.603144 27692 consensus_queue.cc:260] T 472f0fb01263468a828ec13795656624 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.606102 27692 ts_tablet_manager.cc:1428] T 472f0fb01263468a828ec13795656624 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250903 20:05:34.606499 27692 tablet_bootstrap.cc:492] T 09fa32312c9648fbbd6fd327e9738066 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:34.608310 27692 tablet_bootstrap.cc:654] T 09fa32312c9648fbbd6fd327e9738066 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:34.617103 27060 master_service.cc:948] ReplaceTablet: received request to replace tablet 6800e0b6269045f29682704fd847d5d1 from {username='slave'} at 127.0.0.1:45936
W20250903 20:05:34.618105 27184 consensus_peers.cc:489] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 567630e2a93645e58f9bf05ffff87b32. This is attempt 1: this message will repeat every 5th retry.
W20250903 20:05:34.625007 27184 consensus_peers.cc:489] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a253e3399622409e92299069b19d3698. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:34.627347 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 6800e0b6269045f29682704fd847d5d1
I20250903 20:05:34.627972 27060 catalog_manager.cc:6652] ReplaceTablet: tablet 6800e0b6269045f29682704fd847d5d1 deleted and replaced by tablet a3cf18a9a8c94a58bb739ee2f81d19c0
I20250903 20:05:34.629230 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet 6800e0b6269045f29682704fd847d5d1 with delete_type TABLET_DATA_DELETED (replaced by tablet a3cf18a9a8c94a58bb739ee2f81d19c0) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:34.629225 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet 6800e0b6269045f29682704fd847d5d1 with delete_type TABLET_DATA_DELETED (replaced by tablet a3cf18a9a8c94a58bb739ee2f81d19c0) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.601414 27684 ts_tablet_manager.cc:1918] T 2d9c1642fec841d497ff3496dd4564db P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 20:05:34.622927 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet a785adcd0bd943fc97b201b6d29073f4 with delete_type TABLET_DATA_DELETED (replaced by tablet efa7f4417a6d4917a9c805c8018bbfdf) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:34.636474 27684 log.cc:1199] T 2d9c1642fec841d497ff3496dd4564db P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/2d9c1642fec841d497ff3496dd4564db
I20250903 20:05:34.637521 27695 tablet_bootstrap.cc:492] T 1223743918814d88b401782ca4b521dd P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:34.637694 27695 ts_tablet_manager.cc:1397] T 1223743918814d88b401782ca4b521dd P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.200s user 0.000s sys 0.004s
I20250903 20:05:34.638801 27695 raft_consensus.cc:357] T 1223743918814d88b401782ca4b521dd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.639153 27060 master_service.cc:948] ReplaceTablet: received request to replace tablet 8bebea9fa64e459cb24639e1f2d11ac3 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:34.638989 27695 raft_consensus.cc:383] T 1223743918814d88b401782ca4b521dd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:34.641638 27695 raft_consensus.cc:738] T 1223743918814d88b401782ca4b521dd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:34.641981 27695 consensus_queue.cc:260] T 1223743918814d88b401782ca4b521dd P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.642550 29202 tablet.cc:1722] T 67ca1dcf6fc7435db555392235f9900f P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250903 20:05:34.644515 27695 ts_tablet_manager.cc:1428] T 1223743918814d88b401782ca4b521dd P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250903 20:05:34.644901 27695 tablet_bootstrap.cc:492] T ed096b452ba747f89e91edc15e6e3ac1 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:34.645112 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet 6800e0b6269045f29682704fd847d5d1 with delete_type TABLET_DATA_DELETED (replaced by tablet a3cf18a9a8c94a58bb739ee2f81d19c0) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:34.646088 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 8bebea9fa64e459cb24639e1f2d11ac3
I20250903 20:05:34.646633 27060 catalog_manager.cc:6652] ReplaceTablet: tablet 8bebea9fa64e459cb24639e1f2d11ac3 deleted and replaced by tablet 6154aac026f446aca9fffe12a6b0c6a4
I20250903 20:05:34.650271 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet 8bebea9fa64e459cb24639e1f2d11ac3 with delete_type TABLET_DATA_DELETED (replaced by tablet 6154aac026f446aca9fffe12a6b0c6a4) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:34.650432 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet 8bebea9fa64e459cb24639e1f2d11ac3 with delete_type TABLET_DATA_DELETED (replaced by tablet 6154aac026f446aca9fffe12a6b0c6a4) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.652673 27695 tablet_bootstrap.cc:654] T ed096b452ba747f89e91edc15e6e3ac1 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:34.655017 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet 8bebea9fa64e459cb24639e1f2d11ac3 with delete_type TABLET_DATA_DELETED (replaced by tablet 6154aac026f446aca9fffe12a6b0c6a4) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:34.560352 27687 ts_tablet_manager.cc:1905] T 2d9c1642fec841d497ff3496dd4564db P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:34.658288 28808 raft_consensus.cc:491] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:34.658492 28808 raft_consensus.cc:513] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.659339 28808 leader_election.cc:290] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:34.660333 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa9f5e0964b348b7af184f60cd145666" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:34.660595 27383 raft_consensus.cc:2466] T aa9f5e0964b348b7af184f60cd145666 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:34.661051 27184 leader_election.cc:304] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:34.661111 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa9f5e0964b348b7af184f60cd145666" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:34.661434 28808 raft_consensus.cc:2802] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:34.661540 28808 raft_consensus.cc:491] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250903 20:05:34.661650 27183 leader_election.cc:343] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:34.661975 27685 ts_tablet_manager.cc:1918] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250903 20:05:34.661651 28808 raft_consensus.cc:3058] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:34.662286 27685 log.cc:1199] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/e8bc99de4ffe43d58351d170b7c61d8a
I20250903 20:05:34.662776 27685 ts_tablet_manager.cc:1939] T e8bc99de4ffe43d58351d170b7c61d8a P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:34.663166 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 926) took 3933 ms. Trace:
I20250903 20:05:34.663276 27685 rpcz_store.cc:276] 0903 20:05:30.729975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:30.730036 (+ 61us) service_pool.cc:225] Handling call
0903 20:05:34.574861 (+3844825us) ts_tablet_manager.cc:1145] Deleting tablet e8bc99de4ffe43d58351d170b7c61d8a
0903 20:05:34.574870 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.632783 (+ 57913us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:34.661815 (+ 29032us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:34.663130 (+ 1315us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3844614}
I20250903 20:05:34.663739 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 928) took 3651 ms. Trace:
I20250903 20:05:34.663815 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet e8bc99de4ffe43d58351d170b7c61d8a (table default.Table_10a7df6626f24353ae6661cbbdf3efb0 [id=416ff29f66d14564826a6734dde1c604]) successfully deleted
I20250903 20:05:34.663825 27685 rpcz_store.cc:276] 0903 20:05:31.011855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.018288 (+ 6433us) service_pool.cc:225] Handling call
0903 20:05:34.663624 (+3645336us) ts_tablet_manager.cc:1145] Deleting tablet 90776f6369744cec8ce380b7695c44f1
0903 20:05:34.663634 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.663723 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3645003}
I20250903 20:05:34.664135 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 929) took 3564 ms. Trace:
W20250903 20:05:34.664242 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 90776f6369744cec8ce380b7695c44f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90776f6369744cec8ce380b7695c44f1 already in progress: creating tablet
I20250903 20:05:34.664202 27685 rpcz_store.cc:276] 0903 20:05:31.099950 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.114766 (+ 14816us) service_pool.cc:225] Handling call
0903 20:05:34.664028 (+3549262us) ts_tablet_manager.cc:1145] Deleting tablet 96c9875830b246b3a9117536254cbc4e
0903 20:05:34.664040 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.664122 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3548971}
I20250903 20:05:34.664527 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 939) took 3077 ms. Trace:
I20250903 20:05:34.664592 27685 rpcz_store.cc:276] 0903 20:05:31.587073 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.587136 (+ 63us) service_pool.cc:225] Handling call
0903 20:05:34.664429 (+3077293us) ts_tablet_manager.cc:1145] Deleting tablet e64c4a9179b6491ba92df75e0cc1bc50
0903 20:05:34.664438 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.664510 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3077052}
W20250903 20:05:34.664726 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 96c9875830b246b3a9117536254cbc4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96c9875830b246b3a9117536254cbc4e already in progress: creating tablet
I20250903 20:05:34.664876 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 940) took 3077 ms. Trace:
I20250903 20:05:34.664940 27685 rpcz_store.cc:276] 0903 20:05:31.587750 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.587800 (+ 50us) service_pool.cc:225] Handling call
0903 20:05:34.664788 (+3076988us) ts_tablet_manager.cc:1145] Deleting tablet 3f303233e6c347289650c6cf83f8127f
0903 20:05:34.664796 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.664863 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3076835}
I20250903 20:05:34.665025 28808 raft_consensus.cc:513] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:34.665151 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet e64c4a9179b6491ba92df75e0cc1bc50 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e64c4a9179b6491ba92df75e0cc1bc50 already in progress: creating tablet
I20250903 20:05:34.665235 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 941) took 3076 ms. Trace:
I20250903 20:05:34.665309 27685 rpcz_store.cc:276] 0903 20:05:31.588714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.588762 (+ 48us) service_pool.cc:225] Handling call
0903 20:05:34.665142 (+3076380us) ts_tablet_manager.cc:1145] Deleting tablet 17514a2af39b402abbe8263d7c1abc7f
0903 20:05:34.665151 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.665222 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3076243}
I20250903 20:05:34.665548 27685 tablet_replica.cc:331] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:34.665732 27685 raft_consensus.cc:2241] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
W20250903 20:05:34.665783 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 3f303233e6c347289650c6cf83f8127f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f303233e6c347289650c6cf83f8127f already in progress: creating tablet
I20250903 20:05:34.665997 27685 raft_consensus.cc:2270] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:34.666029 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 17514a2af39b402abbe8263d7c1abc7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17514a2af39b402abbe8263d7c1abc7f already in progress: creating tablet
I20250903 20:05:34.666553 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa9f5e0964b348b7af184f60cd145666" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:34.666796 27383 raft_consensus.cc:3058] T aa9f5e0964b348b7af184f60cd145666 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:34.670281 28808 leader_election.cc:290] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:34.670985 27383 raft_consensus.cc:2466] T aa9f5e0964b348b7af184f60cd145666 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:34.671169 27695 tablet_bootstrap.cc:492] T ed096b452ba747f89e91edc15e6e3ac1 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:34.671321 27695 ts_tablet_manager.cc:1397] T ed096b452ba747f89e91edc15e6e3ac1 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.001s
I20250903 20:05:34.671516 27184 leader_election.cc:304] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:34.671842 28808 raft_consensus.cc:2802] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:34.672031 27695 raft_consensus.cc:357] T ed096b452ba747f89e91edc15e6e3ac1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.672199 27695 raft_consensus.cc:383] T ed096b452ba747f89e91edc15e6e3ac1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:34.672252 27695 raft_consensus.cc:738] T ed096b452ba747f89e91edc15e6e3ac1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:34.672456 27695 consensus_queue.cc:260] T ed096b452ba747f89e91edc15e6e3ac1 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.672927 27695 ts_tablet_manager.cc:1428] T ed096b452ba747f89e91edc15e6e3ac1 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:34.673211 27695 tablet_bootstrap.cc:492] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:34.674008 28808 raft_consensus.cc:695] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:34.674343 28808 consensus_queue.cc:237] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.675565 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa9f5e0964b348b7af184f60cd145666" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
W20250903 20:05:34.676153 27183 leader_election.cc:343] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:34.677443 27692 tablet_bootstrap.cc:492] T 09fa32312c9648fbbd6fd327e9738066 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:34.677596 27692 ts_tablet_manager.cc:1397] T 09fa32312c9648fbbd6fd327e9738066 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.071s user 0.003s sys 0.001s
I20250903 20:05:34.678488 27692 raft_consensus.cc:357] T 09fa32312c9648fbbd6fd327e9738066 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.678668 27692 raft_consensus.cc:383] T 09fa32312c9648fbbd6fd327e9738066 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:34.678721 27692 raft_consensus.cc:738] T 09fa32312c9648fbbd6fd327e9738066 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:34.678921 27692 consensus_queue.cc:260] T 09fa32312c9648fbbd6fd327e9738066 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.684756 27058 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 7a9e55f0811345f7afa71645f38ebdaf (table default.Table_adf05e59a7be478b89666c3162ac83f1 [id=e69321e60a3a4e71a463a0011c0c6d65]). Expected version 2 got 0
I20250903 20:05:34.685057 27058 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet fc634116a78f45dd833034ddda90e9bf (table default.Table_78e7a5b76ebc469396d8a3d670ab63ee [id=fb43441ca8f2415d99a628e81ef1642f]). Expected version 2 got 0
I20250903 20:05:34.685261 27058 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 24eadc70e10d4d02bcb643d45c9e0e36 (table default.Table_76a88a96cd544cda882c413357a7f8c3 [id=e9a291cff3f4497cad9393ab4c571bf7]). Expected version 3 got 0
I20250903 20:05:34.687813 27064 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 7a9e55f0811345f7afa71645f38ebdaf (table default.Table_adf05e59a7be478b89666c3162ac83f1 [id=e69321e60a3a4e71a463a0011c0c6d65]). Expected version 2 got 0
I20250903 20:05:34.688133 27064 catalog_manager.cc:5582] T 4bcd65022b7b4502983b6b58809047e3 P 51578fb7bceb4e119b3ab1e58c4f5528 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66). New cstate: current_term: 1 leader_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:34.688553 27064 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 4bcd65022b7b4502983b6b58809047e3 (table default.Table_df7e7287722a45988929dfd3f0ff3eb6 [id=7ee5ea1a352941d7ba044bf48f21848a]). Expected version 2 got 0
I20250903 20:05:34.689375 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet a38f65e992884635aae9942681c0dc4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:34.689603 27500 tablet_service.cc:1515] Processing DeleteTablet for tablet f326ec776ba446be8d5dda5c1dc5b6ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:34.689743 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 886b8783aa9d4618aad498d4a1115843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:34.689755 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet 238f2f30377b42c2852b016018aa3245 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:34.690294 27064 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet fc634116a78f45dd833034ddda90e9bf (table default.Table_78e7a5b76ebc469396d8a3d670ab63ee [id=fb43441ca8f2415d99a628e81ef1642f]). Expected version 2 got 0
I20250903 20:05:34.690519 27064 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 7345305820be4d32a3ceb227f045f009 (table default.Table_5f0199a660844c15a254a062c1540498 [id=52f969e50b9d466cbd14de6f252b7706]). Expected version 1 got 0
I20250903 20:05:34.691932 27692 ts_tablet_manager.cc:1428] T 09fa32312c9648fbbd6fd327e9738066 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250903 20:05:34.694272 29220 raft_consensus.cc:491] T 09fa32312c9648fbbd6fd327e9738066 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:34.694412 29220 raft_consensus.cc:513] T 09fa32312c9648fbbd6fd327e9738066 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.694968 29220 leader_election.cc:290] T 09fa32312c9648fbbd6fd327e9738066 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:34.695856 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09fa32312c9648fbbd6fd327e9738066" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:34.696365 27462 leader_election.cc:343] T 09fa32312c9648fbbd6fd327e9738066 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:34.696674 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09fa32312c9648fbbd6fd327e9738066" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:34.697161 27462 leader_election.cc:343] T 09fa32312c9648fbbd6fd327e9738066 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:34.697252 27462 leader_election.cc:304] T 09fa32312c9648fbbd6fd327e9738066 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:34.697672 29220 raft_consensus.cc:2747] T 09fa32312c9648fbbd6fd327e9738066 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:34.701567 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet aa9f5e0964b348b7af184f60cd145666 with delete_type TABLET_DATA_DELETED (replaced by tablet 26decef5841a4a3b85f2a5f578ae95dc) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.701882 27692 tablet_bootstrap.cc:492] T f1d22e76ea72441db51f7e9db26eb520 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:34.703936 27692 tablet_bootstrap.cc:654] T f1d22e76ea72441db51f7e9db26eb520 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:34.704182 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet 46f49fd718be4a2996d781ab31268217 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:23 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.704630 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet bf459db3327640f88574fca7b5f2ca98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:29 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.705232 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet a38f65e992884635aae9942681c0dc4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.705720 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet 548b50819f694fa3ad508da9f442a99c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.706252 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet 1223743918814d88b401782ca4b521dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.706809 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet 886b8783aa9d4618aad498d4a1115843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.707074 27685 ts_tablet_manager.cc:1905] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:34.707399 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet ed096b452ba747f89e91edc15e6e3ac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.713416 27687 ts_tablet_manager.cc:1918] T 2d9c1642fec841d497ff3496dd4564db P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 20:05:34.715255 27058 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_27479e1172ec4dcf9b4919e521916766 [id=5568f788d1414c83b4359b5adf28632a] alter complete (version 3)
I20250903 20:05:34.721041 27214 tablet_service.cc:1468] Processing CreateTablet for tablet c51076f613fa46a8bf1309efba28195f (DEFAULT_TABLE table=default.Table_92312eea209d4177bae77b8fa25fb7a8 [id=e74b3ba372f041568057c1a01e3d8a6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.721588 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c51076f613fa46a8bf1309efba28195f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.718731 27497 tablet_service.cc:1468] Processing CreateTablet for tablet c51076f613fa46a8bf1309efba28195f (DEFAULT_TABLE table=default.Table_92312eea209d4177bae77b8fa25fb7a8 [id=e74b3ba372f041568057c1a01e3d8a6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.720844 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 1285ea33be67467893d0c13c2592bca1 (DEFAULT_TABLE table=default.Table_4bcfc1c4f42c4c179852756650e5a925 [id=1526e45fffe34252b83e0f1c6aeabbf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.722999 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1285ea33be67467893d0c13c2592bca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.723328 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c51076f613fa46a8bf1309efba28195f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.729454 27695 tablet_bootstrap.cc:654] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:34.736367 29245 tablet.cc:1722] T 67ca1dcf6fc7435db555392235f9900f P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250903 20:05:34.739238 27058 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_94f3e8f67f00459caaeb1430fa5203d5 [id=53074cd97dec40c7b0e1444183d6707a] alter complete (version 3)
I20250903 20:05:34.742163 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet 09fa32312c9648fbbd6fd327e9738066 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:34.743341 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet a3103e1595ec4c38ba65dca26fe1167c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:34.744369 27497 tablet_service.cc:1468] Processing CreateTablet for tablet 514a0dc0bb3347648e83813943fcd489 (DEFAULT_TABLE table=default.Table_16c1f5a369af4b17a52df8480211496c [id=5f6c456e53a243fba531480d4fbf9077]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.744901 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 514a0dc0bb3347648e83813943fcd489. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.745826 27692 tablet_bootstrap.cc:492] T f1d22e76ea72441db51f7e9db26eb520 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:34.745966 27692 ts_tablet_manager.cc:1397] T f1d22e76ea72441db51f7e9db26eb520 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.000s
I20250903 20:05:34.746857 27692 raft_consensus.cc:357] T f1d22e76ea72441db51f7e9db26eb520 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.747021 27692 raft_consensus.cc:383] T f1d22e76ea72441db51f7e9db26eb520 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:34.747076 27692 raft_consensus.cc:738] T f1d22e76ea72441db51f7e9db26eb520 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:34.747282 27692 consensus_queue.cc:260] T f1d22e76ea72441db51f7e9db26eb520 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.749183 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_76a88a96cd544cda882c413357a7f8c3" } modify_external_catalogs: true
I20250903 20:05:34.749424 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_76a88a96cd544cda882c413357a7f8c3" } modify_external_catalogs: true
I20250903 20:05:34.750883 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_10275f9babd149008387d2c114944d27" } modify_external_catalogs: true
I20250903 20:05:34.751092 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_10275f9babd149008387d2c114944d27" } modify_external_catalogs: true
I20250903 20:05:34.756852 27692 ts_tablet_manager.cc:1428] T f1d22e76ea72441db51f7e9db26eb520 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250903 20:05:34.757220 27692 tablet_bootstrap.cc:492] T 4e34d6d5d68e4395aed78821ea077c1b P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:34.762837 27685 ts_tablet_manager.cc:1918] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250903 20:05:34.762995 27685 log.cc:1199] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/c8a293c6fa034e9fb29682439cb3bcc2
I20250903 20:05:34.763374 27685 ts_tablet_manager.cc:1939] T c8a293c6fa034e9fb29682439cb3bcc2 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:34.764384 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 52ee6ba19a734329acde7608327f2a78
I20250903 20:05:34.765916 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 52ee6ba19a734329acde7608327f2a78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:34.766180 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 24eadc70e10d4d02bcb643d45c9e0e36
I20250903 20:05:34.766367 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 952) took 2852 ms. Trace:
I20250903 20:05:34.766448 27685 rpcz_store.cc:276] 0903 20:05:31.913930 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.913973 (+ 43us) service_pool.cc:225] Handling call
0903 20:05:34.665503 (+2751530us) ts_tablet_manager.cc:1145] Deleting tablet c8a293c6fa034e9fb29682439cb3bcc2
0903 20:05:34.665512 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.714403 (+ 48891us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:34.762705 (+ 48302us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:34.766344 (+ 3639us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2751345}
I20250903 20:05:34.766863 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 951) took 2853 ms. Trace:
I20250903 20:05:34.766939 27685 rpcz_store.cc:276] 0903 20:05:31.913812 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:31.914668 (+ 856us) service_pool.cc:225] Handling call
0903 20:05:34.766724 (+2852056us) ts_tablet_manager.cc:1145] Deleting tablet b5bff419ed31453e80ceec23ea05caad
0903 20:05:34.766733 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.766844 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2851851}
I20250903 20:05:34.767270 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 954) took 2705 ms. Trace:
I20250903 20:05:34.767350 27685 rpcz_store.cc:276] 0903 20:05:32.061794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.061975 (+ 181us) service_pool.cc:225] Handling call
0903 20:05:34.767148 (+2705173us) ts_tablet_manager.cc:1145] Deleting tablet 4e34d6d5d68e4395aed78821ea077c1b
0903 20:05:34.767163 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.767247 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2704997}
I20250903 20:05:34.767642 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 24eadc70e10d4d02bcb643d45c9e0e36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:34.769299 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet c8a293c6fa034e9fb29682439cb3bcc2 (table default.Table_397211088a974a09bd7f031d5c7ca1cc [id=5cfc8827e4ca4f4a94424c48d6f71eb5]) successfully deleted
I20250903 20:05:34.769588 27216 tablet_service.cc:1515] Processing DeleteTablet for tablet 52ee6ba19a734329acde7608327f2a78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:34.769621 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet b5bff419ed31453e80ceec23ea05caad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5bff419ed31453e80ceec23ea05caad already in progress: creating tablet
I20250903 20:05:34.769794 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet 24eadc70e10d4d02bcb643d45c9e0e36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:34.769850 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 4e34d6d5d68e4395aed78821ea077c1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e34d6d5d68e4395aed78821ea077c1b already in progress: creating tablet
I20250903 20:05:34.772902 27222 tablet_service.cc:1468] Processing CreateTablet for tablet 514a0dc0bb3347648e83813943fcd489 (DEFAULT_TABLE table=default.Table_16c1f5a369af4b17a52df8480211496c [id=5f6c456e53a243fba531480d4fbf9077]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.773380 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 514a0dc0bb3347648e83813943fcd489. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.776350 27216 tablet_service.cc:1515] Processing DeleteTablet for tablet 1223743918814d88b401782ca4b521dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:34.778074 27216 tablet_service.cc:1515] Processing DeleteTablet for tablet ed096b452ba747f89e91edc15e6e3ac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:34.778527 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_dc1d2e1d3de342378af29da68425a820"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:34.779187 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc1d2e1d3de342378af29da68425a820 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:34.783715 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_01cecfeda8a247628e40e6d815ddb410" } modify_external_catalogs: true
I20250903 20:05:34.783958 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_01cecfeda8a247628e40e6d815ddb410" } modify_external_catalogs: true
I20250903 20:05:34.784854 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 1285ea33be67467893d0c13c2592bca1 (DEFAULT_TABLE table=default.Table_4bcfc1c4f42c4c179852756650e5a925 [id=1526e45fffe34252b83e0f1c6aeabbf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.785531 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1285ea33be67467893d0c13c2592bca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.786715 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f04fc2303d4c487f8d21543807e16df5" } new_table_name: "default.Table_6135232af1594805bd74e6219011444a" modify_external_catalogs: true
I20250903 20:05:34.786988 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_9c1562dad48f49b59b791ce173464002"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:34.787518 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c1562dad48f49b59b791ce173464002 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:34.788025 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_8f6632599c6148db883ee6c79d9d168b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:34.788494 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f6632599c6148db883ee6c79d9d168b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:34.789121 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_de11ae4b3b9b42c7b6e256023fb6a3a3" } modify_external_catalogs: true
I20250903 20:05:34.789279 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_de11ae4b3b9b42c7b6e256023fb6a3a3" } modify_external_catalogs: true
I20250903 20:05:34.789640 27057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ba146e717b8d4d7db2b5bec3d05a16a6" } new_table_name: "default.Table_fd33384721334ad0b70806e8e70a60f2" modify_external_catalogs: true
I20250903 20:05:34.793464 29196 raft_consensus.cc:491] T 950a6fcd35f64f2d889831fe9cdf503a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:34.793699 29196 raft_consensus.cc:513] T 950a6fcd35f64f2d889831fe9cdf503a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.794565 29196 leader_election.cc:290] T 950a6fcd35f64f2d889831fe9cdf503a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:34.795401 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "950a6fcd35f64f2d889831fe9cdf503a" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:34.795648 27511 raft_consensus.cc:2391] T 950a6fcd35f64f2d889831fe9cdf503a P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89f8f0ee68b940db9ce250c0604f934a in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
I20250903 20:05:34.811784 27692 tablet_bootstrap.cc:654] T 4e34d6d5d68e4395aed78821ea077c1b P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:34.811784 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 7203b4ba9047430ab782d4c0498466cc
I20250903 20:05:34.812678 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 7203b4ba9047430ab782d4c0498466cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:34.814338 29220 raft_consensus.cc:491] T a3103e1595ec4c38ba65dca26fe1167c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:34.814477 29220 raft_consensus.cc:513] T a3103e1595ec4c38ba65dca26fe1167c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.815119 29220 leader_election.cc:290] T a3103e1595ec4c38ba65dca26fe1167c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:34.817034 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b54f1ddd84a4489a88fa3ec0bc3dc6a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:34.817976 27216 tablet_service.cc:1515] Processing DeleteTablet for tablet 7203b4ba9047430ab782d4c0498466cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:34.818737 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3103e1595ec4c38ba65dca26fe1167c" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:34.820458 27462 leader_election.cc:343] T a3103e1595ec4c38ba65dca26fe1167c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:34.824358 29245 tablet.cc:1722] T 6238740ce1894e899be9a40321f0ab73 P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250903 20:05:34.829661 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 955) took 2767 ms. Trace:
I20250903 20:05:34.830291 27685 rpcz_store.cc:276] 0903 20:05:32.061888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.062187 (+ 299us) service_pool.cc:225] Handling call
0903 20:05:34.829508 (+2767321us) ts_tablet_manager.cc:1145] Deleting tablet 1ad2ca3e72434b69ad48e533da72e8fd
0903 20:05:34.829520 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.829637 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2705126}
W20250903 20:05:34.822429 27057 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 421dc8af40c94a89bc372e8779872a4e consensus state has no leader
W20250903 20:05:34.822747 27059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1fce05bd4e6c425e952001ffdc25c78a consensus state has no leader
I20250903 20:05:34.831795 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_4bc2d6a26f9b41f59aaee304f156a6c5" } new_table_name: "default.Table_66c6bafa9fa64867af5ff30e048122ed" modify_external_catalogs: true
I20250903 20:05:34.831846 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_559f9c5a607c4af7b3ff4c1094d48005"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:34.832438 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_559f9c5a607c4af7b3ff4c1094d48005 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:34.833953 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b54f1ddd84a4489a88fa3ec0bc3dc6a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:34.825912 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_01bdabab48594c29a7af573f7062658f" } new_table_name: "default.Table_3edadd8954444273b91ea4d39cf3a834" modify_external_catalogs: true
I20250903 20:05:34.826861 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_9702c9ad8b8a4f0286cefc21dcfd6f10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:34.835507 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9702c9ad8b8a4f0286cefc21dcfd6f10 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:34.836824 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 1ad2ca3e72434b69ad48e533da72e8fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ad2ca3e72434b69ad48e533da72e8fd already in progress: creating tablet
I20250903 20:05:34.830862 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_aded1170b7314a75b0d6c9d5ce4bcc00" } new_table_name: "default.Table_c30aa02092a4461cb1ee2f56c471d3f9" modify_external_catalogs: true
I20250903 20:05:34.837388 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 953) took 2775 ms. Trace:
I20250903 20:05:34.837493 27685 rpcz_store.cc:276] 0903 20:05:32.061649 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.061932 (+ 283us) service_pool.cc:225] Handling call
0903 20:05:34.837220 (+2775288us) ts_tablet_manager.cc:1145] Deleting tablet 2478fe89414b4845b0ba377c1cde98b7
0903 20:05:34.837232 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.837338 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2774487}
I20250903 20:05:34.831251 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet baac5b97f7eb4d18a0f294ca1595a383
I20250903 20:05:34.842545 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet baac5b97f7eb4d18a0f294ca1595a383 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:34.842516 27216 tablet_service.cc:1515] Processing DeleteTablet for tablet baac5b97f7eb4d18a0f294ca1595a383 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:34.843007 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_df2bc52dc58c498f8b5ad1165c9be27a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:34.843751 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df2bc52dc58c498f8b5ad1165c9be27a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:34.845710 27692 tablet_bootstrap.cc:492] T 4e34d6d5d68e4395aed78821ea077c1b P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:34.848065 27692 ts_tablet_manager.cc:1397] T 4e34d6d5d68e4395aed78821ea077c1b P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.091s user 0.001s sys 0.003s
I20250903 20:05:34.849435 27692 raft_consensus.cc:357] T 4e34d6d5d68e4395aed78821ea077c1b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:34.852828 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 2478fe89414b4845b0ba377c1cde98b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2478fe89414b4845b0ba377c1cde98b7 already in progress: creating tablet
W20250903 20:05:34.853035 27183 consensus_peers.cc:489] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c18a98d5c0f4a43ae91c9130047b681. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:34.857323 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 956) took 2781 ms. Trace:
I20250903 20:05:34.857465 27685 rpcz_store.cc:276] 0903 20:05:32.076122 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.076200 (+ 78us) service_pool.cc:225] Handling call
0903 20:05:34.857124 (+2780924us) ts_tablet_manager.cc:1145] Deleting tablet 6d21f38759d743af972635f8b5b8d8d2
0903 20:05:34.857140 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.857269 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2780722}
W20250903 20:05:34.858076 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 6d21f38759d743af972635f8b5b8d8d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d21f38759d743af972635f8b5b8d8d2 already in progress: creating tablet
I20250903 20:05:34.858738 27692 raft_consensus.cc:383] T 4e34d6d5d68e4395aed78821ea077c1b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:34.859251 27692 raft_consensus.cc:738] T 4e34d6d5d68e4395aed78821ea077c1b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:34.859517 27692 consensus_queue.cc:260] T 4e34d6d5d68e4395aed78821ea077c1b P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.860064 27692 ts_tablet_manager.cc:1428] T 4e34d6d5d68e4395aed78821ea077c1b P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250903 20:05:34.864521 27685 tablet_replica.cc:331] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:34.869369 27685 raft_consensus.cc:2241] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:34.876130 27685 raft_consensus.cc:2270] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:34.878535 27692 tablet_bootstrap.cc:492] T 8ba2636641654d8596fd2a5b0644d073 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:34.716357 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet 950a6fcd35f64f2d889831fe9cdf503a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.720110 27349 tablet_service.cc:1468] Processing CreateTablet for tablet c51076f613fa46a8bf1309efba28195f (DEFAULT_TABLE table=default.Table_92312eea209d4177bae77b8fa25fb7a8 [id=e74b3ba372f041568057c1a01e3d8a6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.879313 27687 log.cc:1199] T 2d9c1642fec841d497ff3496dd4564db P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/2d9c1642fec841d497ff3496dd4564db
I20250903 20:05:34.879700 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c51076f613fa46a8bf1309efba28195f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.726287 29224 consensus_queue.cc:1035] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250903 20:05:34.728700 27320 consensus_peers.cc:489] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e877d56bbbfa4c5a92d025d0311ce5c6. This is attempt 1: this message will repeat every 5th retry.
W20250903 20:05:34.882421 27320 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 153730us
I20250903 20:05:34.728853 29223 consensus_queue.cc:1035] T 4bcd65022b7b4502983b6b58809047e3 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:34.746724 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 1285ea33be67467893d0c13c2592bca1 (DEFAULT_TABLE table=default.Table_4bcfc1c4f42c4c179852756650e5a925 [id=1526e45fffe34252b83e0f1c6aeabbf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.886082 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1285ea33be67467893d0c13c2592bca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.887811 29200 tablet.cc:1722] T 67ca1dcf6fc7435db555392235f9900f P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250903 20:05:34.765823 27352 tablet_service.cc:1515] Processing DeleteTablet for tablet 52ee6ba19a734329acde7608327f2a78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.768560 27347 tablet_service.cc:1515] Processing DeleteTablet for tablet 24eadc70e10d4d02bcb643d45c9e0e36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.771270 27350 tablet_service.cc:1468] Processing CreateTablet for tablet 514a0dc0bb3347648e83813943fcd489 (DEFAULT_TABLE table=default.Table_16c1f5a369af4b17a52df8480211496c [id=5f6c456e53a243fba531480d4fbf9077]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.889899 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 514a0dc0bb3347648e83813943fcd489. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.895183 27692 tablet_bootstrap.cc:654] T 8ba2636641654d8596fd2a5b0644d073 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:34.898098 29224 raft_consensus.cc:491] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:34.818006 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet 7203b4ba9047430ab782d4c0498466cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.898367 29224 raft_consensus.cc:513] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.899022 29224 leader_election.cc:290] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:34.899629 27508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "238f2f30377b42c2852b016018aa3245" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:34.899821 27508 raft_consensus.cc:2466] T 238f2f30377b42c2852b016018aa3245 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:34.900353 27320 leader_election.cc:304] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:34.840663 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet baac5b97f7eb4d18a0f294ca1595a383 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.900709 29224 raft_consensus.cc:2802] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:34.900805 29224 raft_consensus.cc:491] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:34.900898 29224 raft_consensus.cc:3058] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:34.903460 29224 raft_consensus.cc:513] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.904098 29224 leader_election.cc:290] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:34.864107 27693 tablet_bootstrap.cc:492] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:34.904358 27693 ts_tablet_manager.cc:1397] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.342s user 0.043s sys 0.062s
I20250903 20:05:34.905184 27693 raft_consensus.cc:357] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.905365 27693 raft_consensus.cc:383] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:34.905452 27693 raft_consensus.cc:738] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:34.905659 27693 consensus_queue.cc:260] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:34.905833 27059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a7f25f00dff7426195ac51214a14022b consensus state has no leader
I20250903 20:05:34.906101 27693 ts_tablet_manager.cc:1428] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:34.906486 27693 tablet_bootstrap.cc:492] T 8ba2636641654d8596fd2a5b0644d073 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:34.884948 28567 consensus_queue.cc:1035] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:34.908088 28710 raft_consensus.cc:491] T 24eadc70e10d4d02bcb643d45c9e0e36 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:34.908236 28710 raft_consensus.cc:513] T 24eadc70e10d4d02bcb643d45c9e0e36 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.908842 28710 leader_election.cc:290] T 24eadc70e10d4d02bcb643d45c9e0e36 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:34.802184 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "950a6fcd35f64f2d889831fe9cdf503a" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:34.910147 27507 tablet_service.cc:1468] Processing CreateTablet for tablet cf341e1600a14cb1893d36c435e8667f (DEFAULT_TABLE table=default.Table_cedba29d5f42445cba418e98e554bb61 [id=f87638b05ddb417d8a82e0bb53a60f16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.910501 27500 tablet_service.cc:1468] Processing CreateTablet for tablet 42778818e90a4c16a6611bfd91975923 (DEFAULT_TABLE table=default.Table_d5ec146fd09a42cca06f2e0665bc3441 [id=fa66720ea68f49a6a7c168da520e1735]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.910590 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf341e1600a14cb1893d36c435e8667f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.910776 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42778818e90a4c16a6611bfd91975923. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.912542 28567 raft_consensus.cc:491] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:34.912727 28567 raft_consensus.cc:513] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.913419 28567 leader_election.cc:290] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:34.913869 27246 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46f49fd718be4a2996d781ab31268217" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:34.914042 27693 tablet_bootstrap.cc:654] T 8ba2636641654d8596fd2a5b0644d073 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:34.914739 27239 raft_consensus.cc:1273] T 4bcd65022b7b4502983b6b58809047e3 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Refusing update from remote peer 51578fb7bceb4e119b3ab1e58c4f5528: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 20:05:34.916020 27222 tablet_service.cc:1468] Processing CreateTablet for tablet cf341e1600a14cb1893d36c435e8667f (DEFAULT_TABLE table=default.Table_cedba29d5f42445cba418e98e554bb61 [id=f87638b05ddb417d8a82e0bb53a60f16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.916465 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf341e1600a14cb1893d36c435e8667f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.916472 27184 leader_election.cc:304] T 950a6fcd35f64f2d889831fe9cdf503a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:34.914448 27321 leader_election.cc:343] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:34.916950 29196 raft_consensus.cc:3058] T 950a6fcd35f64f2d889831fe9cdf503a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:34.917511 28567 consensus_queue.cc:1035] T 4bcd65022b7b4502983b6b58809047e3 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:34.918859 29108 raft_consensus.cc:491] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:34.919291 27517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46f49fd718be4a2996d781ab31268217" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:34.919502 27517 raft_consensus.cc:2466] T 46f49fd718be4a2996d781ab31268217 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:34.919639 29196 raft_consensus.cc:2747] T 950a6fcd35f64f2d889831fe9cdf503a P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:34.919291 27508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "238f2f30377b42c2852b016018aa3245" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:34.919831 27508 raft_consensus.cc:3058] T 238f2f30377b42c2852b016018aa3245 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:34.913841 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "238f2f30377b42c2852b016018aa3245" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:34.919087 29108 raft_consensus.cc:513] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:34.923672 27321 leader_election.cc:343] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:34.926111 28710 raft_consensus.cc:491] T 4e34d6d5d68e4395aed78821ea077c1b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:34.926304 28710 raft_consensus.cc:513] T 4e34d6d5d68e4395aed78821ea077c1b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.926537 27508 raft_consensus.cc:2466] T 238f2f30377b42c2852b016018aa3245 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:34.926893 28710 leader_election.cc:290] T 4e34d6d5d68e4395aed78821ea077c1b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:34.928074 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24eadc70e10d4d02bcb643d45c9e0e36" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:34.928755 27320 leader_election.cc:304] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:34.929104 29224 raft_consensus.cc:2802] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250903 20:05:34.930433 27462 leader_election.cc:343] T 24eadc70e10d4d02bcb643d45c9e0e36 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:34.930783 29224 raft_consensus.cc:491] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:34.930979 29224 raft_consensus.cc:3058] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:34.931751 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e34d6d5d68e4395aed78821ea077c1b" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:34.932401 27462 leader_election.cc:343] T 4e34d6d5d68e4395aed78821ea077c1b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:34.933522 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b4b15d77d07c4fa7856d9c198cf15450 consensus state has no leader
I20250903 20:05:34.933718 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24eadc70e10d4d02bcb643d45c9e0e36" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:34.933701 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e34d6d5d68e4395aed78821ea077c1b" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:34.934198 27462 leader_election.cc:343] T 24eadc70e10d4d02bcb643d45c9e0e36 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:34.934338 27462 leader_election.cc:304] T 24eadc70e10d4d02bcb643d45c9e0e36 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:34.934239 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_fc0ceede2760485d94134d2b4532cb9a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:34.934726 28710 raft_consensus.cc:2747] T 24eadc70e10d4d02bcb643d45c9e0e36 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:34.935402 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc0ceede2760485d94134d2b4532cb9a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:34.935359 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "238f2f30377b42c2852b016018aa3245" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:34.935842 29108 leader_election.cc:290] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:34.935876 27065 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet fc634116a78f45dd833034ddda90e9bf (table default.Table_78e7a5b76ebc469396d8a3d670ab63ee [id=fb43441ca8f2415d99a628e81ef1642f]). Expected version 2 got 0
I20250903 20:05:34.935985 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet 238f2f30377b42c2852b016018aa3245 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:34.936694 27065 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_072fbb13945d42d88062f1b63efb5b6c" } modify_external_catalogs: true
I20250903 20:05:34.937121 27063 master_service.cc:948] ReplaceTablet: received request to replace tablet aa9f5e0964b348b7af184f60cd145666 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:34.937580 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3103e1595ec4c38ba65dca26fe1167c" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:34.938270 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1223743918814d88b401782ca4b521dd" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:34.938474 27350 tablet_service.cc:1468] Processing CreateTablet for tablet cf341e1600a14cb1893d36c435e8667f (DEFAULT_TABLE table=default.Table_cedba29d5f42445cba418e98e554bb61 [id=f87638b05ddb417d8a82e0bb53a60f16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.938709 27248 raft_consensus.cc:2466] T 1223743918814d88b401782ca4b521dd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:34.939306 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_aaa4d127435d4fea8862bda75da34909"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:34.940150 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aaa4d127435d4fea8862bda75da34909 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:34.940409 27321 leader_election.cc:343] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:34.940672 29224 raft_consensus.cc:513] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.940928 27321 leader_election.cc:304] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:34.940390 27320 leader_election.cc:304] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:34.941226 29108 raft_consensus.cc:2802] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:34.941341 29108 raft_consensus.cc:491] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:34.941431 29108 raft_consensus.cc:3058] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:34.942050 28567 raft_consensus.cc:2802] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:34.942256 28567 raft_consensus.cc:695] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:34.942546 28567 consensus_queue.cc:237] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.944669 29108 raft_consensus.cc:513] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:34.944747 27462 leader_election.cc:343] T 4e34d6d5d68e4395aed78821ea077c1b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:34.944871 27462 leader_election.cc:304] T 4e34d6d5d68e4395aed78821ea077c1b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:34.945325 29108 leader_election.cc:290] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:34.945323 27462 leader_election.cc:343] T a3103e1595ec4c38ba65dca26fe1167c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:34.945422 27462 leader_election.cc:304] T a3103e1595ec4c38ba65dca26fe1167c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:34.945662 29221 raft_consensus.cc:2747] T 4e34d6d5d68e4395aed78821ea077c1b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:34.945817 29221 raft_consensus.cc:2747] T a3103e1595ec4c38ba65dca26fe1167c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:34.945780 27065 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_072fbb13945d42d88062f1b63efb5b6c" } modify_external_catalogs: true
I20250903 20:05:34.946514 27508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1223743918814d88b401782ca4b521dd" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:34.946753 27508 raft_consensus.cc:2466] T 1223743918814d88b401782ca4b521dd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:34.946749 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1223743918814d88b401782ca4b521dd" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:34.947000 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf341e1600a14cb1893d36c435e8667f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.947014 27248 raft_consensus.cc:3058] T 1223743918814d88b401782ca4b521dd P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:34.947916 29224 leader_election.cc:290] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:34.947885 27216 tablet_service.cc:1468] Processing CreateTablet for tablet 42778818e90a4c16a6611bfd91975923 (DEFAULT_TABLE table=default.Table_d5ec146fd09a42cca06f2e0665bc3441 [id=fa66720ea68f49a6a7c168da520e1735]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.948354 27216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42778818e90a4c16a6611bfd91975923. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.949582 27517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46f49fd718be4a2996d781ab31268217" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:34.949757 27517 raft_consensus.cc:3058] T 46f49fd718be4a2996d781ab31268217 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:34.950327 27248 raft_consensus.cc:2466] T 1223743918814d88b401782ca4b521dd P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:34.952515 27321 leader_election.cc:304] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:34.952937 27508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1223743918814d88b401782ca4b521dd" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:34.953058 29224 raft_consensus.cc:2802] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:34.953158 27508 raft_consensus.cc:3058] T 1223743918814d88b401782ca4b521dd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:34.953261 29224 raft_consensus.cc:695] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:34.953172 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ca1f275bff334b12be53e0884b57dfe0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:34.953622 29224 consensus_queue.cc:237] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.955164 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46f49fd718be4a2996d781ab31268217" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:34.955404 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 860ddeb6f6784c789dbb0edd6262e1e1 (DEFAULT_TABLE table=default.Table_64a66d119ffd44619423ae75b74662ec [id=3d5b6fa599fa415d894b8b33693ea910]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.956182 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 860ddeb6f6784c789dbb0edd6262e1e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.955233 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 860ddeb6f6784c789dbb0edd6262e1e1 (DEFAULT_TABLE table=default.Table_64a66d119ffd44619423ae75b74662ec [id=3d5b6fa599fa415d894b8b33693ea910]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.958758 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 860ddeb6f6784c789dbb0edd6262e1e1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:34.956094 27321 leader_election.cc:343] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:34.962468 27050 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet a7f25f00dff7426195ac51214a14022b (table default.Table_66c6bafa9fa64867af5ff30e048122ed [id=28e973c60f364bdd9314ff5a1abbd73d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet a7f25f00dff7426195ac51214a14022b consensus state has no leader
I20250903 20:05:34.963376 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 42778818e90a4c16a6611bfd91975923 (DEFAULT_TABLE table=default.Table_d5ec146fd09a42cca06f2e0665bc3441 [id=fa66720ea68f49a6a7c168da520e1735]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.963912 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42778818e90a4c16a6611bfd91975923. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:34.964092 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca1f275bff334b12be53e0884b57dfe0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:34.964563 27685 ts_tablet_manager.cc:1905] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:34.885663 27687 ts_tablet_manager.cc:1939] T 2d9c1642fec841d497ff3496dd4564db P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:34.966675 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b664a0b055d84bc09046969e26863021" } modify_external_catalogs: true
I20250903 20:05:34.968797 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b664a0b055d84bc09046969e26863021" } modify_external_catalogs: true
I20250903 20:05:34.969262 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 860ddeb6f6784c789dbb0edd6262e1e1 (DEFAULT_TABLE table=default.Table_64a66d119ffd44619423ae75b74662ec [id=3d5b6fa599fa415d894b8b33693ea910]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.969713 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 860ddeb6f6784c789dbb0edd6262e1e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.969759 29223 consensus_queue.cc:1035] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:34.972566 27240 raft_consensus.cc:3058] T ed096b452ba747f89e91edc15e6e3ac1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:34.976502 27500 tablet_service.cc:1468] Processing CreateTablet for tablet 2f25cd0ee9ca411ba98f5df4484fa142 (DEFAULT_TABLE table=default.Table_8df248ec82924d66982d58ce170d32f6 [id=664d72e4ba974cc1858522f4b92959e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:34.976647 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 2d9c1642fec841d497ff3496dd4564db (table default.Table_63d8bb9ad5a14a7d81fb6f15d20e6ff0 [id=f5cb786b03db4395a0f40b2956f5d331]) successfully deleted
I20250903 20:05:34.977051 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f25cd0ee9ca411ba98f5df4484fa142. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:34.978376 27693 tablet_bootstrap.cc:492] T 8ba2636641654d8596fd2a5b0644d073 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:34.978567 27693 ts_tablet_manager.cc:1397] T 8ba2636641654d8596fd2a5b0644d073 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.072s user 0.005s sys 0.005s
I20250903 20:05:34.979387 27693 raft_consensus.cc:357] T 8ba2636641654d8596fd2a5b0644d073 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.979589 27693 raft_consensus.cc:383] T 8ba2636641654d8596fd2a5b0644d073 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:34.979660 27693 raft_consensus.cc:738] T 8ba2636641654d8596fd2a5b0644d073 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:34.984436 27695 tablet_bootstrap.cc:492] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
W20250903 20:05:34.985257 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 22bc87d1a4804b6c80f77f9692500d63 because the tablet was not found. No further retry: Not found: Tablet not found: 22bc87d1a4804b6c80f77f9692500d63
I20250903 20:05:34.986048 27695 ts_tablet_manager.cc:1397] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.313s user 0.009s sys 0.009s
I20250903 20:05:34.986869 27695 raft_consensus.cc:357] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.987087 27695 raft_consensus.cc:383] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:34.987164 27695 raft_consensus.cc:738] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:34.987430 27695 consensus_queue.cc:260] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.989790 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_04c1db2796be4ffd90e43a7e2e29194f" } modify_external_catalogs: true
I20250903 20:05:34.990077 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_04c1db2796be4ffd90e43a7e2e29194f" } modify_external_catalogs: true
I20250903 20:05:34.993294 27517 raft_consensus.cc:2466] T 46f49fd718be4a2996d781ab31268217 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:34.993793 27320 leader_election.cc:304] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
W20250903 20:05:34.994539 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1453a8f6bd7b4b26b75d7fc08578a351 consensus state has no leader
I20250903 20:05:34.994532 28916 raft_consensus.cc:2802] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:34.994675 28916 raft_consensus.cc:695] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:34.994973 28916 consensus_queue.cc:237] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.992125 27693 consensus_queue.cc:260] T 8ba2636641654d8596fd2a5b0644d073 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:34.996569 27693 ts_tablet_manager.cc:1428] T 8ba2636641654d8596fd2a5b0644d073 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20250903 20:05:34.996564 27695 ts_tablet_manager.cc:1428] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250903 20:05:34.996989 27695 tablet_bootstrap.cc:492] T 46f49fd718be4a2996d781ab31268217 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:35.000756 29224 raft_consensus.cc:491] T 8ba2636641654d8596fd2a5b0644d073 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:35.001036 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 2f25cd0ee9ca411ba98f5df4484fa142 (DEFAULT_TABLE table=default.Table_8df248ec82924d66982d58ce170d32f6 [id=664d72e4ba974cc1858522f4b92959e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.001065 29224 raft_consensus.cc:513] T 8ba2636641654d8596fd2a5b0644d073 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.001621 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f25cd0ee9ca411ba98f5df4484fa142. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.002550 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 5bdac444ce53424881623186f79757fb (DEFAULT_TABLE table=default.Table_3a10826c979f434fa1b1a6a3dbae518b [id=bf6a2e485a684262aa3d2c3151d26db3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.002651 27493 tablet_service.cc:1468] Processing CreateTablet for tablet b24dbb492f114fd8834d2773a80ec14e (DEFAULT_TABLE table=default.Table_6225fbd3b8554129a7d622f5bd325fc9 [id=a50b93914b1d40e0b08259ac377f9192]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.003101 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b24dbb492f114fd8834d2773a80ec14e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.003114 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bdac444ce53424881623186f79757fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.005843 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 5bdac444ce53424881623186f79757fb (DEFAULT_TABLE table=default.Table_3a10826c979f434fa1b1a6a3dbae518b [id=bf6a2e485a684262aa3d2c3151d26db3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.006387 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bdac444ce53424881623186f79757fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.007954 27349 tablet_service.cc:1468] Processing CreateTablet for tablet 7be2526b81554322893aa9de732205ad (DEFAULT_TABLE table=default.Table_53b5ec8ef7854180bc6c9d983cbf216d [id=af1351a57fb94c5cac9bef57a30fe9c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.008121 27347 tablet_service.cc:1468] Processing CreateTablet for tablet 8e48dded9483492c86ab9265acc6ef30 (DEFAULT_TABLE table=default.Table_b99ab7e3978a411cb629b6a3d0f4ccd9 [id=b04efcece14f4cd6a6aa4e56c6f4d6f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.008402 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7be2526b81554322893aa9de732205ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.000880 27687 tablet_replica.cc:331] T 45f7068bafaf48d290ded9d1272c0e46 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:35.008484 27359 tablet_service.cc:1468] Processing CreateTablet for tablet b24dbb492f114fd8834d2773a80ec14e (DEFAULT_TABLE table=default.Table_6225fbd3b8554129a7d622f5bd325fc9 [id=a50b93914b1d40e0b08259ac377f9192]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.008709 27687 raft_consensus.cc:2241] T 45f7068bafaf48d290ded9d1272c0e46 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:35.008841 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b24dbb492f114fd8834d2773a80ec14e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.008944 27687 raft_consensus.cc:2270] T 45f7068bafaf48d290ded9d1272c0e46 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:35.009624 27687 ts_tablet_manager.cc:1905] T 45f7068bafaf48d290ded9d1272c0e46 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:35.013288 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e48dded9483492c86ab9265acc6ef30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.014602 29200 tablet.cc:1722] T 4bcd65022b7b4502983b6b58809047e3 P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:35.015250 27062 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 4bcd65022b7b4502983b6b58809047e3 (table default.Table_df7e7287722a45988929dfd3f0ff3eb6 [id=7ee5ea1a352941d7ba044bf48f21848a]). Expected version 2 got 0
I20250903 20:05:35.022351 27221 tablet_service.cc:1468] Processing CreateTablet for tablet 7be2526b81554322893aa9de732205ad (DEFAULT_TABLE table=default.Table_53b5ec8ef7854180bc6c9d983cbf216d [id=af1351a57fb94c5cac9bef57a30fe9c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.022859 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7be2526b81554322893aa9de732205ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.024735 29196 consensus_queue.cc:1035] T 7a9e55f0811345f7afa71645f38ebdaf P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:35.026988 27216 tablet_service.cc:1468] Processing CreateTablet for tablet b24dbb492f114fd8834d2773a80ec14e (DEFAULT_TABLE table=default.Table_6225fbd3b8554129a7d622f5bd325fc9 [id=a50b93914b1d40e0b08259ac377f9192]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.027784 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 8e48dded9483492c86ab9265acc6ef30 (DEFAULT_TABLE table=default.Table_b99ab7e3978a411cb629b6a3d0f4ccd9 [id=b04efcece14f4cd6a6aa4e56c6f4d6f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.029036 27216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b24dbb492f114fd8834d2773a80ec14e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.029402 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e48dded9483492c86ab9265acc6ef30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.030263 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 8e48dded9483492c86ab9265acc6ef30 (DEFAULT_TABLE table=default.Table_b99ab7e3978a411cb629b6a3d0f4ccd9 [id=b04efcece14f4cd6a6aa4e56c6f4d6f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.030349 27498 tablet_service.cc:1468] Processing CreateTablet for tablet 7be2526b81554322893aa9de732205ad (DEFAULT_TABLE table=default.Table_53b5ec8ef7854180bc6c9d983cbf216d [id=af1351a57fb94c5cac9bef57a30fe9c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.030674 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e48dded9483492c86ab9265acc6ef30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.030934 27498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7be2526b81554322893aa9de732205ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.031558 27508 raft_consensus.cc:2466] T 1223743918814d88b401782ca4b521dd P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:35.032799 29227 consensus_queue.cc:1035] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:35.037978 27496 tablet_service.cc:1468] Processing CreateTablet for tablet 7c200f2becd84e3482e528191cbc2868 (DEFAULT_TABLE table=default.Table_fd55f02a3038460fb305e900be86ef28 [id=52a15cfac52043afb5e3bbdf0fb01c1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.038486 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c200f2becd84e3482e528191cbc2868. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:35.044030 27183 consensus_peers.cc:489] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:35.045395 29245 tablet.cc:1722] T 4bcd65022b7b4502983b6b58809047e3 P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:35.051378 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_2f4fdec99f854b22bc5262d859311c48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:35.052346 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f4fdec99f854b22bc5262d859311c48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:35.018505 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_857612f25ca54f8b8d0ea31485d96a5d" } modify_external_catalogs: true
I20250903 20:05:35.054519 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_857612f25ca54f8b8d0ea31485d96a5d" } modify_external_catalogs: true
I20250903 20:05:35.055059 29224 leader_election.cc:290] T 8ba2636641654d8596fd2a5b0644d073 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:35.056531 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 2f25cd0ee9ca411ba98f5df4484fa142 (DEFAULT_TABLE table=default.Table_8df248ec82924d66982d58ce170d32f6 [id=664d72e4ba974cc1858522f4b92959e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.056814 28710 raft_consensus.cc:491] T f326ec776ba446be8d5dda5c1dc5b6ec P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:35.056951 28710 raft_consensus.cc:513] T f326ec776ba446be8d5dda5c1dc5b6ec P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:35.057248 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f25cd0ee9ca411ba98f5df4484fa142. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.057544 28710 leader_election.cc:290] T f326ec776ba446be8d5dda5c1dc5b6ec P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:35.058075 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f326ec776ba446be8d5dda5c1dc5b6ec" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:35.058338 27508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ba2636641654d8596fd2a5b0644d073" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:35.058781 27462 leader_election.cc:343] T f326ec776ba446be8d5dda5c1dc5b6ec P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:35.059034 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 5bdac444ce53424881623186f79757fb (DEFAULT_TABLE table=default.Table_3a10826c979f434fa1b1a6a3dbae518b [id=bf6a2e485a684262aa3d2c3151d26db3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.059474 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bdac444ce53424881623186f79757fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.064615 27692 tablet_bootstrap.cc:492] T 8ba2636641654d8596fd2a5b0644d073 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:35.064843 27692 ts_tablet_manager.cc:1397] T 8ba2636641654d8596fd2a5b0644d073 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.186s user 0.002s sys 0.003s
I20250903 20:05:35.065101 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_d6d3ee9322e745f6843403131aa0f8a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:35.065649 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f326ec776ba446be8d5dda5c1dc5b6ec" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:35.065757 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6d3ee9322e745f6843403131aa0f8a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:35.066887 27692 raft_consensus.cc:357] T 8ba2636641654d8596fd2a5b0644d073 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.067051 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ba2636641654d8596fd2a5b0644d073" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:35.067134 27692 raft_consensus.cc:383] T 8ba2636641654d8596fd2a5b0644d073 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.067193 27692 raft_consensus.cc:738] T 8ba2636641654d8596fd2a5b0644d073 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.067263 28808 consensus_queue.cc:1035] T 7a9e55f0811345f7afa71645f38ebdaf P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:35.067404 27692 consensus_queue.cc:260] T 8ba2636641654d8596fd2a5b0644d073 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:35.067631 27321 leader_election.cc:343] T 8ba2636641654d8596fd2a5b0644d073 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:35.068014 27692 ts_tablet_manager.cc:1428] T 8ba2636641654d8596fd2a5b0644d073 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:35.068282 27692 tablet_bootstrap.cc:492] T aa9f5e0964b348b7af184f60cd145666 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:35.070346 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 45f7068bafaf48d290ded9d1272c0e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:35.071441 27462 leader_election.cc:343] T f326ec776ba446be8d5dda5c1dc5b6ec P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:35.071620 27462 leader_election.cc:304] T f326ec776ba446be8d5dda5c1dc5b6ec P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:35.071828 28710 raft_consensus.cc:2747] T f326ec776ba446be8d5dda5c1dc5b6ec P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:35.074424 27320 leader_election.cc:343] T 8ba2636641654d8596fd2a5b0644d073 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:35.075783 27695 tablet_bootstrap.cc:654] T 46f49fd718be4a2996d781ab31268217 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.076092 27320 leader_election.cc:304] T 8ba2636641654d8596fd2a5b0644d073 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:35.077121 29224 raft_consensus.cc:2747] T 8ba2636641654d8596fd2a5b0644d073 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:35.084270 27693 tablet_bootstrap.cc:492] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:35.085441 27057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_182ace2fdd7f4619b244f455e93d4df9" } new_table_name: "default.Table_2908c6a1f95648b1b4c111d123968384" modify_external_catalogs: true
I20250903 20:05:35.086268 27692 tablet_bootstrap.cc:654] T aa9f5e0964b348b7af184f60cd145666 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.088611 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_5bcfed1071c9400ca66b8e9b8b2f26cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:35.089275 28710 raft_consensus.cc:491] T 8ba2636641654d8596fd2a5b0644d073 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:35.089423 28710 raft_consensus.cc:513] T 8ba2636641654d8596fd2a5b0644d073 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.089970 28710 leader_election.cc:290] T 8ba2636641654d8596fd2a5b0644d073 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:35.090620 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bcfed1071c9400ca66b8e9b8b2f26cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:35.091517 29202 tablet.cc:1722] T 4bcd65022b7b4502983b6b58809047e3 P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:35.102020 27695 tablet_bootstrap.cc:492] T 46f49fd718be4a2996d781ab31268217 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:35.102294 27695 ts_tablet_manager.cc:1397] T 46f49fd718be4a2996d781ab31268217 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.105s user 0.004s sys 0.001s
I20250903 20:05:35.103224 27695 raft_consensus.cc:357] T 46f49fd718be4a2996d781ab31268217 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.103431 27695 raft_consensus.cc:383] T 46f49fd718be4a2996d781ab31268217 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.103499 27695 raft_consensus.cc:738] T 46f49fd718be4a2996d781ab31268217 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.103730 27695 consensus_queue.cc:260] T 46f49fd718be4a2996d781ab31268217 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.104277 27695 ts_tablet_manager.cc:1428] T 46f49fd718be4a2996d781ab31268217 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:35.105196 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ba2636641654d8596fd2a5b0644d073" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:35.108760 27379 raft_consensus.cc:2466] T 8ba2636641654d8596fd2a5b0644d073 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:35.099943 27693 tablet_bootstrap.cc:654] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.109619 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 7c200f2becd84e3482e528191cbc2868 (DEFAULT_TABLE table=default.Table_fd55f02a3038460fb305e900be86ef28 [id=52a15cfac52043afb5e3bbdf0fb01c1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.110160 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c200f2becd84e3482e528191cbc2868. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.111173 27462 leader_election.cc:304] T 8ba2636641654d8596fd2a5b0644d073 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:35.111552 28710 raft_consensus.cc:2802] T 8ba2636641654d8596fd2a5b0644d073 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:35.111632 28710 raft_consensus.cc:491] T 8ba2636641654d8596fd2a5b0644d073 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:35.111701 28710 raft_consensus.cc:3058] T 8ba2636641654d8596fd2a5b0644d073 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:35.114403 29108 raft_consensus.cc:491] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:35.114563 29108 raft_consensus.cc:513] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.115211 29108 leader_election.cc:290] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:35.116304 27692 tablet_bootstrap.cc:492] T aa9f5e0964b348b7af184f60cd145666 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:35.116254 28710 raft_consensus.cc:513] T 8ba2636641654d8596fd2a5b0644d073 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.116478 27692 ts_tablet_manager.cc:1397] T aa9f5e0964b348b7af184f60cd145666 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.001s
I20250903 20:05:35.117106 28710 leader_election.cc:290] T 8ba2636641654d8596fd2a5b0644d073 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:35.117688 27692 raft_consensus.cc:357] T aa9f5e0964b348b7af184f60cd145666 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.117862 27692 raft_consensus.cc:383] T aa9f5e0964b348b7af184f60cd145666 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.117925 27692 raft_consensus.cc:738] T aa9f5e0964b348b7af184f60cd145666 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.118168 27692 consensus_queue.cc:260] T aa9f5e0964b348b7af184f60cd145666 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.118808 27692 ts_tablet_manager.cc:1428] T aa9f5e0964b348b7af184f60cd145666 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:35.118687 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 7c200f2becd84e3482e528191cbc2868 (DEFAULT_TABLE table=default.Table_fd55f02a3038460fb305e900be86ef28 [id=52a15cfac52043afb5e3bbdf0fb01c1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.119124 27692 tablet_bootstrap.cc:492] T 47c1287bff1145a5b2ab6d14dbd608c7 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:35.120970 27221 tablet_service.cc:1468] Processing CreateTablet for tablet 515db6402ced455ea4bb9a0d2e89382f (DEFAULT_TABLE table=default.Table_026e7093a60a495e95489df73d689967 [id=772809879a0048a29129ed25c5e178ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.121177 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5557dfdd953a41e9bd3a8be4cd689604" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:35.121380 27516 raft_consensus.cc:2466] T 5557dfdd953a41e9bd3a8be4cd689604 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:35.122637 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 20d826f1b1d64746bd8e409a12b74eb1
I20250903 20:05:35.124629 27059 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 7345305820be4d32a3ceb227f045f009 (table default.Table_5f0199a660844c15a254a062c1540498 [id=52f969e50b9d466cbd14de6f252b7706]). Expected version 1 got 0
I20250903 20:05:35.124902 27059 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 4bcd65022b7b4502983b6b58809047e3 (table default.Table_df7e7287722a45988929dfd3f0ff3eb6 [id=7ee5ea1a352941d7ba044bf48f21848a]). Expected version 2 got 0
I20250903 20:05:35.126652 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5557dfdd953a41e9bd3a8be4cd689604" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:35.127278 27321 leader_election.cc:343] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:35.127358 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ba2636641654d8596fd2a5b0644d073" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:35.127453 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ba2636641654d8596fd2a5b0644d073" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
W20250903 20:05:35.127847 27462 leader_election.cc:343] T 8ba2636641654d8596fd2a5b0644d073 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:35.128090 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_cbc1343da06443418cb594f7720331d8" } new_table_name: "default.Table_ca778ec138604800beff1588b8a15ba4" modify_external_catalogs: true
W20250903 20:05:35.128291 27462 leader_election.cc:343] T 8ba2636641654d8596fd2a5b0644d073 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:35.128608 27065 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 7a9e55f0811345f7afa71645f38ebdaf (table default.Table_adf05e59a7be478b89666c3162ac83f1 [id=e69321e60a3a4e71a463a0011c0c6d65]). Expected version 2 got 0
I20250903 20:05:35.128907 27065 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 8ba2636641654d8596fd2a5b0644d073 (table default.Table_e7148e16606e4cd3908c86b439f75b78 [id=4bffa29ea98049718ac3ce35de9813c5]). Expected version 1 got 0
I20250903 20:05:35.129415 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 515db6402ced455ea4bb9a0d2e89382f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.130115 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet 20d826f1b1d64746bd8e409a12b74eb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:35.130872 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c200f2becd84e3482e528191cbc2868. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.131675 28808 consensus_queue.cc:1035] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:35.137588 27500 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e34d6d5d68e4395aed78821ea077c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:35.138448 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ba2636641654d8596fd2a5b0644d073" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:35.140468 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 20d826f1b1d64746bd8e409a12b74eb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:35.140972 27695 tablet_bootstrap.cc:492] T 472f0fb01263468a828ec13795656624 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:35.142042 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 515db6402ced455ea4bb9a0d2e89382f (DEFAULT_TABLE table=default.Table_026e7093a60a495e95489df73d689967 [id=772809879a0048a29129ed25c5e178ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.142570 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 515db6402ced455ea4bb9a0d2e89382f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.143653 27695 tablet_bootstrap.cc:654] T 472f0fb01263468a828ec13795656624 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.138878 27379 raft_consensus.cc:3058] T 8ba2636641654d8596fd2a5b0644d073 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:35.145638 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet 20d826f1b1d64746bd8e409a12b74eb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:35.148345 27379 raft_consensus.cc:2466] T 8ba2636641654d8596fd2a5b0644d073 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:35.148656 28808 consensus_queue.cc:1035] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:35.151816 27462 leader_election.cc:304] T 8ba2636641654d8596fd2a5b0644d073 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:35.152374 28710 raft_consensus.cc:2802] T 8ba2636641654d8596fd2a5b0644d073 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:35.152729 28710 raft_consensus.cc:695] T 8ba2636641654d8596fd2a5b0644d073 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:35.153030 28710 consensus_queue.cc:237] T 8ba2636641654d8596fd2a5b0644d073 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.154531 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 515db6402ced455ea4bb9a0d2e89382f (DEFAULT_TABLE table=default.Table_026e7093a60a495e95489df73d689967 [id=772809879a0048a29129ed25c5e178ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.154963 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 515db6402ced455ea4bb9a0d2e89382f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.158433 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet f28fd3e1e0e14cf48a244580b4f355d0
I20250903 20:05:35.158790 27692 tablet_bootstrap.cc:654] T 47c1287bff1145a5b2ab6d14dbd608c7 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.157140 27320 leader_election.cc:304] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:35.160079 28567 raft_consensus.cc:2802] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:35.160176 28567 raft_consensus.cc:491] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:35.160250 28567 raft_consensus.cc:3058] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:35.161123 27350 tablet_service.cc:1515] Processing DeleteTablet for tablet f28fd3e1e0e14cf48a244580b4f355d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:35.162092 27216 tablet_service.cc:1515] Processing DeleteTablet for tablet f28fd3e1e0e14cf48a244580b4f355d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:35.163784 27500 tablet_service.cc:1515] Processing DeleteTablet for tablet f28fd3e1e0e14cf48a244580b4f355d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:35.164752 28567 raft_consensus.cc:513] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.166695 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5557dfdd953a41e9bd3a8be4cd689604" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:35.167135 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_99db6c94d5964e95921925a5bcbc7f0e" } modify_external_catalogs: true
W20250903 20:05:35.167292 27321 leader_election.cc:343] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:35.167349 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_99db6c94d5964e95921925a5bcbc7f0e" } modify_external_catalogs: true
I20250903 20:05:35.168291 27509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5557dfdd953a41e9bd3a8be4cd689604" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:35.168558 27509 raft_consensus.cc:3058] T 5557dfdd953a41e9bd3a8be4cd689604 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:35.171546 27509 raft_consensus.cc:2466] T 5557dfdd953a41e9bd3a8be4cd689604 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:35.173592 27320 leader_election.cc:304] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:35.173975 28916 raft_consensus.cc:2802] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:35.174305 28916 raft_consensus.cc:695] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:35.174885 28916 consensus_queue.cc:237] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.175478 27692 tablet_bootstrap.cc:492] T 47c1287bff1145a5b2ab6d14dbd608c7 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:35.175632 27692 ts_tablet_manager.cc:1397] T 47c1287bff1145a5b2ab6d14dbd608c7 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.001s
I20250903 20:05:35.176463 27692 raft_consensus.cc:357] T 47c1287bff1145a5b2ab6d14dbd608c7 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:35.176644 27692 raft_consensus.cc:383] T 47c1287bff1145a5b2ab6d14dbd608c7 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.176708 27692 raft_consensus.cc:738] T 47c1287bff1145a5b2ab6d14dbd608c7 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.185011 27692 consensus_queue.cc:260] T 47c1287bff1145a5b2ab6d14dbd608c7 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:35.186714 27692 ts_tablet_manager.cc:1428] T 47c1287bff1145a5b2ab6d14dbd608c7 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.011s user 0.002s sys 0.001s
I20250903 20:05:35.195436 27685 ts_tablet_manager.cc:1918] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:35.195762 27685 log.cc:1199] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/726f4d920f3e47fea41b4e12c021c988
I20250903 20:05:35.196458 27685 ts_tablet_manager.cc:1939] T 726f4d920f3e47fea41b4e12c021c988 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:35.207101 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 69f8eef52bee431a86f0eef45fd11f9f
I20250903 20:05:35.207607 27693 tablet_bootstrap.cc:492] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:35.207733 27693 ts_tablet_manager.cc:1397] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.124s user 0.004s sys 0.003s
I20250903 20:05:35.207952 27500 tablet_service.cc:1515] Processing DeleteTablet for tablet 69f8eef52bee431a86f0eef45fd11f9f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:35.208446 27693 raft_consensus.cc:357] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.208598 27693 raft_consensus.cc:383] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.208639 27693 raft_consensus.cc:738] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.208809 27693 consensus_queue.cc:260] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.209372 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6135232af1594805bd74e6219011444a" } new_table_name: "default.Table_cafe58600eb4444089168f96fd5cf593" modify_external_catalogs: true
I20250903 20:05:35.216122 28710 consensus_queue.cc:1035] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:35.220719 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet 69f8eef52bee431a86f0eef45fd11f9f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:35.230886 28567 leader_election.cc:290] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:35.231702 27692 tablet_bootstrap.cc:492] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:35.233413 27692 tablet_bootstrap.cc:654] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.207149 27695 tablet_bootstrap.cc:492] T 472f0fb01263468a828ec13795656624 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:35.234282 27695 ts_tablet_manager.cc:1397] T 472f0fb01263468a828ec13795656624 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.093s user 0.006s sys 0.001s
I20250903 20:05:35.235116 27695 raft_consensus.cc:357] T 472f0fb01263468a828ec13795656624 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.235306 27695 raft_consensus.cc:383] T 472f0fb01263468a828ec13795656624 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.235374 27695 raft_consensus.cc:738] T 472f0fb01263468a828ec13795656624 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.235575 27695 consensus_queue.cc:260] T 472f0fb01263468a828ec13795656624 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.236116 27695 ts_tablet_manager.cc:1428] T 472f0fb01263468a828ec13795656624 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:35.225682 28808 consensus_queue.cc:1035] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:35.230377 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet 69f8eef52bee431a86f0eef45fd11f9f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:35.241508 27184 consensus_peers.cc:489] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d9c1642fec841d497ff3496dd4564db. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:35.191102 27687 ts_tablet_manager.cc:1918] T 45f7068bafaf48d290ded9d1272c0e46 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:35.242110 27687 log.cc:1199] T 45f7068bafaf48d290ded9d1272c0e46 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/45f7068bafaf48d290ded9d1272c0e46
I20250903 20:05:35.243516 27687 ts_tablet_manager.cc:1939] T 45f7068bafaf48d290ded9d1272c0e46 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:35.246311 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7a8637d7aab94246a883a6e29273e755" } new_table_name: "default.Table_cee5c8511c6146ab9915a15c413154cf" modify_external_catalogs: true
I20250903 20:05:35.246883 28710 consensus_queue.cc:1035] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:35.248652 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 45f7068bafaf48d290ded9d1272c0e46 (table default.Table_7179a3a012834e34b703156b9c834722 [id=e660a48b356a4b0784016e527303f1f6]) successfully deleted
I20250903 20:05:35.249007 27693 ts_tablet_manager.cc:1428] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.041s user 0.001s sys 0.001s
I20250903 20:05:35.250244 27692 tablet_bootstrap.cc:492] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:35.250404 27692 ts_tablet_manager.cc:1397] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.005s
I20250903 20:05:35.250962 27693 tablet_bootstrap.cc:492] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:35.251147 27692 raft_consensus.cc:357] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:35.251330 27692 raft_consensus.cc:383] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.251387 27692 raft_consensus.cc:738] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.251579 27692 consensus_queue.cc:260] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:35.252058 27692 ts_tablet_manager.cc:1428] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:35.252326 27692 tablet_bootstrap.cc:492] T 64c4dd2645034a7eaf243f5474603f10 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:35.253723 27692 tablet_bootstrap.cc:654] T 64c4dd2645034a7eaf243f5474603f10 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.254675 27693 tablet_bootstrap.cc:654] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.256299 27687 tablet_replica.cc:331] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:35.256481 27687 raft_consensus.cc:2241] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:35.256774 27687 raft_consensus.cc:2270] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:35.263733 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 8fff36d5a80e45af8a256c3908555e08 because the tablet was not found. No further retry: Not found: Tablet not found: 8fff36d5a80e45af8a256c3908555e08
I20250903 20:05:35.266832 27687 ts_tablet_manager.cc:1905] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:35.269194 27692 tablet_bootstrap.cc:492] T 64c4dd2645034a7eaf243f5474603f10 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:35.269353 27692 ts_tablet_manager.cc:1397] T 64c4dd2645034a7eaf243f5474603f10 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20250903 20:05:35.269654 27684 ts_tablet_manager.cc:1939] T 2d9c1642fec841d497ff3496dd4564db P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:35.269829 27695 tablet_bootstrap.cc:492] T 238f2f30377b42c2852b016018aa3245 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:35.269865 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 958) took 3186 ms. Trace:
I20250903 20:05:35.269999 27685 rpcz_store.cc:276] 0903 20:05:32.083032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.083089 (+ 57us) service_pool.cc:225] Handling call
0903 20:05:34.864413 (+2781324us) ts_tablet_manager.cc:1145] Deleting tablet 726f4d920f3e47fea41b4e12c021c988
0903 20:05:34.864423 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:35.057711 (+193288us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.195273 (+137562us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.269843 (+ 74570us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2781066}
I20250903 20:05:35.271924 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet eaa11d17303142ff935705c16f0b9690
I20250903 20:05:35.273113 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_4e126e6eaed846d186337d5a589023b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:35.274049 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e126e6eaed846d186337d5a589023b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:35.275707 27057 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ed23e706f0b48818184067b9d58d97e consensus state has no leader
I20250903 20:05:35.271378 27695 tablet_bootstrap.cc:654] T 238f2f30377b42c2852b016018aa3245 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.276077 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_8c7c15fd52a0453ca91d7fef1fed6adb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:35.276156 27685 tablet_replica.cc:331] T 120ca61b3da742dc9a2efa483ead7e97 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:35.276409 27685 raft_consensus.cc:2241] T 120ca61b3da742dc9a2efa483ead7e97 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
W20250903 20:05:35.276563 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c7c15fd52a0453ca91d7fef1fed6adb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:35.276585 27685 raft_consensus.cc:2270] T 120ca61b3da742dc9a2efa483ead7e97 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:35.277938 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_0da8589e73bd4ad693cb4d5695b755c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:35.278328 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0da8589e73bd4ad693cb4d5695b755c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:35.278504 27692 raft_consensus.cc:357] T 64c4dd2645034a7eaf243f5474603f10 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.278687 27692 raft_consensus.cc:383] T 64c4dd2645034a7eaf243f5474603f10 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.278749 27692 raft_consensus.cc:738] T 64c4dd2645034a7eaf243f5474603f10 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.278985 27692 consensus_queue.cc:260] T 64c4dd2645034a7eaf243f5474603f10 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.279485 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 726f4d920f3e47fea41b4e12c021c988 (table default.Table_a6608fc0162446a480d8e8ecb07db28f [id=66b42d6c99de4096998cba421c5b4663]) successfully deleted
I20250903 20:05:35.279556 27692 ts_tablet_manager.cc:1428] T 64c4dd2645034a7eaf243f5474603f10 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20250903 20:05:35.279886 27692 tablet_bootstrap.cc:492] T de20d18d7f264811a0631aa1144aef2c P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:35.283619 27062 catalog_manager.cc:5582] T 8ba2636641654d8596fd2a5b0644d073 P e72ef1dbcc5143cf9f04a808f101c80f reported cstate change: term changed from 0 to 1, leader changed from <none> to e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67). New cstate: current_term: 1 leader_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:35.284198 27062 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 8ba2636641654d8596fd2a5b0644d073 (table default.Table_e7148e16606e4cd3908c86b439f75b78 [id=4bffa29ea98049718ac3ce35de9813c5]). Expected version 1 got 0
I20250903 20:05:35.284525 27062 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 7a9e55f0811345f7afa71645f38ebdaf (table default.Table_adf05e59a7be478b89666c3162ac83f1 [id=e69321e60a3a4e71a463a0011c0c6d65]). Expected version 2 got 0
I20250903 20:05:35.285503 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1080) took 1194 ms. Trace:
I20250903 20:05:35.285763 27684 rpcz_store.cc:276] 0903 20:05:34.091377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.102281 (+ 10904us) service_pool.cc:225] Handling call
0903 20:05:34.196089 (+ 93808us) ts_tablet_manager.cc:1145] Deleting tablet 2d9c1642fec841d497ff3496dd4564db
0903 20:05:34.196096 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:34.292670 (+ 96574us) spinlock_profiling.cc:243] Waited 95.9 ms on lock 0x6160000f7c54. stack: 00007f71bc6f7dc6 00007f71bc6f7b80 00007f71bb6df3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f71c8181600 00007f71c9494b13 00007f71cbf772b7 00007f71cbfa9344 00007f71cbf8ada0 00007f71cbf8ab2d 0000000000a774dc 00007f71bc73e14f 00007f71bc74141b 00007f71bc741298
0903 20:05:34.565044 (+272374us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:34.601266 (+ 36222us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.285468 (+684202us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5833,"spinlock_wait_cycles":220680576,"tablet-delete.queue_time_us":93548}
I20250903 20:05:35.286355 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet eaa11d17303142ff935705c16f0b9690 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:35 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:35.286589 27684 tablet_replica.cc:331] T ed096b452ba747f89e91edc15e6e3ac1 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:35.286823 27684 raft_consensus.cc:2241] T ed096b452ba747f89e91edc15e6e3ac1 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:35.286885 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 2d9c1642fec841d497ff3496dd4564db (table default.Table_63d8bb9ad5a14a7d81fb6f15d20e6ff0 [id=f5cb786b03db4395a0f40b2956f5d331]) successfully deleted
I20250903 20:05:35.287031 27684 raft_consensus.cc:2270] T ed096b452ba747f89e91edc15e6e3ac1 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:35.285534 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet eaa11d17303142ff935705c16f0b9690 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:35 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:35.288321 27685 ts_tablet_manager.cc:1905] T 120ca61b3da742dc9a2efa483ead7e97 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:35.292994 27695 tablet_bootstrap.cc:492] T 238f2f30377b42c2852b016018aa3245 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:35.293162 27695 ts_tablet_manager.cc:1397] T 238f2f30377b42c2852b016018aa3245 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.001s
I20250903 20:05:35.293926 27695 raft_consensus.cc:357] T 238f2f30377b42c2852b016018aa3245 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.294113 27695 raft_consensus.cc:383] T 238f2f30377b42c2852b016018aa3245 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.294164 27695 raft_consensus.cc:738] T 238f2f30377b42c2852b016018aa3245 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.294411 27695 consensus_queue.cc:260] T 238f2f30377b42c2852b016018aa3245 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.295387 27695 ts_tablet_manager.cc:1428] T 238f2f30377b42c2852b016018aa3245 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:35.296103 27695 tablet_bootstrap.cc:492] T 5557dfdd953a41e9bd3a8be4cd689604 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:35.297714 27695 tablet_bootstrap.cc:654] T 5557dfdd953a41e9bd3a8be4cd689604 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.300163 27687 ts_tablet_manager.cc:1918] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:35.300468 29199 raft_consensus.cc:491] T 238f2f30377b42c2852b016018aa3245 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:35.300664 29199 raft_consensus.cc:513] T 238f2f30377b42c2852b016018aa3245 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.301919 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "238f2f30377b42c2852b016018aa3245" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:35.302196 27516 raft_consensus.cc:2391] T 238f2f30377b42c2852b016018aa3245 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89f8f0ee68b940db9ce250c0604f934a in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
I20250903 20:05:35.302845 29199 leader_election.cc:290] T 238f2f30377b42c2852b016018aa3245 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:35.306526 27692 tablet_bootstrap.cc:654] T de20d18d7f264811a0631aa1144aef2c P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.309518 27687 log.cc:1199] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/5002b7e8118c49b6a247cb8dc4ca1aea
W20250903 20:05:35.309791 27059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a9adbac18a97487b93009784023348ad consensus state has no leader
I20250903 20:05:35.310276 27687 ts_tablet_manager.cc:1939] T 5002b7e8118c49b6a247cb8dc4ca1aea P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:35.310287 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_19e55aadaad1461e81d6b242dfa41c39" } new_table_name: "default.Table_6d5082ee3fd94074a33157509c0b26c5" modify_external_catalogs: true
I20250903 20:05:35.311452 27687 tablet_replica.cc:331] T 200c06f296b94eab94f35f312c5a443b P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:35.311548 27065 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_df7e7287722a45988929dfd3f0ff3eb6 [id=7ee5ea1a352941d7ba044bf48f21848a] alter complete (version 2)
I20250903 20:05:35.311690 27687 raft_consensus.cc:2241] T 200c06f296b94eab94f35f312c5a443b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:35.311838 27687 raft_consensus.cc:2270] T 200c06f296b94eab94f35f312c5a443b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:35.312345 27687 ts_tablet_manager.cc:1905] T 200c06f296b94eab94f35f312c5a443b P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:35.313097 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_584007f8f96047bfa933decba8b5d048"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:35.313819 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_584007f8f96047bfa933decba8b5d048 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:35.314525 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet eb878263eda9447e870a782907fe6a8d
I20250903 20:05:35.315265 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_23373e831f62480fa62bf3dfe3454bef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:35.315759 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23373e831f62480fa62bf3dfe3454bef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:35.302381 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "238f2f30377b42c2852b016018aa3245" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:35.318096 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1fce05bd4e6c425e952001ffdc25c78a consensus state has no leader
I20250903 20:05:35.319366 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 5002b7e8118c49b6a247cb8dc4ca1aea (table default.Table_b20633c3b14e463aac0b7bd917c3ab9b [id=c6a148f02792496e8d8f2bb2dfc5e69b]) successfully deleted
W20250903 20:05:35.320098 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3f43480c4f2042e18772e7f0d31dd676 consensus state has no leader
I20250903 20:05:35.321632 27693 tablet_bootstrap.cc:492] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:35.321697 27695 tablet_bootstrap.cc:492] T 5557dfdd953a41e9bd3a8be4cd689604 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:35.321832 27695 ts_tablet_manager.cc:1397] T 5557dfdd953a41e9bd3a8be4cd689604 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20250903 20:05:35.321870 27693 ts_tablet_manager.cc:1397] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.071s user 0.003s sys 0.000s
I20250903 20:05:35.322669 27695 raft_consensus.cc:357] T 5557dfdd953a41e9bd3a8be4cd689604 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.322865 27695 raft_consensus.cc:383] T 5557dfdd953a41e9bd3a8be4cd689604 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.322917 27695 raft_consensus.cc:738] T 5557dfdd953a41e9bd3a8be4cd689604 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.322726 27693 raft_consensus.cc:357] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.323066 27693 raft_consensus.cc:383] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.323163 27693 raft_consensus.cc:738] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.323190 27695 consensus_queue.cc:260] T 5557dfdd953a41e9bd3a8be4cd689604 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.323482 27693 consensus_queue.cc:260] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.324573 27695 ts_tablet_manager.cc:1428] T 5557dfdd953a41e9bd3a8be4cd689604 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:35.325320 27695 tablet_bootstrap.cc:492] T 793d6157927143d4b9d8dbafd7cf0fcb P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:35.327656 27695 tablet_bootstrap.cc:654] T 793d6157927143d4b9d8dbafd7cf0fcb P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.328068 27693 ts_tablet_manager.cc:1428] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250903 20:05:35.329092 27693 tablet_bootstrap.cc:492] T f1d22e76ea72441db51f7e9db26eb520 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:35.334440 27693 tablet_bootstrap.cc:654] T f1d22e76ea72441db51f7e9db26eb520 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.334570 27687 ts_tablet_manager.cc:1918] T 200c06f296b94eab94f35f312c5a443b P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:35.335122 27687 log.cc:1199] T 200c06f296b94eab94f35f312c5a443b P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/200c06f296b94eab94f35f312c5a443b
I20250903 20:05:35.335459 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet eb878263eda9447e870a782907fe6a8d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:35 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:35.335695 27687 ts_tablet_manager.cc:1939] T 200c06f296b94eab94f35f312c5a443b P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:35.338747 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_a4a91ad2f79c465cadf33585679b2708"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:35.339636 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4a91ad2f79c465cadf33585679b2708 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:35.341414 27064 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_66c6bafa9fa64867af5ff30e048122ed" } modify_external_catalogs: true
I20250903 20:05:35.341632 27064 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_66c6bafa9fa64867af5ff30e048122ed" } modify_external_catalogs: true
I20250903 20:05:35.343905 27685 ts_tablet_manager.cc:1918] T 120ca61b3da742dc9a2efa483ead7e97 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:35.344074 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 200c06f296b94eab94f35f312c5a443b (table default.Table_c5abd2620f3f45628ce2531cbcb81d9d [id=37960c5e67734caa951c9a151b0ece51]) successfully deleted
I20250903 20:05:35.344291 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_1d12656cde344c7e999bdec0a8729530"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:35.344887 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d12656cde344c7e999bdec0a8729530 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:35.345160 27687 tablet_replica.cc:331] T 4c18a98d5c0f4a43ae91c9130047b681 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:35.345360 27687 raft_consensus.cc:2241] T 4c18a98d5c0f4a43ae91c9130047b681 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:35.345520 27687 raft_consensus.cc:2270] T 4c18a98d5c0f4a43ae91c9130047b681 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:35.347548 27059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7baeb72c8bd440c39c07f34a6cb5e3ea consensus state has no leader
I20250903 20:05:35.344913 27695 tablet_bootstrap.cc:492] T 793d6157927143d4b9d8dbafd7cf0fcb P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:35.347862 27687 ts_tablet_manager.cc:1905] T 4c18a98d5c0f4a43ae91c9130047b681 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:35.347894 27695 ts_tablet_manager.cc:1397] T 793d6157927143d4b9d8dbafd7cf0fcb P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20250903 20:05:35.348618 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet eb878263eda9447e870a782907fe6a8d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:35 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:35.348665 27695 raft_consensus.cc:357] T 793d6157927143d4b9d8dbafd7cf0fcb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.348834 27695 raft_consensus.cc:383] T 793d6157927143d4b9d8dbafd7cf0fcb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.345239 27184 leader_election.cc:304] T 238f2f30377b42c2852b016018aa3245 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:35.349040 27685 log.cc:1199] T 120ca61b3da742dc9a2efa483ead7e97 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/120ca61b3da742dc9a2efa483ead7e97
I20250903 20:05:35.349143 29199 raft_consensus.cc:3058] T 238f2f30377b42c2852b016018aa3245 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:35.348886 27695 raft_consensus.cc:738] T 793d6157927143d4b9d8dbafd7cf0fcb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.349418 27695 consensus_queue.cc:260] T 793d6157927143d4b9d8dbafd7cf0fcb P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.349917 27695 ts_tablet_manager.cc:1428] T 793d6157927143d4b9d8dbafd7cf0fcb P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:35.350193 27695 tablet_bootstrap.cc:492] T 09fa32312c9648fbbd6fd327e9738066 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:35.351755 29199 raft_consensus.cc:2747] T 238f2f30377b42c2852b016018aa3245 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:35.351799 27695 tablet_bootstrap.cc:654] T 09fa32312c9648fbbd6fd327e9738066 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.354286 29227 consensus_queue.cc:1035] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:35.355681 27687 ts_tablet_manager.cc:1918] T 4c18a98d5c0f4a43ae91c9130047b681 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:35.356849 27058 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 7a9e55f0811345f7afa71645f38ebdaf (table default.Table_adf05e59a7be478b89666c3162ac83f1 [id=e69321e60a3a4e71a463a0011c0c6d65]). Expected version 2 got 0
I20250903 20:05:35.356952 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet eb878263eda9447e870a782907fe6a8d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:35 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:35.357314 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet aa9f5e0964b348b7af184f60cd145666 with delete_type TABLET_DATA_DELETED (replaced by tablet 26decef5841a4a3b85f2a5f578ae95dc) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:35.357293 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet eaa11d17303142ff935705c16f0b9690 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:35 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:35.357188 27687 log.cc:1199] T 4c18a98d5c0f4a43ae91c9130047b681 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/4c18a98d5c0f4a43ae91c9130047b681
I20250903 20:05:35.364647 27059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0c63411efaae4fdc900ae09cdd4eed96" } modify_external_catalogs: true
I20250903 20:05:35.364920 27059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0c63411efaae4fdc900ae09cdd4eed96" } modify_external_catalogs: true
I20250903 20:05:35.365023 27687 ts_tablet_manager.cc:1939] T 4c18a98d5c0f4a43ae91c9130047b681 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:35.365445 27684 ts_tablet_manager.cc:1905] T ed096b452ba747f89e91edc15e6e3ac1 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:35.365527 28710 consensus_queue.cc:1035] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:35.365845 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1108) took 1004 ms. Trace:
I20250903 20:05:35.365928 27687 rpcz_store.cc:276] 0903 20:05:34.361624 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.363344 (+ 1720us) service_pool.cc:225] Handling call
0903 20:05:35.345100 (+981756us) ts_tablet_manager.cc:1145] Deleting tablet 4c18a98d5c0f4a43ae91c9130047b681
0903 20:05:35.345112 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:35.351735 (+ 6623us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.355579 (+ 3844us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.365826 (+ 10247us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":981597}
I20250903 20:05:35.366415 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 4c18a98d5c0f4a43ae91c9130047b681 (table default.Table_16b62656e76142339c0a42ca5a7f5e6b [id=189c3642e92c43759dee94f23ef1aec3]) successfully deleted
I20250903 20:05:35.368409 27066 master_service.cc:948] ReplaceTablet: received request to replace tablet 7baeb72c8bd440c39c07f34a6cb5e3ea from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:35.372195 27687 tablet_replica.cc:331] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:35.374434 28710 raft_consensus.cc:491] T aa9f5e0964b348b7af184f60cd145666 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:35.374627 28710 raft_consensus.cc:513] T aa9f5e0964b348b7af184f60cd145666 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.375381 28710 leader_election.cc:290] T aa9f5e0964b348b7af184f60cd145666 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:35.377039 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa9f5e0964b348b7af184f60cd145666" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:35.379004 27687 raft_consensus.cc:2241] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:35.377789 29227 consensus_queue.cc:1035] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:35.378329 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa9f5e0964b348b7af184f60cd145666" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:35.379739 27379 raft_consensus.cc:2391] T aa9f5e0964b348b7af184f60cd145666 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e72ef1dbcc5143cf9f04a808f101c80f in current term 1: Already voted for candidate 89f8f0ee68b940db9ce250c0604f934a in this term.
I20250903 20:05:35.382792 27462 leader_election.cc:304] T aa9f5e0964b348b7af184f60cd145666 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:35.383368 28710 raft_consensus.cc:3058] T aa9f5e0964b348b7af184f60cd145666 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:35.383553 27692 tablet_bootstrap.cc:492] T de20d18d7f264811a0631aa1144aef2c P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:35.383698 27692 ts_tablet_manager.cc:1397] T de20d18d7f264811a0631aa1144aef2c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.104s user 0.007s sys 0.000s
I20250903 20:05:35.384554 27692 raft_consensus.cc:357] T de20d18d7f264811a0631aa1144aef2c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.384723 27692 raft_consensus.cc:383] T de20d18d7f264811a0631aa1144aef2c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.384774 27692 raft_consensus.cc:738] T de20d18d7f264811a0631aa1144aef2c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.385571 27692 consensus_queue.cc:260] T de20d18d7f264811a0631aa1144aef2c P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.386389 28710 raft_consensus.cc:2747] T aa9f5e0964b348b7af184f60cd145666 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:35.387173 27692 ts_tablet_manager.cc:1428] T de20d18d7f264811a0631aa1144aef2c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250903 20:05:35.372850 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet 09fa32312c9648fbbd6fd327e9738066 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:35.387506 27692 tablet_bootstrap.cc:492] T ee40566e497740789575275d0cdca0db P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:35.387840 27687 raft_consensus.cc:2270] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:35.389833 27687 ts_tablet_manager.cc:1905] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:35.390859 27692 tablet_bootstrap.cc:654] T ee40566e497740789575275d0cdca0db P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.398367 27685 ts_tablet_manager.cc:1939] T 120ca61b3da742dc9a2efa483ead7e97 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:35.399436 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 957) took 3316 ms. Trace:
I20250903 20:05:35.399529 27685 rpcz_store.cc:276] 0903 20:05:32.082922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.083402 (+ 480us) service_pool.cc:225] Handling call
0903 20:05:35.276084 (+3192682us) ts_tablet_manager.cc:1145] Deleting tablet 120ca61b3da742dc9a2efa483ead7e97
0903 20:05:35.276098 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:35.332203 (+ 56105us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.343783 (+ 11580us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.399417 (+ 55634us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3192467}
I20250903 20:05:35.399828 27685 tablet_replica.cc:331] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:35.400127 27685 raft_consensus.cc:2241] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:35.400456 27685 raft_consensus.cc:2270] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:35.400717 29221 consensus_queue.cc:1035] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:35.401115 27685 ts_tablet_manager.cc:1905] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:35.402100 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 120ca61b3da742dc9a2efa483ead7e97 (table default.Table_47699f82f0074c7eac5b4b83296c71e5 [id=a10d0a5b11484d078240afb2ad212bff]) successfully deleted
I20250903 20:05:35.409584 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6d2d1ff2291b43f8ae4c0a6a456f5014" } modify_external_catalogs: true
I20250903 20:05:35.409785 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6d2d1ff2291b43f8ae4c0a6a456f5014" } modify_external_catalogs: true
I20250903 20:05:35.409925 29221 raft_consensus.cc:491] T 472f0fb01263468a828ec13795656624 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:35.410065 29221 raft_consensus.cc:513] T 472f0fb01263468a828ec13795656624 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.410171 27695 tablet_bootstrap.cc:492] T 09fa32312c9648fbbd6fd327e9738066 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:35.410319 27695 ts_tablet_manager.cc:1397] T 09fa32312c9648fbbd6fd327e9738066 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.060s user 0.005s sys 0.005s
I20250903 20:05:35.410899 29221 leader_election.cc:290] T 472f0fb01263468a828ec13795656624 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:35.411049 27695 raft_consensus.cc:357] T 09fa32312c9648fbbd6fd327e9738066 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.411198 27695 raft_consensus.cc:383] T 09fa32312c9648fbbd6fd327e9738066 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.411252 27695 raft_consensus.cc:738] T 09fa32312c9648fbbd6fd327e9738066 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.412397 27693 tablet_bootstrap.cc:492] T f1d22e76ea72441db51f7e9db26eb520 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:35.412518 27693 ts_tablet_manager.cc:1397] T f1d22e76ea72441db51f7e9db26eb520 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.084s user 0.004s sys 0.001s
I20250903 20:05:35.413244 27693 raft_consensus.cc:357] T f1d22e76ea72441db51f7e9db26eb520 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.413403 27693 raft_consensus.cc:383] T f1d22e76ea72441db51f7e9db26eb520 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.413450 27693 raft_consensus.cc:738] T f1d22e76ea72441db51f7e9db26eb520 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.414378 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "472f0fb01263468a828ec13795656624" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:35.414553 27240 raft_consensus.cc:2466] T 472f0fb01263468a828ec13795656624 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:35.415033 27462 leader_election.cc:304] T 472f0fb01263468a828ec13795656624 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:35.415422 29221 raft_consensus.cc:2802] T 472f0fb01263468a828ec13795656624 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:35.415498 29221 raft_consensus.cc:491] T 472f0fb01263468a828ec13795656624 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:35.415566 29221 raft_consensus.cc:3058] T 472f0fb01263468a828ec13795656624 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:35.418205 29221 raft_consensus.cc:513] T 472f0fb01263468a828ec13795656624 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.420650 27692 tablet_bootstrap.cc:492] T ee40566e497740789575275d0cdca0db P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:35.420791 27692 ts_tablet_manager.cc:1397] T ee40566e497740789575275d0cdca0db P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.000s
I20250903 20:05:35.421602 27692 raft_consensus.cc:357] T ee40566e497740789575275d0cdca0db P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.421759 27692 raft_consensus.cc:383] T ee40566e497740789575275d0cdca0db P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.421811 27692 raft_consensus.cc:738] T ee40566e497740789575275d0cdca0db P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.422049 27692 consensus_queue.cc:260] T ee40566e497740789575275d0cdca0db P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.422557 27692 ts_tablet_manager.cc:1428] T ee40566e497740789575275d0cdca0db P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:35.424260 27692 tablet_bootstrap.cc:492] T 7824a0bccdc9485abde124572e502791 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:35.426002 27692 tablet_bootstrap.cc:654] T 7824a0bccdc9485abde124572e502791 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.426083 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ec7b30148c954f9e8eff70562aed4edb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:35.426433 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "472f0fb01263468a828ec13795656624" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:35.426596 27240 raft_consensus.cc:3058] T 472f0fb01263468a828ec13795656624 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:35.426739 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec7b30148c954f9e8eff70562aed4edb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:35.429575 27240 raft_consensus.cc:2466] T 472f0fb01263468a828ec13795656624 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:35.430083 27462 leader_election.cc:304] T 472f0fb01263468a828ec13795656624 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:35.430497 29220 raft_consensus.cc:2802] T 472f0fb01263468a828ec13795656624 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:35.430610 29220 raft_consensus.cc:695] T 472f0fb01263468a828ec13795656624 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:35.430855 29220 consensus_queue.cc:237] T 472f0fb01263468a828ec13795656624 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.413813 27693 consensus_queue.cc:260] T f1d22e76ea72441db51f7e9db26eb520 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.431607 27684 ts_tablet_manager.cc:1918] T ed096b452ba747f89e91edc15e6e3ac1 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:35.431707 27684 log.cc:1199] T ed096b452ba747f89e91edc15e6e3ac1 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/ed096b452ba747f89e91edc15e6e3ac1
I20250903 20:05:35.432044 27684 ts_tablet_manager.cc:1939] T ed096b452ba747f89e91edc15e6e3ac1 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:35.434983 27693 ts_tablet_manager.cc:1428] T f1d22e76ea72441db51f7e9db26eb520 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20250903 20:05:35.435396 27693 tablet_bootstrap.cc:492] T f326ec776ba446be8d5dda5c1dc5b6ec P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:35.437620 27693 tablet_bootstrap.cc:654] T f326ec776ba446be8d5dda5c1dc5b6ec P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.440070 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "472f0fb01263468a828ec13795656624" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:35.440382 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "472f0fb01263468a828ec13795656624" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:35.450544 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet a7f25f00dff7426195ac51214a14022b
I20250903 20:05:35.450814 27695 consensus_queue.cc:260] T 09fa32312c9648fbbd6fd327e9738066 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.452255 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet a7f25f00dff7426195ac51214a14022b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:35 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:35.452941 27695 ts_tablet_manager.cc:1428] T 09fa32312c9648fbbd6fd327e9738066 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.043s user 0.002s sys 0.000s
I20250903 20:05:35.455060 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet a7f25f00dff7426195ac51214a14022b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:35 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:35.465164 27695 tablet_bootstrap.cc:492] T 8ba2636641654d8596fd2a5b0644d073 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:35.466627 27063 master_service.cc:948] ReplaceTablet: received request to replace tablet 2c772ab7c50c4994aacde5a05bb0f08e from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:35.468587 27695 tablet_bootstrap.cc:654] T 8ba2636641654d8596fd2a5b0644d073 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.443362 29221 leader_election.cc:290] T 472f0fb01263468a828ec13795656624 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:35.446573 27462 leader_election.cc:343] T 472f0fb01263468a828ec13795656624 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:35.452669 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet a7f25f00dff7426195ac51214a14022b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:35 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:35.467320 29220 raft_consensus.cc:491] T 47c1287bff1145a5b2ab6d14dbd608c7 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:35.474581 29220 raft_consensus.cc:513] T 47c1287bff1145a5b2ab6d14dbd608c7 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:35.474825 27462 leader_election.cc:343] T 472f0fb01263468a828ec13795656624 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:35.476657 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1081) took 1178 ms. Trace:
I20250903 20:05:35.476747 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47c1287bff1145a5b2ab6d14dbd608c7" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:35.477690 27685 ts_tablet_manager.cc:1918] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:35.484141 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_257827f9b2804d37aa0556a8affcf93e" } new_table_name: "default.Table_6485a2bac12d42d98e4ab9dd1afd129b" modify_external_catalogs: true
I20250903 20:05:35.481767 27695 tablet_bootstrap.cc:492] T 8ba2636641654d8596fd2a5b0644d073 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:35.484805 27695 ts_tablet_manager.cc:1397] T 8ba2636641654d8596fd2a5b0644d073 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.004s
I20250903 20:05:35.485256 27685 log.cc:1199] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/1f6cd4d475f147fe8dfcc7e3a73dedf8
I20250903 20:05:35.485548 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_77dba1c3fea74301973e35bf475df57a" } modify_external_catalogs: true
I20250903 20:05:35.485733 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_77dba1c3fea74301973e35bf475df57a" } modify_external_catalogs: true
I20250903 20:05:35.485983 27685 ts_tablet_manager.cc:1939] T 1f6cd4d475f147fe8dfcc7e3a73dedf8 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:35.486763 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 961) took 3403 ms. Trace:
I20250903 20:05:35.487641 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet c9d09ecd8fe347d79283fc9a3056e844
I20250903 20:05:35.490198 27693 tablet_bootstrap.cc:492] T f326ec776ba446be8d5dda5c1dc5b6ec P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:35.490746 27693 ts_tablet_manager.cc:1397] T f326ec776ba446be8d5dda5c1dc5b6ec P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.002s
I20250903 20:05:35.492151 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet c9d09ecd8fe347d79283fc9a3056e844 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:35 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:35.493119 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_9beb5c7440aa47549f15ef79b911dfea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:35.493219 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e57f2570b5f3401cbadd48277ed8993e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:35.494105 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9beb5c7440aa47549f15ef79b911dfea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:35.494097 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e57f2570b5f3401cbadd48277ed8993e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:35.477365 27462 leader_election.cc:343] T 47c1287bff1145a5b2ab6d14dbd608c7 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:35.491024 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet c9d09ecd8fe347d79283fc9a3056e844 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:35 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:35.497372 27692 tablet_bootstrap.cc:492] T 7824a0bccdc9485abde124572e502791 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:35.497525 27692 ts_tablet_manager.cc:1397] T 7824a0bccdc9485abde124572e502791 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.074s user 0.004s sys 0.002s
I20250903 20:05:35.497638 29220 leader_election.cc:290] T 47c1287bff1145a5b2ab6d14dbd608c7 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:35.497815 27684 rpcz_store.cc:276] 0903 20:05:34.297762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.297829 (+ 67us) service_pool.cc:225] Handling call
0903 20:05:35.286527 (+988698us) ts_tablet_manager.cc:1145] Deleting tablet ed096b452ba747f89e91edc15e6e3ac1
0903 20:05:35.286541 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:35.381602 (+ 95061us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.431491 (+ 49889us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.476627 (+ 45136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":987990}
I20250903 20:05:35.498435 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet ed096b452ba747f89e91edc15e6e3ac1 (table default.Table_5bc4b150c12940db90c7800f5d07b9ed [id=27548347ece844e3bb55ba894da5c8e0]) successfully deleted
I20250903 20:05:35.498390 27692 raft_consensus.cc:357] T 7824a0bccdc9485abde124572e502791 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.498555 27692 raft_consensus.cc:383] T 7824a0bccdc9485abde124572e502791 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.498601 27692 raft_consensus.cc:738] T 7824a0bccdc9485abde124572e502791 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.498881 27684 tablet_replica.cc:331] T 950a6fcd35f64f2d889831fe9cdf503a P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:35.499332 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47c1287bff1145a5b2ab6d14dbd608c7" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:35.499845 27684 raft_consensus.cc:2241] T 950a6fcd35f64f2d889831fe9cdf503a P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
W20250903 20:05:35.499858 27462 leader_election.cc:343] T 47c1287bff1145a5b2ab6d14dbd608c7 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:35.499992 27462 leader_election.cc:304] T 47c1287bff1145a5b2ab6d14dbd608c7 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:35.500509 29220 raft_consensus.cc:2747] T 47c1287bff1145a5b2ab6d14dbd608c7 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:35.498797 27692 consensus_queue.cc:260] T 7824a0bccdc9485abde124572e502791 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.501117 27692 ts_tablet_manager.cc:1428] T 7824a0bccdc9485abde124572e502791 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:35.501452 27692 tablet_bootstrap.cc:492] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:35.503304 27692 tablet_bootstrap.cc:654] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.503268 27693 raft_consensus.cc:357] T f326ec776ba446be8d5dda5c1dc5b6ec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:35.503995 27693 raft_consensus.cc:383] T f326ec776ba446be8d5dda5c1dc5b6ec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.504105 27693 raft_consensus.cc:738] T f326ec776ba446be8d5dda5c1dc5b6ec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.504336 27693 consensus_queue.cc:260] T f326ec776ba446be8d5dda5c1dc5b6ec P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:35.500033 27684 raft_consensus.cc:2270] T 950a6fcd35f64f2d889831fe9cdf503a P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:35.504873 27693 ts_tablet_manager.cc:1428] T f326ec776ba446be8d5dda5c1dc5b6ec P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:35.487560 27695 raft_consensus.cc:357] T 8ba2636641654d8596fd2a5b0644d073 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.505676 27695 raft_consensus.cc:383] T 8ba2636641654d8596fd2a5b0644d073 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.505795 27695 raft_consensus.cc:738] T 8ba2636641654d8596fd2a5b0644d073 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.489460 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet c9d09ecd8fe347d79283fc9a3056e844 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:35 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:35.506716 27684 ts_tablet_manager.cc:1905] T 950a6fcd35f64f2d889831fe9cdf503a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:35.490582 27687 ts_tablet_manager.cc:1918] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:35.507043 27687 log.cc:1199] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/e877d56bbbfa4c5a92d025d0311ce5c6
I20250903 20:05:35.507442 27687 ts_tablet_manager.cc:1939] T e877d56bbbfa4c5a92d025d0311ce5c6 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:35.506150 27695 consensus_queue.cc:260] T 8ba2636641654d8596fd2a5b0644d073 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.505098 27685 rpcz_store.cc:276] 0903 20:05:32.083700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.084147 (+ 447us) service_pool.cc:225] Handling call
0903 20:05:35.399784 (+3315637us) ts_tablet_manager.cc:1145] Deleting tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8
0903 20:05:35.399792 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:35.460116 (+ 60324us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.477570 (+ 17454us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.486743 (+ 9173us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3315486}
I20250903 20:05:35.508273 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8 (table default.Table_943bea896c744410a2a4d80ffb9ca439 [id=a950fbaecd7341989961ff3ea9a713e0]) successfully deleted
I20250903 20:05:35.509562 29285 raft_consensus.cc:491] T 46f49fd718be4a2996d781ab31268217 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:35.509688 29285 raft_consensus.cc:513] T 46f49fd718be4a2996d781ab31268217 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.511394 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46f49fd718be4a2996d781ab31268217" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:35.511617 27516 raft_consensus.cc:2391] T 46f49fd718be4a2996d781ab31268217 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89f8f0ee68b940db9ce250c0604f934a in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
I20250903 20:05:35.512286 29285 leader_election.cc:290] T 46f49fd718be4a2996d781ab31268217 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
W20250903 20:05:35.514525 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2ce6e8b068bf4c06916f0c23da1d5824 consensus state has no leader
I20250903 20:05:35.518419 27685 tablet_replica.cc:331] T 4af8c03c9d834405b4e25d35441db961 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:35.518689 27685 raft_consensus.cc:2241] T 4af8c03c9d834405b4e25d35441db961 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:35.518936 27685 raft_consensus.cc:2270] T 4af8c03c9d834405b4e25d35441db961 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:35.519737 27685 ts_tablet_manager.cc:1905] T 4af8c03c9d834405b4e25d35441db961 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:35.514647 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46f49fd718be4a2996d781ab31268217" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:35.517017 27693 tablet_bootstrap.cc:492] T cb2f50683e904dd39ee6eb128717d38c P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:35.523408 27184 leader_election.cc:304] T 46f49fd718be4a2996d781ab31268217 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:35.524034 29285 raft_consensus.cc:3058] T 46f49fd718be4a2996d781ab31268217 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:35.524813 27693 tablet_bootstrap.cc:654] T cb2f50683e904dd39ee6eb128717d38c P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.527249 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_83ea6dfcee144a4fbc86931567b676bb" } new_table_name: "default.Table_1153bf74b285448b88b364efe2169784" modify_external_catalogs: true
I20250903 20:05:35.527690 29285 raft_consensus.cc:2747] T 46f49fd718be4a2996d781ab31268217 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:35.529024 27064 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_193a0ff511204b3b8130c17e783b51ad" } modify_external_catalogs: true
I20250903 20:05:35.529251 27064 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_193a0ff511204b3b8130c17e783b51ad" } modify_external_catalogs: true
I20250903 20:05:35.537559 29199 consensus_queue.cc:1035] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:35.539352 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1d12656cde344c7e999bdec0a8729530" } new_table_name: "default.Table_c86263c481ce4c5ebeb98439c682f1a2" modify_external_catalogs: true
I20250903 20:05:35.539858 27695 ts_tablet_manager.cc:1428] T 8ba2636641654d8596fd2a5b0644d073 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.055s user 0.002s sys 0.000s
I20250903 20:05:35.541520 27695 tablet_bootstrap.cc:492] T 24eadc70e10d4d02bcb643d45c9e0e36 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:35.545328 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet f326ec776ba446be8d5dda5c1dc5b6ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:35.551990 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_32800b753a384355979ff8944551c12c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:35.552800 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32800b753a384355979ff8944551c12c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:35.554582 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1107) took 1193 ms. Trace:
I20250903 20:05:35.554772 27687 rpcz_store.cc:276] 0903 20:05:34.361536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.366854 (+ 5318us) service_pool.cc:225] Handling call
0903 20:05:35.372122 (+1005268us) ts_tablet_manager.cc:1145] Deleting tablet e877d56bbbfa4c5a92d025d0311ce5c6
0903 20:05:35.372135 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:35.394089 (+ 21954us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.490476 (+ 96387us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.554548 (+ 64072us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1004755}
I20250903 20:05:35.555509 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet e877d56bbbfa4c5a92d025d0311ce5c6 (table default.Table_d9105c35c28247f1a71866f2096d93f2 [id=33b22e921daa4ccf9406155bac438d08]) successfully deleted
I20250903 20:05:35.561235 29196 consensus_queue.cc:1035] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:35.562330 27693 tablet_bootstrap.cc:492] T cb2f50683e904dd39ee6eb128717d38c P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:35.562566 27693 ts_tablet_manager.cc:1397] T cb2f50683e904dd39ee6eb128717d38c P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.046s user 0.006s sys 0.005s
I20250903 20:05:35.563107 27692 tablet_bootstrap.cc:492] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:35.563247 27692 ts_tablet_manager.cc:1397] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.000s
I20250903 20:05:35.563700 27693 raft_consensus.cc:357] T cb2f50683e904dd39ee6eb128717d38c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:35.563969 27693 raft_consensus.cc:383] T cb2f50683e904dd39ee6eb128717d38c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.564076 27693 raft_consensus.cc:738] T cb2f50683e904dd39ee6eb128717d38c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.565213 27693 consensus_queue.cc:260] T cb2f50683e904dd39ee6eb128717d38c P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:35.565783 27692 raft_consensus.cc:357] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.565956 27692 raft_consensus.cc:383] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.566010 27692 raft_consensus.cc:738] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.566913 27692 consensus_queue.cc:260] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.567761 27692 ts_tablet_manager.cc:1428] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:35.568050 27692 tablet_bootstrap.cc:492] T 3f303233e6c347289650c6cf83f8127f P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:35.569456 27693 ts_tablet_manager.cc:1428] T cb2f50683e904dd39ee6eb128717d38c P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250903 20:05:35.569593 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_cb85e96534984e88a664c7183a732fe4" } new_table_name: "default.Table_9ea78ec0f6314be48893b91681dd50f3" modify_external_catalogs: true
I20250903 20:05:35.570356 27692 tablet_bootstrap.cc:654] T 3f303233e6c347289650c6cf83f8127f P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.571341 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1113) took 1129 ms. Trace:
I20250903 20:05:35.571934 27687 rpcz_store.cc:276] 0903 20:05:34.442001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.442087 (+ 86us) service_pool.cc:225] Handling call
0903 20:05:35.555861 (+1113774us) ts_tablet_manager.cc:1145] Deleting tablet 13137053365a428e9d1e8d8852a53b6a
0903 20:05:35.570031 (+ 14170us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:35.571317 (+ 1286us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1113498}
W20250903 20:05:35.572970 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 13137053365a428e9d1e8d8852a53b6a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13137053365a428e9d1e8d8852a53b6a already in progress: creating tablet
I20250903 20:05:35.578976 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1115) took 1115 ms. Trace:
W20250903 20:05:35.579584 27059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9942bb4b868d4a08866298d7790ecad2 has no consensus state
I20250903 20:05:35.579347 27687 rpcz_store.cc:276] 0903 20:05:34.463352 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.463594 (+ 242us) service_pool.cc:225] Handling call
0903 20:05:35.573494 (+1109900us) ts_tablet_manager.cc:1145] Deleting tablet 2035344c7f6d478b89a89981d89f22ca
0903 20:05:35.578085 (+ 4591us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:35.578899 (+ 814us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1109705}
I20250903 20:05:35.580120 27687 tablet_replica.cc:331] T f1d22e76ea72441db51f7e9db26eb520 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:35.580376 27687 raft_consensus.cc:2241] T f1d22e76ea72441db51f7e9db26eb520 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:35.580571 27687 raft_consensus.cc:2270] T f1d22e76ea72441db51f7e9db26eb520 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:35.580996 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 2035344c7f6d478b89a89981d89f22ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2035344c7f6d478b89a89981d89f22ca already in progress: creating tablet
I20250903 20:05:35.582026 27687 ts_tablet_manager.cc:1905] T f1d22e76ea72441db51f7e9db26eb520 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:35.584944 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet cb2f50683e904dd39ee6eb128717d38c with delete_type TABLET_DATA_DELETED (replaced by tablet 72fcb7b0cba34313a7dc2b8c29e120fc) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:35.585414 27695 tablet_bootstrap.cc:654] T 24eadc70e10d4d02bcb643d45c9e0e36 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.586131 27693 tablet_bootstrap.cc:492] T 793d6157927143d4b9d8dbafd7cf0fcb P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:35.586809 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_694c90191216407784c7fea5d5379e2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:35.587625 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_694c90191216407784c7fea5d5379e2d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:35.588697 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_9533e6ddb5a84196b359ee85dd13e42c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:35.589248 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9533e6ddb5a84196b359ee85dd13e42c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:35.593983 27693 tablet_bootstrap.cc:654] T 793d6157927143d4b9d8dbafd7cf0fcb P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.600557 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3bcd6f6a18284a278b1467fe1c2374a3" } new_table_name: "default.Table_520d12af7b354cff9d2a356e9b59d94a" modify_external_catalogs: true
I20250903 20:05:35.601109 27685 ts_tablet_manager.cc:1918] T 4af8c03c9d834405b4e25d35441db961 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250903 20:05:35.602733 26890 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 52311.1us
I20250903 20:05:35.603804 27693 tablet_bootstrap.cc:492] T 793d6157927143d4b9d8dbafd7cf0fcb P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:35.604132 27693 ts_tablet_manager.cc:1397] T 793d6157927143d4b9d8dbafd7cf0fcb P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.002s
I20250903 20:05:35.609337 29220 raft_consensus.cc:491] T f1d22e76ea72441db51f7e9db26eb520 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:35.609496 29220 raft_consensus.cc:513] T f1d22e76ea72441db51f7e9db26eb520 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.610080 29220 leader_election.cc:290] T f1d22e76ea72441db51f7e9db26eb520 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:35.611290 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_949650a78270483689230d33cb7c4a3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:35.612025 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_949650a78270483689230d33cb7c4a3b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:35.613298 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f1d22e76ea72441db51f7e9db26eb520" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:35.614063 27462 leader_election.cc:343] T f1d22e76ea72441db51f7e9db26eb520 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: Tablet not RUNNING: STOPPED
I20250903 20:05:35.614640 27693 raft_consensus.cc:357] T 793d6157927143d4b9d8dbafd7cf0fcb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.618038 27058 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 8ba2636641654d8596fd2a5b0644d073 (table default.Table_e7148e16606e4cd3908c86b439f75b78 [id=4bffa29ea98049718ac3ce35de9813c5]). Expected version 1 got 0
I20250903 20:05:35.618443 27058 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 7a9e55f0811345f7afa71645f38ebdaf (table default.Table_adf05e59a7be478b89666c3162ac83f1 [id=e69321e60a3a4e71a463a0011c0c6d65]). Expected version 2 got 0
I20250903 20:05:35.619910 27507 tablet_service.cc:1468] Processing CreateTablet for tablet bde699e79b3b4911823641275bec96dd (DEFAULT_TABLE table=default.Table_9702c9ad8b8a4f0286cefc21dcfd6f10 [id=f584f3f92f104097997ec4e32fa740bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.620838 27692 tablet_bootstrap.cc:492] T 3f303233e6c347289650c6cf83f8127f P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:35.620957 27692 ts_tablet_manager.cc:1397] T 3f303233e6c347289650c6cf83f8127f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.053s user 0.001s sys 0.002s
I20250903 20:05:35.622177 27060 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 7a9e55f0811345f7afa71645f38ebdaf (table default.Table_adf05e59a7be478b89666c3162ac83f1 [id=e69321e60a3a4e71a463a0011c0c6d65]). Expected version 2 got 0
I20250903 20:05:35.628692 27692 raft_consensus.cc:357] T 3f303233e6c347289650c6cf83f8127f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.628892 27692 raft_consensus.cc:383] T 3f303233e6c347289650c6cf83f8127f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.629649 27692 raft_consensus.cc:738] T 3f303233e6c347289650c6cf83f8127f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.629861 27692 consensus_queue.cc:260] T 3f303233e6c347289650c6cf83f8127f P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.630900 27692 ts_tablet_manager.cc:1428] T 3f303233e6c347289650c6cf83f8127f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20250903 20:05:35.631191 27692 tablet_bootstrap.cc:492] T 793d6157927143d4b9d8dbafd7cf0fcb P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:35.631381 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet 47c1287bff1145a5b2ab6d14dbd608c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:35.632863 27692 tablet_bootstrap.cc:654] T 793d6157927143d4b9d8dbafd7cf0fcb P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.633517 27500 tablet_service.cc:1515] Processing DeleteTablet for tablet cb2f50683e904dd39ee6eb128717d38c with delete_type TABLET_DATA_DELETED (replaced by tablet 72fcb7b0cba34313a7dc2b8c29e120fc) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:35.633701 27496 tablet_service.cc:1515] Processing DeleteTablet for tablet 64c4dd2645034a7eaf243f5474603f10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:35.634589 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_0276baa7ef3b41c994e5d131569d5955"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:35.635344 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0276baa7ef3b41c994e5d131569d5955 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:35.619849 27361 tablet_service.cc:1468] Processing CreateTablet for tablet bde699e79b3b4911823641275bec96dd (DEFAULT_TABLE table=default.Table_9702c9ad8b8a4f0286cefc21dcfd6f10 [id=f584f3f92f104097997ec4e32fa740bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.643164 27693 raft_consensus.cc:383] T 793d6157927143d4b9d8dbafd7cf0fcb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.643268 27693 raft_consensus.cc:738] T 793d6157927143d4b9d8dbafd7cf0fcb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.643462 27693 consensus_queue.cc:260] T 793d6157927143d4b9d8dbafd7cf0fcb P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.643564 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bde699e79b3b4911823641275bec96dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.643985 27693 ts_tablet_manager.cc:1428] T 793d6157927143d4b9d8dbafd7cf0fcb P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.040s user 0.000s sys 0.001s
I20250903 20:05:35.644271 27693 tablet_bootstrap.cc:492] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:35.646000 27693 tablet_bootstrap.cc:654] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.660447 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bde699e79b3b4911823641275bec96dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.670204 29319 tablet.cc:1722] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:35.671552 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 6ec346392c4d4260b98b4acacdd14bbb (DEFAULT_TABLE table=default.Table_9c1562dad48f49b59b791ce173464002 [id=300d59bcd84a466893fd7f2ea45d891a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.672401 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ec346392c4d4260b98b4acacdd14bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.673118 27684 ts_tablet_manager.cc:1918] T 950a6fcd35f64f2d889831fe9cdf503a P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:35.673354 27684 log.cc:1199] T 950a6fcd35f64f2d889831fe9cdf503a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/950a6fcd35f64f2d889831fe9cdf503a
I20250903 20:05:35.673771 27684 ts_tablet_manager.cc:1939] T 950a6fcd35f64f2d889831fe9cdf503a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:35.612419 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f1d22e76ea72441db51f7e9db26eb520" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:35.674646 27685 log.cc:1199] T 4af8c03c9d834405b4e25d35441db961 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/4af8c03c9d834405b4e25d35441db961
I20250903 20:05:35.675040 27685 ts_tablet_manager.cc:1939] T 4af8c03c9d834405b4e25d35441db961 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
W20250903 20:05:35.675647 27462 leader_election.cc:343] T f1d22e76ea72441db51f7e9db26eb520 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:35.675760 27462 leader_election.cc:304] T f1d22e76ea72441db51f7e9db26eb520 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:35.675794 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 962) took 3592 ms. Trace:
I20250903 20:05:35.675863 27685 rpcz_store.cc:276] 0903 20:05:32.083783 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.086266 (+ 2483us) service_pool.cc:225] Handling call
0903 20:05:35.518345 (+3432079us) ts_tablet_manager.cc:1145] Deleting tablet 4af8c03c9d834405b4e25d35441db961
0903 20:05:35.518360 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:35.597806 (+ 79446us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.600972 (+ 3166us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.675777 (+ 74805us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3431929}
I20250903 20:05:35.676147 29220 raft_consensus.cc:2747] T f1d22e76ea72441db51f7e9db26eb520 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:35.676360 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 959) took 3593 ms. Trace:
I20250903 20:05:35.676295 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 4af8c03c9d834405b4e25d35441db961 (table default.Table_9eac2fd82a524aae943ebd76428f95eb [id=058a64daaf104dfdace7d05d52388b79]) successfully deleted
I20250903 20:05:35.676427 27685 rpcz_store.cc:276] 0903 20:05:32.083157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.086659 (+ 3502us) service_pool.cc:225] Handling call
0903 20:05:35.676262 (+3589603us) ts_tablet_manager.cc:1145] Deleting tablet a0293a69f1c246d08d414e95729b4ab4
0903 20:05:35.676271 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:35.676342 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3589450}
W20250903 20:05:35.676813 27052 catalog_manager.cc:4895] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet a0293a69f1c246d08d414e95729b4ab4 because the tablet was not found. No further retry: Not found: Tablet not found: a0293a69f1c246d08d414e95729b4ab4
I20250903 20:05:35.676862 27685 tablet_replica.cc:331] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:35.677093 27685 raft_consensus.cc:2241] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:35.677448 27685 raft_consensus.cc:2270] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:35.678077 27685 ts_tablet_manager.cc:1905] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:35.680444 27221 tablet_service.cc:1468] Processing CreateTablet for tablet 6ec346392c4d4260b98b4acacdd14bbb (DEFAULT_TABLE table=default.Table_9c1562dad48f49b59b791ce173464002 [id=300d59bcd84a466893fd7f2ea45d891a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.681038 29199 raft_consensus.cc:491] T b7a6d5d8a97a4769a81ec7224d591f8c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e72ef1dbcc5143cf9f04a808f101c80f)
I20250903 20:05:35.681190 29199 raft_consensus.cc:513] T b7a6d5d8a97a4769a81ec7224d591f8c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.681833 29199 leader_election.cc:290] T b7a6d5d8a97a4769a81ec7224d591f8c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
W20250903 20:05:35.682677 27184 leader_election.cc:343] T b7a6d5d8a97a4769a81ec7224d591f8c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: b7a6d5d8a97a4769a81ec7224d591f8c
W20250903 20:05:35.682945 27183 leader_election.cc:343] T b7a6d5d8a97a4769a81ec7224d591f8c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: b7a6d5d8a97a4769a81ec7224d591f8c
I20250903 20:05:35.683060 27183 leader_election.cc:304] T b7a6d5d8a97a4769a81ec7224d591f8c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:35.683389 29199 raft_consensus.cc:2747] T b7a6d5d8a97a4769a81ec7224d591f8c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:35.684849 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 2da6c143daa6410780a2c32f97b84ae2 (DEFAULT_TABLE table=default.Table_021085b0fd254dbf961aa3d9aeadd6d6 [id=7f3821b85d7b45eabfe9957a159f57a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.685278 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2da6c143daa6410780a2c32f97b84ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.617312 27695 tablet_bootstrap.cc:492] T 24eadc70e10d4d02bcb643d45c9e0e36 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:35.625430 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet 46f49fd718be4a2996d781ab31268217 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:23 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:35.625931 27216 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:14 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:35.626050 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet 238f2f30377b42c2852b016018aa3245 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:35.639819 29321 tablet.cc:1722] T 06a79706f1184fa49d10182ba669b0ab P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:35.666458 29285 raft_consensus.cc:491] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e72ef1dbcc5143cf9f04a808f101c80f)
I20250903 20:05:35.690735 29285 raft_consensus.cc:513] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.669587 27215 tablet_service.cc:1468] Processing CreateTablet for tablet bde699e79b3b4911823641275bec96dd (DEFAULT_TABLE table=default.Table_9702c9ad8b8a4f0286cefc21dcfd6f10 [id=f584f3f92f104097997ec4e32fa740bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.691365 29285 leader_election.cc:290] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
W20250903 20:05:35.691505 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1157920321454d8790b2620690878579 consensus state has no leader
W20250903 20:05:35.692335 27183 leader_election.cc:343] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 63a1f26fac3b4b939ebb79c6a07d8f8e
W20250903 20:05:35.695390 27184 leader_election.cc:343] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 63a1f26fac3b4b939ebb79c6a07d8f8e
I20250903 20:05:35.695544 27184 leader_election.cc:304] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:35.686177 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1082) took 1332 ms. Trace:
I20250903 20:05:35.696013 29285 raft_consensus.cc:2747] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:35.695864 27684 rpcz_store.cc:276] 0903 20:05:34.354282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.354386 (+ 104us) service_pool.cc:225] Handling call
0903 20:05:35.498833 (+1144447us) ts_tablet_manager.cc:1145] Deleting tablet 950a6fcd35f64f2d889831fe9cdf503a
0903 20:05:35.498843 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:35.591435 (+ 92592us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.672983 (+ 81548us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.686149 (+ 13166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11470,"tablet-delete.queue_time_us":1144101}
I20250903 20:05:35.704909 27222 tablet_service.cc:1468] Processing CreateTablet for tablet 2da6c143daa6410780a2c32f97b84ae2 (DEFAULT_TABLE table=default.Table_021085b0fd254dbf961aa3d9aeadd6d6 [id=7f3821b85d7b45eabfe9957a159f57a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.705164 27684 tablet_replica.cc:331] T 1223743918814d88b401782ca4b521dd P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:35.705422 27684 raft_consensus.cc:2241] T 1223743918814d88b401782ca4b521dd P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:35.705672 27684 raft_consensus.cc:2270] T 1223743918814d88b401782ca4b521dd P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:35.706012 27062 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 793d6157927143d4b9d8dbafd7cf0fcb (table default.Table_e79f52615fe142b9823a2a99b8b9c406 [id=3432f6ac076b4e49b9fe63bf5fce8591]). Expected version 1 got 0
I20250903 20:05:35.690536 27695 ts_tablet_manager.cc:1397] T 24eadc70e10d4d02bcb643d45c9e0e36 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.149s user 0.005s sys 0.000s
I20250903 20:05:35.706867 27684 ts_tablet_manager.cc:1905] T 1223743918814d88b401782ca4b521dd P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:35.708256 27695 raft_consensus.cc:357] T 24eadc70e10d4d02bcb643d45c9e0e36 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.708600 27695 raft_consensus.cc:383] T 24eadc70e10d4d02bcb643d45c9e0e36 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.708688 27695 raft_consensus.cc:738] T 24eadc70e10d4d02bcb643d45c9e0e36 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.709263 27360 tablet_service.cc:1468] Processing CreateTablet for tablet b27494321cc144b0a875b128813c74c7 (DEFAULT_TABLE table=default.Table_aab525602fac468990c81cb00e05f1ba [id=ed75409b6d7440a9bcc76c6445d4cd9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.709352 27695 consensus_queue.cc:260] T 24eadc70e10d4d02bcb643d45c9e0e36 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.710539 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b27494321cc144b0a875b128813c74c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.717471 27216 tablet_service.cc:1468] Processing CreateTablet for tablet b27494321cc144b0a875b128813c74c7 (DEFAULT_TABLE table=default.Table_aab525602fac468990c81cb00e05f1ba [id=ed75409b6d7440a9bcc76c6445d4cd9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.718669 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 6ec346392c4d4260b98b4acacdd14bbb (DEFAULT_TABLE table=default.Table_9c1562dad48f49b59b791ce173464002 [id=300d59bcd84a466893fd7f2ea45d891a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.719651 27349 tablet_service.cc:1468] Processing CreateTablet for tablet 07dd51da81d444fea3aebaffb871e361 (DEFAULT_TABLE table=default.Table_fc0ceede2760485d94134d2b4532cb9a [id=8ec0126253f942f2b0e9a22cec35b734]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.720106 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07dd51da81d444fea3aebaffb871e361. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.721065 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 07dd51da81d444fea3aebaffb871e361 (DEFAULT_TABLE table=default.Table_fc0ceede2760485d94134d2b4532cb9a [id=8ec0126253f942f2b0e9a22cec35b734]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.722644 27350 tablet_service.cc:1468] Processing CreateTablet for tablet 765ebbbc3d5e4f8397c244b663703c0d (DEFAULT_TABLE table=default.Table_fbe1657165754851afaaadbf3028cc72 [id=41d3834483744a428aee6ac6416449de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.722746 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ec346392c4d4260b98b4acacdd14bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.723008 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 765ebbbc3d5e4f8397c244b663703c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.729523 27684 ts_tablet_manager.cc:1918] T 1223743918814d88b401782ca4b521dd P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:35.729700 27684 log.cc:1199] T 1223743918814d88b401782ca4b521dd P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/1223743918814d88b401782ca4b521dd
I20250903 20:05:35.731426 27684 ts_tablet_manager.cc:1939] T 1223743918814d88b401782ca4b521dd P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:35.732519 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1083) took 1377 ms. Trace:
I20250903 20:05:35.732110 29328 tablet.cc:1722] T 06a79706f1184fa49d10182ba669b0ab P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:35.732635 27684 rpcz_store.cc:276] 0903 20:05:34.355252 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.355407 (+ 155us) service_pool.cc:225] Handling call
0903 20:05:35.705092 (+1349685us) ts_tablet_manager.cc:1145] Deleting tablet 1223743918814d88b401782ca4b521dd
0903 20:05:35.705108 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:35.712793 (+ 7685us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.729353 (+ 16560us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.732493 (+ 3140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1349454}
I20250903 20:05:35.735050 28567 raft_consensus.cc:491] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:35.735211 28567 raft_consensus.cc:513] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.735896 28567 leader_election.cc:290] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:35.734185 27062 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 8ba2636641654d8596fd2a5b0644d073 (table default.Table_e7148e16606e4cd3908c86b439f75b78 [id=4bffa29ea98049718ac3ce35de9813c5]). Expected version 1 got 0
I20250903 20:05:35.736646 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24eadc70e10d4d02bcb643d45c9e0e36" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:35.736889 27248 raft_consensus.cc:2466] T 24eadc70e10d4d02bcb643d45c9e0e36 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
W20250903 20:05:35.737072 27320 consensus_peers.cc:489] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1223743918814d88b401782ca4b521dd. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:35.737517 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 1598824a733a4a93bed043e81b86abf3
I20250903 20:05:35.737521 27321 leader_election.cc:304] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:35.737843 29285 raft_consensus.cc:491] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:35.737979 29285 raft_consensus.cc:513] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.737715 27692 tablet_bootstrap.cc:492] T 793d6157927143d4b9d8dbafd7cf0fcb P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:35.738755 29285 leader_election.cc:290] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:35.739396 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet 1598824a733a4a93bed043e81b86abf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:35 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:35.739748 28567 raft_consensus.cc:2802] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:35.739615 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24eadc70e10d4d02bcb643d45c9e0e36" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:35.739833 28567 raft_consensus.cc:491] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:35.739903 28567 raft_consensus.cc:3058] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:35.740003 27183 leader_election.cc:343] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 45f7068bafaf48d290ded9d1272c0e46
I20250903 20:05:35.740000 27516 raft_consensus.cc:2466] T 24eadc70e10d4d02bcb643d45c9e0e36 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
W20250903 20:05:35.740286 27184 leader_election.cc:343] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 45f7068bafaf48d290ded9d1272c0e46
I20250903 20:05:35.740397 27184 leader_election.cc:304] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:35.740800 29285 raft_consensus.cc:2747] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:35.740669 27692 ts_tablet_manager.cc:1397] T 793d6157927143d4b9d8dbafd7cf0fcb P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.109s user 0.002s sys 0.002s
I20250903 20:05:35.743016 28567 raft_consensus.cc:513] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.745080 28567 leader_election.cc:290] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:35.746567 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24eadc70e10d4d02bcb643d45c9e0e36" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:35.746857 27516 raft_consensus.cc:3058] T 24eadc70e10d4d02bcb643d45c9e0e36 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:35.749087 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24eadc70e10d4d02bcb643d45c9e0e36" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:35.749460 27684 tablet_replica.cc:331] T 548b50819f694fa3ad508da9f442a99c P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:35.749437 27243 raft_consensus.cc:3058] T 24eadc70e10d4d02bcb643d45c9e0e36 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:35.749768 27684 raft_consensus.cc:2241] T 548b50819f694fa3ad508da9f442a99c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:35.750058 27684 raft_consensus.cc:2270] T 548b50819f694fa3ad508da9f442a99c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:35.751165 27684 ts_tablet_manager.cc:1905] T 548b50819f694fa3ad508da9f442a99c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:35.754122 27243 raft_consensus.cc:2466] T 24eadc70e10d4d02bcb643d45c9e0e36 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:35.754276 27516 raft_consensus.cc:2466] T 24eadc70e10d4d02bcb643d45c9e0e36 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:35.754912 27320 leader_election.cc:304] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:35.755424 29227 raft_consensus.cc:2802] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:35.755568 29227 raft_consensus.cc:695] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:35.755852 29227 consensus_queue.cc:237] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.756191 27695 ts_tablet_manager.cc:1428] T 24eadc70e10d4d02bcb643d45c9e0e36 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.050s user 0.003s sys 0.000s
I20250903 20:05:35.756917 27695 tablet_bootstrap.cc:492] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:35.760637 29221 raft_consensus.cc:491] T ee40566e497740789575275d0cdca0db P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:35.760869 29221 raft_consensus.cc:513] T ee40566e497740789575275d0cdca0db P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.761799 29221 leader_election.cc:290] T ee40566e497740789575275d0cdca0db P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:35.763633 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ee40566e497740789575275d0cdca0db" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:35.764359 27462 leader_election.cc:343] T ee40566e497740789575275d0cdca0db P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:35.772168 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 4440a6c2afb247e0a58a83806cf5b334
I20250903 20:05:35.774412 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet 4440a6c2afb247e0a58a83806cf5b334 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:35 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:35.774688 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 4440a6c2afb247e0a58a83806cf5b334 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:35 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:35.774940 27350 tablet_service.cc:1515] Processing DeleteTablet for tablet 1598824a733a4a93bed043e81b86abf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:35 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:35.775790 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ee40566e497740789575275d0cdca0db" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:35.776563 27462 leader_election.cc:343] T ee40566e497740789575275d0cdca0db P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:35.776731 27216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b27494321cc144b0a875b128813c74c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.776757 27462 leader_election.cc:304] T ee40566e497740789575275d0cdca0db P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:35.776925 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ec346392c4d4260b98b4acacdd14bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.777112 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bde699e79b3b4911823641275bec96dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.777320 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07dd51da81d444fea3aebaffb871e361. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.777325 29220 raft_consensus.cc:2747] T ee40566e497740789575275d0cdca0db P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:35.782368 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2da6c143daa6410780a2c32f97b84ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.783979 27695 tablet_bootstrap.cc:654] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.784020 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet 09fa32312c9648fbbd6fd327e9738066 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:35.784454 27692 raft_consensus.cc:357] T 793d6157927143d4b9d8dbafd7cf0fcb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.784677 27692 raft_consensus.cc:383] T 793d6157927143d4b9d8dbafd7cf0fcb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.784773 27692 raft_consensus.cc:738] T 793d6157927143d4b9d8dbafd7cf0fcb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.785039 27692 consensus_queue.cc:260] T 793d6157927143d4b9d8dbafd7cf0fcb P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.785650 27692 ts_tablet_manager.cc:1428] T 793d6157927143d4b9d8dbafd7cf0fcb P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.030s user 0.000s sys 0.001s
I20250903 20:05:35.789505 29331 tablet.cc:1722] T ac1b0dc95fca49b4af6697c0a70e9730 P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250903 20:05:35.792021 29321 tablet.cc:1722] T ac1b0dc95fca49b4af6697c0a70e9730 P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250903 20:05:35.810729 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 7baeb72c8bd440c39c07f34a6cb5e3ea
I20250903 20:05:35.811196 27066 catalog_manager.cc:6652] ReplaceTablet: tablet 7baeb72c8bd440c39c07f34a6cb5e3ea deleted and replaced by tablet ce87a2b6f3a44d3c8ae3b39d158ddb33
I20250903 20:05:35.811656 27687 ts_tablet_manager.cc:1918] T f1d22e76ea72441db51f7e9db26eb520 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:35.811877 27687 log.cc:1199] T f1d22e76ea72441db51f7e9db26eb520 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/f1d22e76ea72441db51f7e9db26eb520
I20250903 20:05:35.812137 27350 tablet_service.cc:1515] Processing DeleteTablet for tablet 7baeb72c8bd440c39c07f34a6cb5e3ea with delete_type TABLET_DATA_DELETED (replaced by tablet ce87a2b6f3a44d3c8ae3b39d158ddb33) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:35.812460 27214 tablet_service.cc:1515] Processing DeleteTablet for tablet 7baeb72c8bd440c39c07f34a6cb5e3ea with delete_type TABLET_DATA_DELETED (replaced by tablet ce87a2b6f3a44d3c8ae3b39d158ddb33) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:35.812696 27051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.125:44059, user_credentials={real_user=slave}} blocked reactor thread for 112559us
I20250903 20:05:35.813146 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 950a6fcd35f64f2d889831fe9cdf503a (table default.Table_7dae8ff5ca9744708d223312079a4315 [id=7032fe3b789041d380f5db114f241e29]) successfully deleted
I20250903 20:05:35.813374 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 1223743918814d88b401782ca4b521dd (table default.Table_f038423efb3f4fd5bba9911713436b97 [id=22a7d6c72c2e4dbfadd44a229bb175e3]) successfully deleted
I20250903 20:05:35.813925 27057 master_service.cc:948] ReplaceTablet: received request to replace tablet b437eca9eb0b4c60bf797121a5a8bad8 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:35.816459 27496 tablet_service.cc:1468] Processing CreateTablet for tablet 2da6c143daa6410780a2c32f97b84ae2 (DEFAULT_TABLE table=default.Table_021085b0fd254dbf961aa3d9aeadd6d6 [id=7f3821b85d7b45eabfe9957a159f57a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.816733 27493 tablet_service.cc:1468] Processing CreateTablet for tablet 07dd51da81d444fea3aebaffb871e361 (DEFAULT_TABLE table=default.Table_fc0ceede2760485d94134d2b4532cb9a [id=8ec0126253f942f2b0e9a22cec35b734]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.816793 27498 tablet_service.cc:1515] Processing DeleteTablet for tablet 1598824a733a4a93bed043e81b86abf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:35 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:35.816866 27500 tablet_service.cc:1468] Processing CreateTablet for tablet b27494321cc144b0a875b128813c74c7 (DEFAULT_TABLE table=default.Table_aab525602fac468990c81cb00e05f1ba [id=ed75409b6d7440a9bcc76c6445d4cd9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.817680 29330 tablet.cc:1722] T ac1b0dc95fca49b4af6697c0a70e9730 P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250903 20:05:35.817778 27498 tablet_service.cc:1515] Processing DeleteTablet for tablet 4440a6c2afb247e0a58a83806cf5b334 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:35 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:35.818751 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2da6c143daa6410780a2c32f97b84ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.819137 27692 tablet_bootstrap.cc:492] T d4c703dafbca4e669dbb18946ffac699 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
W20250903 20:05:35.820324 27051 consensus_peers.cc:489] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 -> Peer 35dd69f3fa7a4fd8843b97f8a5d68296 (127.26.65.125:44059): Couldn't send request to peer 35dd69f3fa7a4fd8843b97f8a5d68296. Status: Network error: Client connection negotiation failed: client connection to 127.26.65.125:44059: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250903 20:05:35.820940 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e02c690220fb4ab2a949d23f2230eeca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:35.821738 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e02c690220fb4ab2a949d23f2230eeca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:35.822458 27066 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 793d6157927143d4b9d8dbafd7cf0fcb (table default.Table_e79f52615fe142b9823a2a99b8b9c406 [id=3432f6ac076b4e49b9fe63bf5fce8591]). Expected version 1 got 0
I20250903 20:05:35.825982 27066 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 17f16e9c8f794e03a2f8b3b6b35f8e84 (table default.Table_c9ba436d242a498582b9009fa5d299e4 [id=d7a41e621fde497d8fcfb842c1e6f9e7]). Expected version 1 got 0
I20250903 20:05:35.826391 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_392e604d6bdd46bfbe7e3b6d21529019" } modify_external_catalogs: true
I20250903 20:05:35.827684 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_392e604d6bdd46bfbe7e3b6d21529019" } modify_external_catalogs: true
I20250903 20:05:35.829020 27695 tablet_bootstrap.cc:492] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:35.829234 27695 ts_tablet_manager.cc:1397] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.072s user 0.005s sys 0.000s
I20250903 20:05:35.826565 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_badd71a0f177471b805d80d3e5652d50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:35.826987 27059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6b71f7836da34d58a17dacbfb7bcac07" } modify_external_catalogs: true
I20250903 20:05:35.830034 27059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6b71f7836da34d58a17dacbfb7bcac07" } modify_external_catalogs: true
W20250903 20:05:35.830392 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_badd71a0f177471b805d80d3e5652d50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:35.827018 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_83abf243847040dc8165106073aaa138"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:35.831681 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83abf243847040dc8165106073aaa138 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:35.832393 27066 catalog_manager.cc:5582] T 472f0fb01263468a828ec13795656624 P e72ef1dbcc5143cf9f04a808f101c80f reported cstate change: term changed from 0 to 1, leader changed from <none> to e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67). New cstate: current_term: 1 leader_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:35.834502 27695 raft_consensus.cc:357] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.834867 27695 raft_consensus.cc:383] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.835059 27695 raft_consensus.cc:738] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.819700 27498 tablet_service.cc:1515] Processing DeleteTablet for tablet 7baeb72c8bd440c39c07f34a6cb5e3ea with delete_type TABLET_DATA_DELETED (replaced by tablet ce87a2b6f3a44d3c8ae3b39d158ddb33) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:35.817904 27693 tablet_bootstrap.cc:492] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:35.824189 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07dd51da81d444fea3aebaffb871e361. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.835497 27695 consensus_queue.cc:260] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.835703 27693 ts_tablet_manager.cc:1397] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.192s user 0.004s sys 0.000s
I20250903 20:05:35.835947 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b27494321cc144b0a875b128813c74c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.836586 27693 raft_consensus.cc:357] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.836740 27693 raft_consensus.cc:383] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.836783 27693 raft_consensus.cc:738] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.836984 27693 consensus_queue.cc:260] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.837421 27693 ts_tablet_manager.cc:1428] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:35.838289 27693 tablet_bootstrap.cc:492] T 4e34d6d5d68e4395aed78821ea077c1b P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:35.840389 27693 tablet_bootstrap.cc:654] T 4e34d6d5d68e4395aed78821ea077c1b P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.818403 27687 ts_tablet_manager.cc:1939] T f1d22e76ea72441db51f7e9db26eb520 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:35.842262 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1116) took 1378 ms. Trace:
I20250903 20:05:35.835028 27350 tablet_service.cc:1468] Processing CreateTablet for tablet d0dc42fba99f491e9a946714d0980218 (DEFAULT_TABLE table=default.Table_8f6632599c6148db883ee6c79d9d168b [id=eae2208912fe43db84c55f49ee626b69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.842837 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0dc42fba99f491e9a946714d0980218. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.842391 27687 rpcz_store.cc:276] 0903 20:05:34.463445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.463819 (+ 374us) service_pool.cc:225] Handling call
0903 20:05:35.580034 (+1116215us) ts_tablet_manager.cc:1145] Deleting tablet f1d22e76ea72441db51f7e9db26eb520
0903 20:05:35.580047 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:35.653475 (+ 73428us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.811533 (+158058us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.842037 (+ 30504us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1116080}
I20250903 20:05:35.843473 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet f1d22e76ea72441db51f7e9db26eb520 (table default.Table_258f358903dd4f6fa7e310d72562a570 [id=ae5e27c159a9462482d09a79e148f795]) successfully deleted
I20250903 20:05:35.843555 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1117) took 1380 ms. Trace:
I20250903 20:05:35.843619 27687 rpcz_store.cc:276] 0903 20:05:34.463551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.463970 (+ 419us) service_pool.cc:225] Handling call
0903 20:05:35.843419 (+1379449us) ts_tablet_manager.cc:1145] Deleting tablet a2358eac86914f3ba51bfc6c299838c3
0903 20:05:35.843429 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:35.843539 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1379332}
I20250903 20:05:35.843986 27687 tablet_replica.cc:331] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
W20250903 20:05:35.843976 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet a2358eac86914f3ba51bfc6c299838c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2358eac86914f3ba51bfc6c299838c3 already in progress: creating tablet
I20250903 20:05:35.844175 27687 raft_consensus.cc:2241] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:35.844421 27687 pending_rounds.cc:70] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528: Trying to abort 1 pending ops.
I20250903 20:05:35.844485 27687 pending_rounds.cc:77] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196385014480384000 op_type: NO_OP noop_request { }
I20250903 20:05:35.844592 27687 raft_consensus.cc:2887] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:35.844624 27685 ts_tablet_manager.cc:1918] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 20:05:35.844681 27687 raft_consensus.cc:2270] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:35.845533 27687 ts_tablet_manager.cc:1905] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:35.849581 27684 ts_tablet_manager.cc:1918] T 548b50819f694fa3ad508da9f442a99c P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:35.849836 27684 log.cc:1199] T 548b50819f694fa3ad508da9f442a99c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/548b50819f694fa3ad508da9f442a99c
I20250903 20:05:35.850025 27685 log.cc:1199] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/b0d79c78e0024b7ba3bb3356be7c8ceb
I20250903 20:05:35.850637 27684 ts_tablet_manager.cc:1939] T 548b50819f694fa3ad508da9f442a99c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:35.851079 27685 ts_tablet_manager.cc:1939] T b0d79c78e0024b7ba3bb3356be7c8ceb P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:35.853435 27692 tablet_bootstrap.cc:654] T d4c703dafbca4e669dbb18946ffac699 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.845073 27695 ts_tablet_manager.cc:1428] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20250903 20:05:35.850358 27222 tablet_service.cc:1468] Processing CreateTablet for tablet d0dc42fba99f491e9a946714d0980218 (DEFAULT_TABLE table=default.Table_8f6632599c6148db883ee6c79d9d168b [id=eae2208912fe43db84c55f49ee626b69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.857867 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0dc42fba99f491e9a946714d0980218. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.859776 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1084) took 1473 ms. Trace:
I20250903 20:05:35.859931 27684 rpcz_store.cc:276] 0903 20:05:34.386469 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.386522 (+ 53us) service_pool.cc:225] Handling call
0903 20:05:35.732974 (+1346452us) ts_tablet_manager.cc:1145] Deleting tablet 548b50819f694fa3ad508da9f442a99c
0903 20:05:35.749407 (+ 16433us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:35.816011 (+ 66604us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.849415 (+ 33404us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.859718 (+ 10303us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1346168}
I20250903 20:05:35.860332 27684 tablet_replica.cc:331] T 46f49fd718be4a2996d781ab31268217 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:35.859905 27695 tablet_bootstrap.cc:492] T f326ec776ba446be8d5dda5c1dc5b6ec P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:35.860591 27684 raft_consensus.cc:2241] T 46f49fd718be4a2996d781ab31268217 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:35.860805 27684 raft_consensus.cc:2270] T 46f49fd718be4a2996d781ab31268217 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:35.861469 27684 ts_tablet_manager.cc:1905] T 46f49fd718be4a2996d781ab31268217 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:35.862529 27695 tablet_bootstrap.cc:654] T f326ec776ba446be8d5dda5c1dc5b6ec P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.863289 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 963) took 3760 ms. Trace:
I20250903 20:05:35.864341 27685 rpcz_store.cc:276] 0903 20:05:32.102501 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.102977 (+ 476us) service_pool.cc:225] Handling call
0903 20:05:35.676805 (+3573828us) ts_tablet_manager.cc:1145] Deleting tablet b0d79c78e0024b7ba3bb3356be7c8ceb
0903 20:05:35.676817 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:35.806765 (+129948us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.844513 (+ 37748us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.863267 (+ 18754us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3573589}
I20250903 20:05:35.865024 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet b0d79c78e0024b7ba3bb3356be7c8ceb (table default.Table_94e9e32ef1134f07ade1cdf8da69ead6 [id=e248f9c4b9c9406395aaea7ba3feb9ba]) successfully deleted
I20250903 20:05:35.868036 27685 tablet_replica.cc:331] T 22bc87d1a4804b6c80f77f9692500d63 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:35.868539 27685 raft_consensus.cc:2241] T 22bc87d1a4804b6c80f77f9692500d63 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:35.868844 27685 raft_consensus.cc:2270] T 22bc87d1a4804b6c80f77f9692500d63 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:35.869554 27685 ts_tablet_manager.cc:1905] T 22bc87d1a4804b6c80f77f9692500d63 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:35.872567 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet c066691b278c4278937b4d8ef98b395f
I20250903 20:05:35.874378 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet c066691b278c4278937b4d8ef98b395f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:35 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:35.874764 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet c066691b278c4278937b4d8ef98b395f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:35 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:35.849967 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 765ebbbc3d5e4f8397c244b663703c0d (DEFAULT_TABLE table=default.Table_fbe1657165754851afaaadbf3028cc72 [id=41d3834483744a428aee6ac6416449de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.875960 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 765ebbbc3d5e4f8397c244b663703c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.876181 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d7af80d1b68748c99d85f6914f57b251" } new_table_name: "default.Table_a0858c440ad84423888fcb05f56ca68d" modify_external_catalogs: true
W20250903 20:05:35.879191 27320 consensus_peers.cc:489] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:35.880290 27695 tablet_bootstrap.cc:492] T f326ec776ba446be8d5dda5c1dc5b6ec P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:35.880451 27695 ts_tablet_manager.cc:1397] T f326ec776ba446be8d5dda5c1dc5b6ec P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20250903 20:05:35.882009 27695 raft_consensus.cc:357] T f326ec776ba446be8d5dda5c1dc5b6ec P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:35.882284 27695 raft_consensus.cc:383] T f326ec776ba446be8d5dda5c1dc5b6ec P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.882344 27695 raft_consensus.cc:738] T f326ec776ba446be8d5dda5c1dc5b6ec P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.882578 27695 consensus_queue.cc:260] T f326ec776ba446be8d5dda5c1dc5b6ec P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:35.884847 27695 ts_tablet_manager.cc:1428] T f326ec776ba446be8d5dda5c1dc5b6ec P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250903 20:05:35.886305 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_d73ecd1c981c4cfdbebd10521a2b86eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:35.887066 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d73ecd1c981c4cfdbebd10521a2b86eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:35.889036 27695 tablet_bootstrap.cc:492] T a3103e1595ec4c38ba65dca26fe1167c P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:35.891605 27493 tablet_service.cc:1468] Processing CreateTablet for tablet 765ebbbc3d5e4f8397c244b663703c0d (DEFAULT_TABLE table=default.Table_fbe1657165754851afaaadbf3028cc72 [id=41d3834483744a428aee6ac6416449de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.892791 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 765ebbbc3d5e4f8397c244b663703c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.894526 27065 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_9ea78ec0f6314be48893b91681dd50f3 [id=6952319d78524829b3bc6c3de6425a67] alter complete (version 3)
I20250903 20:05:35.896389 27695 tablet_bootstrap.cc:654] T a3103e1595ec4c38ba65dca26fe1167c P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.888718 27692 tablet_bootstrap.cc:492] T d4c703dafbca4e669dbb18946ffac699 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:35.906874 27692 ts_tablet_manager.cc:1397] T d4c703dafbca4e669dbb18946ffac699 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.088s user 0.000s sys 0.004s
I20250903 20:05:35.908331 27692 raft_consensus.cc:357] T d4c703dafbca4e669dbb18946ffac699 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.911132 29227 raft_consensus.cc:491] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:35.911305 29227 raft_consensus.cc:513] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.911985 29227 leader_election.cc:290] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:35.912838 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09fa32312c9648fbbd6fd327e9738066" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:35.913046 27695 tablet_bootstrap.cc:492] T a3103e1595ec4c38ba65dca26fe1167c P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:35.913209 27695 ts_tablet_manager.cc:1397] T a3103e1595ec4c38ba65dca26fe1167c P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.003s
I20250903 20:05:35.913055 27243 raft_consensus.cc:2466] T 09fa32312c9648fbbd6fd327e9738066 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:35.914170 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet b437eca9eb0b4c60bf797121a5a8bad8
I20250903 20:05:35.914083 27695 raft_consensus.cc:357] T a3103e1595ec4c38ba65dca26fe1167c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.914325 27695 raft_consensus.cc:383] T a3103e1595ec4c38ba65dca26fe1167c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.914390 27695 raft_consensus.cc:738] T a3103e1595ec4c38ba65dca26fe1167c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.914618 27695 consensus_queue.cc:260] T a3103e1595ec4c38ba65dca26fe1167c P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.914741 27057 catalog_manager.cc:6652] ReplaceTablet: tablet b437eca9eb0b4c60bf797121a5a8bad8 deleted and replaced by tablet 7fda4c6cb0504d89bb94f1dc78ffc59b
I20250903 20:05:35.914768 27685 ts_tablet_manager.cc:1918] T 22bc87d1a4804b6c80f77f9692500d63 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:35.914783 27321 leader_election.cc:304] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:35.914870 27685 log.cc:1199] T 22bc87d1a4804b6c80f77f9692500d63 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/22bc87d1a4804b6c80f77f9692500d63
I20250903 20:05:35.915334 27685 ts_tablet_manager.cc:1939] T 22bc87d1a4804b6c80f77f9692500d63 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:35.915746 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet b437eca9eb0b4c60bf797121a5a8bad8 with delete_type TABLET_DATA_DELETED (replaced by tablet 7fda4c6cb0504d89bb94f1dc78ffc59b) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:35.916242 29227 raft_consensus.cc:2802] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:35.916332 29227 raft_consensus.cc:491] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:35.916411 29227 raft_consensus.cc:3058] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:35.919206 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ff9d1c55236648f295a6715fd27166f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:35.919385 29227 raft_consensus.cc:513] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:35.919976 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff9d1c55236648f295a6715fd27166f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:35.921684 29227 leader_election.cc:290] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:35.922391 27361 tablet_service.cc:1468] Processing CreateTablet for tablet bc568d43339d4674829e36b8918eb969 (DEFAULT_TABLE table=default.Table_3838c77991ba4638b506d7cc7fc43fe9 [id=d3bf0847c48a4eceb094f09c3d2e1605]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.922941 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc568d43339d4674829e36b8918eb969. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.913846 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09fa32312c9648fbbd6fd327e9738066" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:35.923177 27692 raft_consensus.cc:383] T d4c703dafbca4e669dbb18946ffac699 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.923244 27516 raft_consensus.cc:2466] T 09fa32312c9648fbbd6fd327e9738066 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:35.923293 27692 raft_consensus.cc:738] T d4c703dafbca4e669dbb18946ffac699 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.920066 29220 raft_consensus.cc:491] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:35.923595 29220 raft_consensus.cc:513] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:35.923568 27692 consensus_queue.cc:260] T d4c703dafbca4e669dbb18946ffac699 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.924149 27692 ts_tablet_manager.cc:1428] T d4c703dafbca4e669dbb18946ffac699 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.017s user 0.000s sys 0.001s
I20250903 20:05:35.924223 29220 leader_election.cc:290] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:35.915969 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet b437eca9eb0b4c60bf797121a5a8bad8 with delete_type TABLET_DATA_DELETED (replaced by tablet 7fda4c6cb0504d89bb94f1dc78ffc59b) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:35.916529 27695 ts_tablet_manager.cc:1428] T a3103e1595ec4c38ba65dca26fe1167c P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:35.925335 27695 tablet_bootstrap.cc:492] T 4e34d6d5d68e4395aed78821ea077c1b P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:35.921532 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09fa32312c9648fbbd6fd327e9738066" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:35.925400 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb2f50683e904dd39ee6eb128717d38c" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:35.925521 27243 raft_consensus.cc:3058] T 09fa32312c9648fbbd6fd327e9738066 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:35.925951 27462 leader_election.cc:343] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:35.926080 27687 ts_tablet_manager.cc:1918] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:35.926180 27687 log.cc:1199] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/5557dfdd953a41e9bd3a8be4cd689604
I20250903 20:05:35.926561 27687 ts_tablet_manager.cc:1939] T 5557dfdd953a41e9bd3a8be4cd689604 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:35.927197 27695 tablet_bootstrap.cc:654] T 4e34d6d5d68e4395aed78821ea077c1b P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.927625 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb2f50683e904dd39ee6eb128717d38c" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:35.927809 27381 raft_consensus.cc:2466] T cb2f50683e904dd39ee6eb128717d38c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:35.928577 27243 raft_consensus.cc:2466] T 09fa32312c9648fbbd6fd327e9738066 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:35.929075 27321 leader_election.cc:304] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:35.929652 27462 leader_election.cc:304] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:35.930099 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 974) took 3458 ms. Trace:
I20250903 20:05:35.930190 29220 raft_consensus.cc:2802] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:35.930334 29220 raft_consensus.cc:491] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:35.930323 29227 raft_consensus.cc:2802] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:35.930446 29220 raft_consensus.cc:3058] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:35.930462 29227 raft_consensus.cc:695] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:35.930763 29227 consensus_queue.cc:237] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.921532 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09fa32312c9648fbbd6fd327e9738066" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:35.932003 27511 raft_consensus.cc:3058] T 09fa32312c9648fbbd6fd327e9738066 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:35.930424 27685 rpcz_store.cc:276] 0903 20:05:32.471189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.471236 (+ 47us) service_pool.cc:225] Handling call
0903 20:05:35.865411 (+3394175us) ts_tablet_manager.cc:1145] Deleting tablet 22bc87d1a4804b6c80f77f9692500d63
0903 20:05:35.867927 (+ 2516us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:35.901619 (+ 33692us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.914643 (+ 13024us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.930072 (+ 15429us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3393974}
I20250903 20:05:35.932580 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 22bc87d1a4804b6c80f77f9692500d63 (table default.Table_c967f8b0d1314f589eae529cc492493b [id=93b379ddcb834087a42ce197c318f764]) successfully deleted
I20250903 20:05:35.932966 27685 tablet_replica.cc:331] T b7a6d5d8a97a4769a81ec7224d591f8c P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:35.933182 27685 raft_consensus.cc:2241] T b7a6d5d8a97a4769a81ec7224d591f8c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:35.933483 27685 raft_consensus.cc:2270] T b7a6d5d8a97a4769a81ec7224d591f8c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:35.934150 29220 raft_consensus.cc:513] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:35.934427 27685 ts_tablet_manager.cc:1905] T b7a6d5d8a97a4769a81ec7224d591f8c P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:35.935032 29220 leader_election.cc:290] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:35.935810 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb2f50683e904dd39ee6eb128717d38c" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:35.936059 27381 raft_consensus.cc:3058] T cb2f50683e904dd39ee6eb128717d38c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:35.936362 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d6d3ee9322e745f6843403131aa0f8a5" } new_table_name: "default.Table_1124e9ecb8de4091bb37e8cc347dc8ab" modify_external_catalogs: true
I20250903 20:05:35.935261 27511 raft_consensus.cc:2466] T 09fa32312c9648fbbd6fd327e9738066 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:35.939560 27381 raft_consensus.cc:2466] T cb2f50683e904dd39ee6eb128717d38c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:35.939859 27692 tablet_bootstrap.cc:492] T 39c2ebe4ada5426590ec3769c8662644 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:35.940325 27462 leader_election.cc:304] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:35.940939 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb2f50683e904dd39ee6eb128717d38c" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
W20250903 20:05:35.941689 27462 leader_election.cc:343] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:35.942173 29220 raft_consensus.cc:2802] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:35.942502 29220 raft_consensus.cc:695] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:35.942869 29220 consensus_queue.cc:237] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:35.946375 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 435c002dffd24aeaa6221ab00ee5b24b
I20250903 20:05:35.947489 27500 tablet_service.cc:1468] Processing CreateTablet for tablet d0dc42fba99f491e9a946714d0980218 (DEFAULT_TABLE table=default.Table_8f6632599c6148db883ee6c79d9d168b [id=eae2208912fe43db84c55f49ee626b69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.947867 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 435c002dffd24aeaa6221ab00ee5b24b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:35 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:35.948019 27498 tablet_service.cc:1515] Processing DeleteTablet for tablet c066691b278c4278937b4d8ef98b395f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:35 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:35.948302 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 435c002dffd24aeaa6221ab00ee5b24b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:35 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:35.948560 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 548b50819f694fa3ad508da9f442a99c (table default.Table_b838fda30d4043a5ad578ec527c1d105 [id=1eb6b859ca0049699466cc72e5a2b6f7]) successfully deleted
I20250903 20:05:35.949234 27498 tablet_service.cc:1515] Processing DeleteTablet for tablet b437eca9eb0b4c60bf797121a5a8bad8 with delete_type TABLET_DATA_DELETED (replaced by tablet 7fda4c6cb0504d89bb94f1dc78ffc59b) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:35.950790 27498 tablet_service.cc:1515] Processing DeleteTablet for tablet 435c002dffd24aeaa6221ab00ee5b24b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:35 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:35.950798 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_dbeec832140d416aa6667b1958c1abf2" } new_table_name: "default.Table_519f51d4b7354368a5ef5fcd9312f525" modify_external_catalogs: true
I20250903 20:05:35.953415 27216 tablet_service.cc:1468] Processing CreateTablet for tablet 6154aac026f446aca9fffe12a6b0c6a4 (DEFAULT_TABLE table=default.Table_fae7d62089f24ca88970b4691b51f867 [id=bd8f3bd183644082856d3862b55ea3f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.953905 27216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6154aac026f446aca9fffe12a6b0c6a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.954401 27215 tablet_service.cc:1468] Processing CreateTablet for tablet bc568d43339d4674829e36b8918eb969 (DEFAULT_TABLE table=default.Table_3838c77991ba4638b506d7cc7fc43fe9 [id=d3bf0847c48a4eceb094f09c3d2e1605]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.954744 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc568d43339d4674829e36b8918eb969. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.955474 27685 ts_tablet_manager.cc:1918] T b7a6d5d8a97a4769a81ec7224d591f8c P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:35.955538 27692 tablet_bootstrap.cc:654] T 39c2ebe4ada5426590ec3769c8662644 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.955575 27685 log.cc:1199] T b7a6d5d8a97a4769a81ec7224d591f8c P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/b7a6d5d8a97a4769a81ec7224d591f8c
I20250903 20:05:35.955932 27685 ts_tablet_manager.cc:1939] T b7a6d5d8a97a4769a81ec7224d591f8c P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:35.956717 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 976) took 3484 ms. Trace:
I20250903 20:05:35.956801 27685 rpcz_store.cc:276] 0903 20:05:32.472117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.472164 (+ 47us) service_pool.cc:225] Handling call
0903 20:05:35.932902 (+3460738us) ts_tablet_manager.cc:1145] Deleting tablet b7a6d5d8a97a4769a81ec7224d591f8c
0903 20:05:35.932915 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:35.948791 (+ 15876us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.955375 (+ 6584us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.956694 (+ 1319us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3460566}
I20250903 20:05:35.957324 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet b7a6d5d8a97a4769a81ec7224d591f8c (table default.Table_f752a52fa3ba4ac99bc29428fa871318 [id=21b4fd8022f6461691f87200e040e264]) successfully deleted
I20250903 20:05:35.958596 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0dc42fba99f491e9a946714d0980218. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.961666 27685 tablet_replica.cc:331] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:35.961692 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 7bb0da31c68440788dc296e67b38ec7c
I20250903 20:05:35.961884 27685 raft_consensus.cc:2241] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:35.962338 27685 raft_consensus.cc:2270] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:35.962924 27685 ts_tablet_manager.cc:1905] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:35.963537 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 7bb0da31c68440788dc296e67b38ec7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:35 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:35.965406 27349 tablet_service.cc:1468] Processing CreateTablet for tablet 6154aac026f446aca9fffe12a6b0c6a4 (DEFAULT_TABLE table=default.Table_fae7d62089f24ca88970b4691b51f867 [id=bd8f3bd183644082856d3862b55ea3f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.965701 27347 tablet_service.cc:1468] Processing CreateTablet for tablet 68363657d04842d9882941dc85853902 (DEFAULT_TABLE table=default.Table_559f9c5a607c4af7b3ff4c1094d48005 [id=fe584fcd12a9468596defcbb9411218e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.965798 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6154aac026f446aca9fffe12a6b0c6a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.966023 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68363657d04842d9882941dc85853902. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.966429 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 620a34d0cac94d218d56ab455cf2250b (DEFAULT_TABLE table=default.Table_aaa4d127435d4fea8862bda75da34909 [id=7e8d6817439546108262bc846df54e8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.966699 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 68363657d04842d9882941dc85853902 (DEFAULT_TABLE table=default.Table_559f9c5a607c4af7b3ff4c1094d48005 [id=fe584fcd12a9468596defcbb9411218e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.966785 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 620a34d0cac94d218d56ab455cf2250b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:35.966979 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68363657d04842d9882941dc85853902. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:35.970528 27052 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 9942bb4b868d4a08866298d7790ecad2 (table default.Table_c86263c481ce4c5ebeb98439c682f1a2 [id=fb1b537879be4681a7f1a764a3ec6914], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9942bb4b868d4a08866298d7790ecad2 has no consensus state
I20250903 20:05:35.975688 28567 consensus_queue.cc:1035] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:35.977975 27221 tablet_service.cc:1515] Processing DeleteTablet for tablet 7bb0da31c68440788dc296e67b38ec7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:35 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:35.978406 28567 consensus_queue.cc:1035] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:35.981441 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 620a34d0cac94d218d56ab455cf2250b (DEFAULT_TABLE table=default.Table_aaa4d127435d4fea8862bda75da34909 [id=7e8d6817439546108262bc846df54e8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:35.983158 27695 tablet_bootstrap.cc:492] T 4e34d6d5d68e4395aed78821ea077c1b P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:35.983206 27693 tablet_bootstrap.cc:492] T 4e34d6d5d68e4395aed78821ea077c1b P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:35.983335 27693 ts_tablet_manager.cc:1397] T 4e34d6d5d68e4395aed78821ea077c1b P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.145s user 0.006s sys 0.002s
I20250903 20:05:35.983314 27695 ts_tablet_manager.cc:1397] T 4e34d6d5d68e4395aed78821ea077c1b P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.002s
I20250903 20:05:35.984906 27695 raft_consensus.cc:357] T 4e34d6d5d68e4395aed78821ea077c1b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.985100 27695 raft_consensus.cc:383] T 4e34d6d5d68e4395aed78821ea077c1b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.985031 27693 raft_consensus.cc:357] T 4e34d6d5d68e4395aed78821ea077c1b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.985154 27695 raft_consensus.cc:738] T 4e34d6d5d68e4395aed78821ea077c1b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.985194 27693 raft_consensus.cc:383] T 4e34d6d5d68e4395aed78821ea077c1b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:35.985255 27693 raft_consensus.cc:738] T 4e34d6d5d68e4395aed78821ea077c1b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:35.985476 27693 consensus_queue.cc:260] T 4e34d6d5d68e4395aed78821ea077c1b P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.985591 27695 consensus_queue.cc:260] T 4e34d6d5d68e4395aed78821ea077c1b P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:35.986343 27693 ts_tablet_manager.cc:1428] T 4e34d6d5d68e4395aed78821ea077c1b P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250903 20:05:35.986843 27693 tablet_bootstrap.cc:492] T 2385d3006f90486890ebe43e16a53d7a P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:35.986891 27695 ts_tablet_manager.cc:1428] T 4e34d6d5d68e4395aed78821ea077c1b P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250903 20:05:35.987195 27695 tablet_bootstrap.cc:492] T ee40566e497740789575275d0cdca0db P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:35.990413 27695 tablet_bootstrap.cc:654] T ee40566e497740789575275d0cdca0db P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:35.998963 27695 tablet_bootstrap.cc:492] T ee40566e497740789575275d0cdca0db P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:35.999292 27695 ts_tablet_manager.cc:1397] T ee40566e497740789575275d0cdca0db P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.003s
I20250903 20:05:36.000209 27695 raft_consensus.cc:357] T ee40566e497740789575275d0cdca0db P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.000396 27695 raft_consensus.cc:383] T ee40566e497740789575275d0cdca0db P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:36.000460 27695 raft_consensus.cc:738] T ee40566e497740789575275d0cdca0db P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:36.000833 27685 ts_tablet_manager.cc:1918] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:36.001489 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1114) took 1538 ms. Trace:
I20250903 20:05:36.001647 27687 rpcz_store.cc:276] 0903 20:05:34.463239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.464735 (+ 1496us) service_pool.cc:225] Handling call
0903 20:05:35.843938 (+1379203us) ts_tablet_manager.cc:1145] Deleting tablet 5557dfdd953a41e9bd3a8be4cd689604
0903 20:05:35.843947 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:35.848609 (+ 4662us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:35.925963 (+ 77354us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.001462 (+ 75499us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1379026}
I20250903 20:05:36.002324 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 5557dfdd953a41e9bd3a8be4cd689604 (table default.Table_672b60689df64b1d8c9ebd27468d68d4 [id=6d5615e3adc24110a36e1486e69cde7b]) successfully deleted
I20250903 20:05:36.003639 27684 ts_tablet_manager.cc:1918] T 46f49fd718be4a2996d781ab31268217 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:36.005137 27693 tablet_bootstrap.cc:654] T 2385d3006f90486890ebe43e16a53d7a P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:36.006024 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_91cfcef3088c4292bdc92b3cac245994"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:36.006767 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91cfcef3088c4292bdc92b3cac245994 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:36.009485 27684 log.cc:1199] T 46f49fd718be4a2996d781ab31268217 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/46f49fd718be4a2996d781ab31268217
I20250903 20:05:36.010144 27684 ts_tablet_manager.cc:1939] T 46f49fd718be4a2996d781ab31268217 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:36.010143 27685 log.cc:1199] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/1bf69315f07a470bb2b897ad7c590729
I20250903 20:05:36.010802 27685 ts_tablet_manager.cc:1939] T 1bf69315f07a470bb2b897ad7c590729 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:36.011905 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 975) took 3539 ms. Trace:
I20250903 20:05:36.012106 27685 rpcz_store.cc:276] 0903 20:05:32.471998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.472364 (+ 366us) service_pool.cc:225] Handling call
0903 20:05:35.961597 (+3489233us) ts_tablet_manager.cc:1145] Deleting tablet 1bf69315f07a470bb2b897ad7c590729
0903 20:05:35.961610 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:35.980415 (+ 18805us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.000688 (+ 20273us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.011878 (+ 11190us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3489098}
I20250903 20:05:36.012799 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 1bf69315f07a470bb2b897ad7c590729 (table default.Table_d6ccf445d4a743398a91f41658b38f8b [id=b82c8f8f0f6b415580c2b590f05c0f7a]) successfully deleted
I20250903 20:05:36.013224 27685 tablet_replica.cc:331] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:36.013535 27685 raft_consensus.cc:2241] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:36.013805 27685 raft_consensus.cc:2270] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:36.014600 27685 ts_tablet_manager.cc:1905] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:36.017576 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ea5cb66a574f4466bf7e7f635f22b426" } new_table_name: "default.Table_9f63732bf0ff4219886e8692abc459ce" modify_external_catalogs: true
I20250903 20:05:36.011108 27692 tablet_bootstrap.cc:492] T 39c2ebe4ada5426590ec3769c8662644 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:36.013118 27500 tablet_service.cc:1468] Processing CreateTablet for tablet bc568d43339d4674829e36b8918eb969 (DEFAULT_TABLE table=default.Table_3838c77991ba4638b506d7cc7fc43fe9 [id=d3bf0847c48a4eceb094f09c3d2e1605]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.019642 27692 ts_tablet_manager.cc:1397] T 39c2ebe4ada5426590ec3769c8662644 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.080s user 0.004s sys 0.000s
I20250903 20:05:36.020519 27692 raft_consensus.cc:357] T 39c2ebe4ada5426590ec3769c8662644 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.020707 27692 raft_consensus.cc:383] T 39c2ebe4ada5426590ec3769c8662644 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:36.020790 27692 raft_consensus.cc:738] T 39c2ebe4ada5426590ec3769c8662644 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:36.021361 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc568d43339d4674829e36b8918eb969. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.023015 27498 tablet_service.cc:1515] Processing DeleteTablet for tablet 7bb0da31c68440788dc296e67b38ec7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:35 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:36.023871 29285 raft_consensus.cc:491] T f326ec776ba446be8d5dda5c1dc5b6ec P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:36.024004 29285 raft_consensus.cc:513] T f326ec776ba446be8d5dda5c1dc5b6ec P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:36.024116 27498 tablet_service.cc:1468] Processing CreateTablet for tablet 6154aac026f446aca9fffe12a6b0c6a4 (DEFAULT_TABLE table=default.Table_fae7d62089f24ca88970b4691b51f867 [id=bd8f3bd183644082856d3862b55ea3f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.024499 27498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6154aac026f446aca9fffe12a6b0c6a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.024622 29285 leader_election.cc:290] T f326ec776ba446be8d5dda5c1dc5b6ec P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:36.025838 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f326ec776ba446be8d5dda5c1dc5b6ec" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:36.026046 27516 raft_consensus.cc:2466] T f326ec776ba446be8d5dda5c1dc5b6ec P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:36.026394 29329 raft_consensus.cc:491] T f326ec776ba446be8d5dda5c1dc5b6ec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:36.026463 27183 leader_election.cc:304] T f326ec776ba446be8d5dda5c1dc5b6ec P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:36.026539 29329 raft_consensus.cc:513] T f326ec776ba446be8d5dda5c1dc5b6ec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:36.026765 29199 raft_consensus.cc:2802] T f326ec776ba446be8d5dda5c1dc5b6ec P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:36.026837 29199 raft_consensus.cc:491] T f326ec776ba446be8d5dda5c1dc5b6ec P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:36.026896 29199 raft_consensus.cc:3058] T f326ec776ba446be8d5dda5c1dc5b6ec P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:36.027109 29329 leader_election.cc:290] T f326ec776ba446be8d5dda5c1dc5b6ec P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
W20250903 20:05:36.014827 27320 consensus_peers.cc:489] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed096b452ba747f89e91edc15e6e3ac1. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:36.025240 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f326ec776ba446be8d5dda5c1dc5b6ec" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:36.028513 27381 raft_consensus.cc:2466] T f326ec776ba446be8d5dda5c1dc5b6ec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:36.029549 29199 raft_consensus.cc:513] T f326ec776ba446be8d5dda5c1dc5b6ec P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:36.029778 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f326ec776ba446be8d5dda5c1dc5b6ec" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:36.030290 27516 raft_consensus.cc:2466] T f326ec776ba446be8d5dda5c1dc5b6ec P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:36.030287 29285 raft_consensus.cc:491] T 8fff36d5a80e45af8a256c3908555e08 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51578fb7bceb4e119b3ab1e58c4f5528)
I20250903 20:05:36.030443 29285 raft_consensus.cc:513] T 8fff36d5a80e45af8a256c3908555e08 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.031076 29285 leader_election.cc:290] T 8fff36d5a80e45af8a256c3908555e08 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:36.031196 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f326ec776ba446be8d5dda5c1dc5b6ec" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:36.031360 27381 raft_consensus.cc:3058] T f326ec776ba446be8d5dda5c1dc5b6ec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:36.032215 27183 leader_election.cc:343] T 8fff36d5a80e45af8a256c3908555e08 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 8fff36d5a80e45af8a256c3908555e08
W20250903 20:05:36.032537 27184 leader_election.cc:343] T 8fff36d5a80e45af8a256c3908555e08 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 8fff36d5a80e45af8a256c3908555e08
I20250903 20:05:36.032629 27184 leader_election.cc:304] T 8fff36d5a80e45af8a256c3908555e08 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:36.032958 29285 raft_consensus.cc:2747] T 8fff36d5a80e45af8a256c3908555e08 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:36.012243 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1118) took 1484 ms. Trace:
I20250903 20:05:36.034317 27687 rpcz_store.cc:276] 0903 20:05:34.527514 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.527680 (+ 166us) service_pool.cc:225] Handling call
0903 20:05:36.002676 (+1474996us) ts_tablet_manager.cc:1145] Deleting tablet 54f1ce5192f547f595dea23a9eabf8e6
0903 20:05:36.002807 (+ 131us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.012218 (+ 9411us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1474724}
I20250903 20:05:36.030181 29199 leader_election.cc:290] T f326ec776ba446be8d5dda5c1dc5b6ec P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
W20250903 20:05:36.034946 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 54f1ce5192f547f595dea23a9eabf8e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54f1ce5192f547f595dea23a9eabf8e6 already in progress: creating tablet
I20250903 20:05:36.035212 27685 ts_tablet_manager.cc:1918] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:36.035354 27685 log.cc:1199] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/63a1f26fac3b4b939ebb79c6a07d8f8e
I20250903 20:05:36.035262 27381 raft_consensus.cc:2466] T f326ec776ba446be8d5dda5c1dc5b6ec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:36.035755 27685 ts_tablet_manager.cc:1939] T 63a1f26fac3b4b939ebb79c6a07d8f8e P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:36.036142 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_1b90687e7c414e00ba0897296f8c4ea5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:36.036872 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b90687e7c414e00ba0897296f8c4ea5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:36.036887 27320 leader_election.cc:304] T f326ec776ba446be8d5dda5c1dc5b6ec P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:36.037248 29329 raft_consensus.cc:2762] T f326ec776ba446be8d5dda5c1dc5b6ec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250903 20:05:36.037405 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f326ec776ba446be8d5dda5c1dc5b6ec" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:36.037609 27516 raft_consensus.cc:3058] T f326ec776ba446be8d5dda5c1dc5b6ec P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:36.038400 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1085) took 1631 ms. Trace:
I20250903 20:05:36.035450 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1119) took 1474 ms. Trace:
I20250903 20:05:36.039408 27687 rpcz_store.cc:276] 0903 20:05:34.561105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.561462 (+ 357us) service_pool.cc:225] Handling call
0903 20:05:36.035258 (+1473796us) ts_tablet_manager.cc:1145] Deleting tablet a785adcd0bd943fc97b201b6d29073f4
0903 20:05:36.035271 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.035429 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1472599}
I20250903 20:05:36.039438 27684 rpcz_store.cc:276] 0903 20:05:34.387974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.388028 (+ 54us) service_pool.cc:225] Handling call
0903 20:05:35.860238 (+1472210us) ts_tablet_manager.cc:1145] Deleting tablet 46f49fd718be4a2996d781ab31268217
0903 20:05:35.860249 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:35.903749 (+ 43500us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.003490 (+ 99741us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.019053 (+ 15563us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1471963}
I20250903 20:05:36.035874 27184 leader_election.cc:304] T f326ec776ba446be8d5dda5c1dc5b6ec P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:36.037703 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f326ec776ba446be8d5dda5c1dc5b6ec" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:36.039923 27684 tablet_replica.cc:331] T bf459db3327640f88574fca7b5f2ca98 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
W20250903 20:05:36.040064 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet a785adcd0bd943fc97b201b6d29073f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a785adcd0bd943fc97b201b6d29073f4 already in progress: creating tablet
I20250903 20:05:36.040145 27243 raft_consensus.cc:2391] T f326ec776ba446be8d5dda5c1dc5b6ec P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 51578fb7bceb4e119b3ab1e58c4f5528 in current term 1: Already voted for candidate 89f8f0ee68b940db9ce250c0604f934a in this term.
I20250903 20:05:36.040238 27684 raft_consensus.cc:2241] T bf459db3327640f88574fca7b5f2ca98 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:36.040457 27684 raft_consensus.cc:2270] T bf459db3327640f88574fca7b5f2ca98 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:36.040527 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1120) took 1411 ms. Trace:
I20250903 20:05:36.040602 27687 rpcz_store.cc:276] 0903 20:05:34.629096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.629154 (+ 58us) service_pool.cc:225] Handling call
0903 20:05:36.040368 (+1411214us) ts_tablet_manager.cc:1145] Deleting tablet 6800e0b6269045f29682704fd847d5d1
0903 20:05:36.040381 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.040507 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1410985}
I20250903 20:05:36.040786 27692 consensus_queue.cc:260] T 39c2ebe4ada5426590ec3769c8662644 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:36.041030 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 6800e0b6269045f29682704fd847d5d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6800e0b6269045f29682704fd847d5d1 already in progress: creating tablet
I20250903 20:05:36.041095 29199 raft_consensus.cc:2802] T f326ec776ba446be8d5dda5c1dc5b6ec P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:36.041227 29199 raft_consensus.cc:695] T f326ec776ba446be8d5dda5c1dc5b6ec P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:36.041486 29199 consensus_queue.cc:237] T f326ec776ba446be8d5dda5c1dc5b6ec P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:36.042163 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1121) took 1392 ms. Trace:
I20250903 20:05:36.042295 27687 rpcz_store.cc:276] 0903 20:05:34.650090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.650366 (+ 276us) service_pool.cc:225] Handling call
0903 20:05:36.041517 (+1391151us) ts_tablet_manager.cc:1145] Deleting tablet 8bebea9fa64e459cb24639e1f2d11ac3
0903 20:05:36.041540 (+ 23us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.042142 (+ 602us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1390950}
W20250903 20:05:36.043251 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 8bebea9fa64e459cb24639e1f2d11ac3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8bebea9fa64e459cb24639e1f2d11ac3 already in progress: creating tablet
I20250903 20:05:36.045053 27516 raft_consensus.cc:2466] T f326ec776ba446be8d5dda5c1dc5b6ec P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:36.045573 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 972) took 3574 ms. Trace:
I20250903 20:05:36.046253 27687 tablet_replica.cc:331] T aa9f5e0964b348b7af184f60cd145666 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:36.046728 27687 raft_consensus.cc:2241] T aa9f5e0964b348b7af184f60cd145666 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:36.046769 27685 rpcz_store.cc:276] 0903 20:05:32.470970 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.472526 (+ 1556us) service_pool.cc:225] Handling call
0903 20:05:36.013156 (+3540630us) ts_tablet_manager.cc:1145] Deleting tablet 63a1f26fac3b4b939ebb79c6a07d8f8e
0903 20:05:36.013168 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.022537 (+ 9369us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.035085 (+ 12548us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.045550 (+ 10465us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3540507}
I20250903 20:05:36.046974 27687 raft_consensus.cc:2270] T aa9f5e0964b348b7af184f60cd145666 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:36.047495 27685 tablet_replica.cc:331] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:36.047679 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 63a1f26fac3b4b939ebb79c6a07d8f8e (table default.Table_362ac8cb413e426ebf154eae4b6b5c7e [id=10ef25498f7448dbb19cd66d4c6ba713]) successfully deleted
I20250903 20:05:36.047955 27685 raft_consensus.cc:2241] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:36.048025 27687 ts_tablet_manager.cc:1905] T aa9f5e0964b348b7af184f60cd145666 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:36.048166 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 620a34d0cac94d218d56ab455cf2250b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.048398 27695 consensus_queue.cc:260] T ee40566e497740789575275d0cdca0db P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.048625 27685 raft_consensus.cc:2270] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:36.050426 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 46f49fd718be4a2996d781ab31268217 (table default.Table_b5110c69d0364c50a92330c1824b8416 [id=889259ae7ac44e97aaff581d77e05009]) successfully deleted
I20250903 20:05:36.051651 27685 ts_tablet_manager.cc:1905] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:36.051584 27498 tablet_service.cc:1468] Processing CreateTablet for tablet 620a34d0cac94d218d56ab455cf2250b (DEFAULT_TABLE table=default.Table_aaa4d127435d4fea8862bda75da34909 [id=7e8d6817439546108262bc846df54e8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.052002 27498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 620a34d0cac94d218d56ab455cf2250b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.052678 27695 ts_tablet_manager.cc:1428] T ee40566e497740789575275d0cdca0db P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.053s user 0.003s sys 0.000s
I20250903 20:05:36.053931 27695 tablet_bootstrap.cc:492] T e2441aa3c2c748bba5360259fb663c81 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:36.054859 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet 3f303233e6c347289650c6cf83f8127f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:36.055181 27500 tablet_service.cc:1468] Processing CreateTablet for tablet 68363657d04842d9882941dc85853902 (DEFAULT_TABLE table=default.Table_559f9c5a607c4af7b3ff4c1094d48005 [id=fe584fcd12a9468596defcbb9411218e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.055614 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68363657d04842d9882941dc85853902. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:36.058449 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04094bb7dd324b53a865980c53d4a17a consensus state has no leader
I20250903 20:05:36.061828 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_7ad5f88730c24cbbb97feda4328aa2ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:36.062582 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ad5f88730c24cbbb97feda4328aa2ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:36.065596 27692 ts_tablet_manager.cc:1428] T 39c2ebe4ada5426590ec3769c8662644 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.046s user 0.001s sys 0.001s
I20250903 20:05:36.066193 27695 tablet_bootstrap.cc:654] T e2441aa3c2c748bba5360259fb663c81 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:36.066242 27684 ts_tablet_manager.cc:1905] T bf459db3327640f88574fca7b5f2ca98 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:36.067425 29285 raft_consensus.cc:491] T 793d6157927143d4b9d8dbafd7cf0fcb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:36.067574 29285 raft_consensus.cc:513] T 793d6157927143d4b9d8dbafd7cf0fcb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.068198 29285 leader_election.cc:290] T 793d6157927143d4b9d8dbafd7cf0fcb P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:36.069998 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "793d6157927143d4b9d8dbafd7cf0fcb" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:36.070248 27381 raft_consensus.cc:2466] T 793d6157927143d4b9d8dbafd7cf0fcb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:36.070935 27184 leader_election.cc:304] T 793d6157927143d4b9d8dbafd7cf0fcb P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:36.071270 29285 raft_consensus.cc:2802] T 793d6157927143d4b9d8dbafd7cf0fcb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:36.071393 29285 raft_consensus.cc:491] T 793d6157927143d4b9d8dbafd7cf0fcb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:36.071476 29285 raft_consensus.cc:3058] T 793d6157927143d4b9d8dbafd7cf0fcb P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:36.072878 27687 ts_tablet_manager.cc:1918] T aa9f5e0964b348b7af184f60cd145666 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:36.072991 27687 log.cc:1199] T aa9f5e0964b348b7af184f60cd145666 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/aa9f5e0964b348b7af184f60cd145666
I20250903 20:05:36.073362 27687 ts_tablet_manager.cc:1939] T aa9f5e0964b348b7af184f60cd145666 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:36.074648 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a7795c6eb7c64c4ba9f47fbff01490ae" } new_table_name: "default.Table_39ed8a54eb79449f8803f39d06006379" modify_external_catalogs: true
I20250903 20:05:36.074662 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1122) took 1373 ms. Trace:
I20250903 20:05:36.074870 27687 rpcz_store.cc:276] 0903 20:05:34.701389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.701451 (+ 62us) service_pool.cc:225] Handling call
0903 20:05:36.046149 (+1344698us) ts_tablet_manager.cc:1145] Deleting tablet aa9f5e0964b348b7af184f60cd145666
0903 20:05:36.046168 (+ 19us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.069613 (+ 23445us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.072756 (+ 3143us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.074603 (+ 1847us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1344380}
I20250903 20:05:36.074170 29285 raft_consensus.cc:513] T 793d6157927143d4b9d8dbafd7cf0fcb P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.075562 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet aa9f5e0964b348b7af184f60cd145666 (table default.Table_14e02c1be9c74bd99f4184faa6753f52 [id=a6dc7550a6d643008c92b3a59ba022ce]) successfully deleted
I20250903 20:05:36.076761 29285 leader_election.cc:290] T 793d6157927143d4b9d8dbafd7cf0fcb P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:36.077050 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "793d6157927143d4b9d8dbafd7cf0fcb" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:36.077351 27381 raft_consensus.cc:3058] T 793d6157927143d4b9d8dbafd7cf0fcb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:36.078583 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "793d6157927143d4b9d8dbafd7cf0fcb" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:36.078781 27511 raft_consensus.cc:2466] T 793d6157927143d4b9d8dbafd7cf0fcb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:36.080267 27381 raft_consensus.cc:2466] T 793d6157927143d4b9d8dbafd7cf0fcb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:36.080650 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "793d6157927143d4b9d8dbafd7cf0fcb" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:36.080829 27184 leader_election.cc:304] T 793d6157927143d4b9d8dbafd7cf0fcb P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:36.080904 27516 raft_consensus.cc:3058] T 793d6157927143d4b9d8dbafd7cf0fcb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:36.081154 29196 raft_consensus.cc:491] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e72ef1dbcc5143cf9f04a808f101c80f)
I20250903 20:05:36.081306 29196 raft_consensus.cc:513] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.082562 29196 leader_election.cc:290] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:36.082546 27065 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_1153bf74b285448b88b364efe2169784 [id=f321a32c1ac043238c2fc92bfd5ec1ae] alter complete (version 1)
I20250903 20:05:36.082727 29196 raft_consensus.cc:2802] T 793d6157927143d4b9d8dbafd7cf0fcb P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:36.082855 29196 raft_consensus.cc:695] T 793d6157927143d4b9d8dbafd7cf0fcb P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:36.083108 29196 consensus_queue.cc:237] T 793d6157927143d4b9d8dbafd7cf0fcb P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.083921 27516 raft_consensus.cc:2466] T 793d6157927143d4b9d8dbafd7cf0fcb P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:36.090342 27693 tablet_bootstrap.cc:492] T 2385d3006f90486890ebe43e16a53d7a P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:36.090494 27693 ts_tablet_manager.cc:1397] T 2385d3006f90486890ebe43e16a53d7a P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.104s user 0.002s sys 0.002s
I20250903 20:05:36.090916 27349 tablet_service.cc:1468] Processing CreateTablet for tablet a3cf18a9a8c94a58bb739ee2f81d19c0 (DEFAULT_TABLE table=default.Table_6656b728e12a482fa9e7893ef4842ac7 [id=6788b524a8654e34a4d9ee8cc8fc84d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.091267 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3cf18a9a8c94a58bb739ee2f81d19c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.091192 27347 tablet_service.cc:1468] Processing CreateTablet for tablet 5bea9ae235764109b2ca0794ebe9e047 (DEFAULT_TABLE table=default.Table_df2bc52dc58c498f8b5ad1165c9be27a [id=a00c3c40faac476a95ff55d4559e9634]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.091549 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bea9ae235764109b2ca0794ebe9e047. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.093344 27685 ts_tablet_manager.cc:1918] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:36.093451 27685 log.cc:1199] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/512908bc4d6941ec81b145ee5795405f
I20250903 20:05:36.093387 28567 consensus_queue.cc:1035] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:36.093823 27685 ts_tablet_manager.cc:1939] T 512908bc4d6941ec81b145ee5795405f P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:36.094338 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet a3103e1595ec4c38ba65dca26fe1167c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:36.094802 27352 tablet_service.cc:1468] Processing CreateTablet for tablet b3b83f32cf424788aaa2691ef2ccada0 (DEFAULT_TABLE table=default.Table_903b10284919440483b8f23fb840c6dc [id=4b204b91727246caa786579ca4fe4876]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.095104 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3b83f32cf424788aaa2691ef2ccada0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.097606 27693 raft_consensus.cc:357] T 2385d3006f90486890ebe43e16a53d7a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.097750 27693 raft_consensus.cc:383] T 2385d3006f90486890ebe43e16a53d7a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:36.097791 27693 raft_consensus.cc:738] T 2385d3006f90486890ebe43e16a53d7a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:36.098493 27357 tablet_service.cc:1468] Processing CreateTablet for tablet ea6c245fe0904a3099d21b542d50a0ff (DEFAULT_TABLE table=default.Table_dc1d2e1d3de342378af29da68425a820 [id=6de230f007d840afbf692310662285e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.098807 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea6c245fe0904a3099d21b542d50a0ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.104740 27221 tablet_service.cc:1468] Processing CreateTablet for tablet a3cf18a9a8c94a58bb739ee2f81d19c0 (DEFAULT_TABLE table=default.Table_6656b728e12a482fa9e7893ef4842ac7 [id=6788b524a8654e34a4d9ee8cc8fc84d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.104816 27215 tablet_service.cc:1468] Processing CreateTablet for tablet b3b83f32cf424788aaa2691ef2ccada0 (DEFAULT_TABLE table=default.Table_903b10284919440483b8f23fb840c6dc [id=4b204b91727246caa786579ca4fe4876]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.105197 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3cf18a9a8c94a58bb739ee2f81d19c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.105369 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3b83f32cf424788aaa2691ef2ccada0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.107188 27687 tablet_replica.cc:331] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:36.107393 27687 raft_consensus.cc:2241] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:36.107681 27687 pending_rounds.cc:70] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528: Trying to abort 1 pending ops.
I20250903 20:05:36.107723 27687 pending_rounds.cc:77] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196385013742235648 op_type: NO_OP noop_request { }
I20250903 20:05:36.107811 27687 raft_consensus.cc:2887] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:36.107884 27687 raft_consensus.cc:2270] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:36.108465 27687 ts_tablet_manager.cc:1905] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:36.110679 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 5bea9ae235764109b2ca0794ebe9e047 (DEFAULT_TABLE table=default.Table_df2bc52dc58c498f8b5ad1165c9be27a [id=a00c3c40faac476a95ff55d4559e9634]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.111047 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bea9ae235764109b2ca0794ebe9e047. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.112248 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 973) took 3641 ms. Trace:
I20250903 20:05:36.112332 27685 rpcz_store.cc:276] 0903 20:05:32.471099 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.472735 (+ 1636us) service_pool.cc:225] Handling call
0903 20:05:36.047443 (+3574708us) ts_tablet_manager.cc:1145] Deleting tablet 512908bc4d6941ec81b145ee5795405f
0903 20:05:36.047454 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.057615 (+ 10161us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.093236 (+ 35621us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.112226 (+ 18990us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1965,"spinlock_wait_cycles":3456,"tablet-delete.queue_time_us":3574568}
I20250903 20:05:36.112709 27685 tablet_replica.cc:331] T 541278ca9ffb4e41b341139c2faa337f P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:36.112907 27685 raft_consensus.cc:2241] T 541278ca9ffb4e41b341139c2faa337f P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:36.113070 27685 raft_consensus.cc:2270] T 541278ca9ffb4e41b341139c2faa337f P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:36.113611 27685 ts_tablet_manager.cc:1905] T 541278ca9ffb4e41b341139c2faa337f P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:36.115401 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet de20d18d7f264811a0631aa1144aef2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:36.118119 27507 tablet_service.cc:1515] Processing DeleteTablet for tablet ee40566e497740789575275d0cdca0db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:36.121618 27184 leader_election.cc:343] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 9367b3a22f624c4087848099de32c080
I20250903 20:05:36.121802 27059 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 7a9e55f0811345f7afa71645f38ebdaf (table default.Table_adf05e59a7be478b89666c3162ac83f1 [id=e69321e60a3a4e71a463a0011c0c6d65]). Expected version 2 got 0
I20250903 20:05:36.123148 27065 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3f1f99f4ab084617b3a72447d6e84209" } modify_external_catalogs: true
I20250903 20:05:36.123368 27065 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3f1f99f4ab084617b3a72447d6e84209" } modify_external_catalogs: true
W20250903 20:05:36.125603 27183 leader_election.cc:343] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 9367b3a22f624c4087848099de32c080
I20250903 20:05:36.125726 27183 leader_election.cc:304] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:36.126299 29196 raft_consensus.cc:2747] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:36.126413 27059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_aa4be8035e9441ac8dc4e936724f5e75" } modify_external_catalogs: true
I20250903 20:05:36.126655 27059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_aa4be8035e9441ac8dc4e936724f5e75" } modify_external_catalogs: true
I20250903 20:05:36.128008 27507 tablet_service.cc:1468] Processing CreateTablet for tablet e971126750214c15be004ade8c3b41d2 (DEFAULT_TABLE table=default.Table_69b1032a1e7f4f4d972120dca05a0e38 [id=152e289829b844c5b7699448125e0c5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.128415 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e971126750214c15be004ade8c3b41d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.130187 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet ee40566e497740789575275d0cdca0db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:36.130740 27350 tablet_service.cc:1468] Processing CreateTablet for tablet e971126750214c15be004ade8c3b41d2 (DEFAULT_TABLE table=default.Table_69b1032a1e7f4f4d972120dca05a0e38 [id=152e289829b844c5b7699448125e0c5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.131186 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e971126750214c15be004ade8c3b41d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.133420 27221 tablet_service.cc:1515] Processing DeleteTablet for tablet a3103e1595ec4c38ba65dca26fe1167c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:36.133684 27218 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e34d6d5d68e4395aed78821ea077c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:36.133924 27216 tablet_service.cc:1468] Processing CreateTablet for tablet e971126750214c15be004ade8c3b41d2 (DEFAULT_TABLE table=default.Table_69b1032a1e7f4f4d972120dca05a0e38 [id=152e289829b844c5b7699448125e0c5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.134454 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_d99e1237827942879902dd30ea22ba38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:36.135048 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d99e1237827942879902dd30ea22ba38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:36.135470 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet f326ec776ba446be8d5dda5c1dc5b6ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:36.136840 27687 ts_tablet_manager.cc:1918] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:36.136981 27687 log.cc:1199] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/46f49fd718be4a2996d781ab31268217
I20250903 20:05:36.137804 27496 tablet_service.cc:1468] Processing CreateTablet for tablet a3cf18a9a8c94a58bb739ee2f81d19c0 (DEFAULT_TABLE table=default.Table_6656b728e12a482fa9e7893ef4842ac7 [id=6788b524a8654e34a4d9ee8cc8fc84d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.138355 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3cf18a9a8c94a58bb739ee2f81d19c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.138545 27497 tablet_service.cc:1468] Processing CreateTablet for tablet b3b83f32cf424788aaa2691ef2ccada0 (DEFAULT_TABLE table=default.Table_903b10284919440483b8f23fb840c6dc [id=4b204b91727246caa786579ca4fe4876]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.138896 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3b83f32cf424788aaa2691ef2ccada0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.142060 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 5bea9ae235764109b2ca0794ebe9e047 (DEFAULT_TABLE table=default.Table_df2bc52dc58c498f8b5ad1165c9be27a [id=a00c3c40faac476a95ff55d4559e9634]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.142470 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bea9ae235764109b2ca0794ebe9e047. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.143966 29221 raft_consensus.cc:491] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:36.144110 29221 raft_consensus.cc:513] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.144670 29221 leader_election.cc:290] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:36.146373 27488 tablet_service.cc:1468] Processing CreateTablet for tablet ea6c245fe0904a3099d21b542d50a0ff (DEFAULT_TABLE table=default.Table_dc1d2e1d3de342378af29da68425a820 [id=6de230f007d840afbf692310662285e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.146776 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea6c245fe0904a3099d21b542d50a0ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.147719 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17f16e9c8f794e03a2f8b3b6b35f8e84" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:36.148362 27462 leader_election.cc:343] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:36.148696 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17f16e9c8f794e03a2f8b3b6b35f8e84" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:36.149453 27462 leader_election.cc:343] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:36.149565 27462 leader_election.cc:304] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
W20250903 20:05:36.150564 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 688cfe6155784b31ad50820bc61c4c4a has no consensus state
I20250903 20:05:36.151103 29221 raft_consensus.cc:2747] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:36.159252 27222 tablet_service.cc:1468] Processing CreateTablet for tablet ea6c245fe0904a3099d21b542d50a0ff (DEFAULT_TABLE table=default.Table_dc1d2e1d3de342378af29da68425a820 [id=6de230f007d840afbf692310662285e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.160367 27218 tablet_service.cc:1468] Processing CreateTablet for tablet efa7f4417a6d4917a9c805c8018bbfdf (DEFAULT_TABLE table=default.Table_7b9de0532ff4407086b7f341fedfb3af [id=5f1249a1893e4f83a8fa045d14637183]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.164077 27062 catalog_manager.cc:5582] T 793d6157927143d4b9d8dbafd7cf0fcb P 89f8f0ee68b940db9ce250c0604f934a reported cstate change: term changed from 0 to 1, leader changed from <none> to 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65). New cstate: current_term: 1 leader_uuid: "89f8f0ee68b940db9ce250c0604f934a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: HEALTHY } } }
I20250903 20:05:36.164306 27352 tablet_service.cc:1468] Processing CreateTablet for tablet efa7f4417a6d4917a9c805c8018bbfdf (DEFAULT_TABLE table=default.Table_7b9de0532ff4407086b7f341fedfb3af [id=5f1249a1893e4f83a8fa045d14637183]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.164819 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efa7f4417a6d4917a9c805c8018bbfdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.164778 27062 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 793d6157927143d4b9d8dbafd7cf0fcb (table default.Table_e79f52615fe142b9823a2a99b8b9c406 [id=3432f6ac076b4e49b9fe63bf5fce8591]). Expected version 1 got 0
I20250903 20:05:36.165320 27687 ts_tablet_manager.cc:1939] T 46f49fd718be4a2996d781ab31268217 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:36.165776 27221 tablet_service.cc:1468] Processing CreateTablet for tablet e285057d32914e789ac1efacebbd91e2 (DEFAULT_TABLE table=default.Table_ca1f275bff334b12be53e0884b57dfe0 [id=1c1725ab45584d5eac52b3c58a5ab2fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.166085 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1123) took 1462 ms. Trace:
I20250903 20:05:36.166172 27687 rpcz_store.cc:276] 0903 20:05:34.703985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.704061 (+ 76us) service_pool.cc:225] Handling call
0903 20:05:36.107124 (+1403063us) ts_tablet_manager.cc:1145] Deleting tablet 46f49fd718be4a2996d781ab31268217
0903 20:05:36.107138 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.117227 (+ 10089us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.136701 (+ 19474us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.166062 (+ 29361us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1402768}
I20250903 20:05:36.167840 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 876f753f95ad4c94a34dea9edeeb4233 (DEFAULT_TABLE table=default.Table_0a09cc2674c449fda58086db253699a6 [id=fe8c40362bc54598bc7ed163c503b771]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.169700 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 069a9e21c417467fa68bac210dd64e4f (DEFAULT_TABLE table=default.Table_258f358903dd4f6fa7e310d72562a570 [id=ae5e27c159a9462482d09a79e148f795]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.172585 27503 tablet_service.cc:1468] Processing CreateTablet for tablet efa7f4417a6d4917a9c805c8018bbfdf (DEFAULT_TABLE table=default.Table_7b9de0532ff4407086b7f341fedfb3af [id=5f1249a1893e4f83a8fa045d14637183]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:36.172887 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ca9fffda31624dc0827e6234ab02bb3e consensus state has no leader
I20250903 20:05:36.173012 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efa7f4417a6d4917a9c805c8018bbfdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.173225 27695 tablet_bootstrap.cc:492] T e2441aa3c2c748bba5360259fb663c81 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:36.173363 27695 ts_tablet_manager.cc:1397] T e2441aa3c2c748bba5360259fb663c81 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.120s user 0.007s sys 0.008s
I20250903 20:05:36.174036 27695 raft_consensus.cc:357] T e2441aa3c2c748bba5360259fb663c81 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.174242 27695 raft_consensus.cc:383] T e2441aa3c2c748bba5360259fb663c81 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:36.174320 27695 raft_consensus.cc:738] T e2441aa3c2c748bba5360259fb663c81 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:36.174573 27695 consensus_queue.cc:260] T e2441aa3c2c748bba5360259fb663c81 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.175074 27695 ts_tablet_manager.cc:1428] T e2441aa3c2c748bba5360259fb663c81 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:36.175352 27695 tablet_bootstrap.cc:492] T 47c1287bff1145a5b2ab6d14dbd608c7 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:36.176806 27695 tablet_bootstrap.cc:654] T 47c1287bff1145a5b2ab6d14dbd608c7 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:36.177718 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efa7f4417a6d4917a9c805c8018bbfdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.178020 27687 tablet_replica.cc:331] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:36.177772 27685 ts_tablet_manager.cc:1918] T 541278ca9ffb4e41b341139c2faa337f P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:36.178157 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 876f753f95ad4c94a34dea9edeeb4233. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.178310 27685 log.cc:1199] T 541278ca9ffb4e41b341139c2faa337f P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/541278ca9ffb4e41b341139c2faa337f
I20250903 20:05:36.178467 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 46f49fd718be4a2996d781ab31268217 (table default.Table_b5110c69d0364c50a92330c1824b8416 [id=889259ae7ac44e97aaff581d77e05009]) successfully deleted
I20250903 20:05:36.178733 27685 ts_tablet_manager.cc:1939] T 541278ca9ffb4e41b341139c2faa337f P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:36.178843 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 069a9e21c417467fa68bac210dd64e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.179108 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e285057d32914e789ac1efacebbd91e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.179570 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 971) took 3709 ms. Trace:
I20250903 20:05:36.179657 27685 rpcz_store.cc:276] 0903 20:05:32.470510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.473655 (+ 3145us) service_pool.cc:225] Handling call
0903 20:05:36.112658 (+3639003us) ts_tablet_manager.cc:1145] Deleting tablet 541278ca9ffb4e41b341139c2faa337f
0903 20:05:36.112670 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.127424 (+ 14754us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.177658 (+ 50234us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.179549 (+ 1891us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3638292}
I20250903 20:05:36.180091 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 541278ca9ffb4e41b341139c2faa337f (table default.Table_01215977307c453e9236e5bd72bc8427 [id=7ae71054c4d34d66a210e7c8f7460d4b]) successfully deleted
I20250903 20:05:36.180433 27687 raft_consensus.cc:2241] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:36.180775 27687 raft_consensus.cc:2270] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:36.181357 27687 ts_tablet_manager.cc:1905] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:36.169034 27357 tablet_service.cc:1468] Processing CreateTablet for tablet e285057d32914e789ac1efacebbd91e2 (DEFAULT_TABLE table=default.Table_ca1f275bff334b12be53e0884b57dfe0 [id=1c1725ab45584d5eac52b3c58a5ab2fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.182528 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e285057d32914e789ac1efacebbd91e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.186087 29285 raft_consensus.cc:491] T a0f79831866b40c49c5b27ce2014c3f6 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e72ef1dbcc5143cf9f04a808f101c80f)
I20250903 20:05:36.186318 29285 raft_consensus.cc:513] T a0f79831866b40c49c5b27ce2014c3f6 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.187178 29285 leader_election.cc:290] T a0f79831866b40c49c5b27ce2014c3f6 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:36.188118 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 978) took 3688 ms. Trace:
I20250903 20:05:36.188213 27685 rpcz_store.cc:276] 0903 20:05:32.499541 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.504830 (+ 5289us) service_pool.cc:225] Handling call
0903 20:05:36.187986 (+3683156us) ts_tablet_manager.cc:1145] Deleting tablet e35574d360334150bb494e13f55d3656
0903 20:05:36.187996 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.188099 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3682867}
W20250903 20:05:36.188355 27184 leader_election.cc:343] T a0f79831866b40c49c5b27ce2014c3f6 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: a0f79831866b40c49c5b27ce2014c3f6
W20250903 20:05:36.189342 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet e35574d360334150bb494e13f55d3656 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e35574d360334150bb494e13f55d3656 already in progress: creating tablet
I20250903 20:05:36.189662 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 977) took 3693 ms. Trace:
I20250903 20:05:36.189776 27489 tablet_service.cc:1468] Processing CreateTablet for tablet e285057d32914e789ac1efacebbd91e2 (DEFAULT_TABLE table=default.Table_ca1f275bff334b12be53e0884b57dfe0 [id=1c1725ab45584d5eac52b3c58a5ab2fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.189782 27685 rpcz_store.cc:276] 0903 20:05:32.496115 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.508381 (+ 12266us) service_pool.cc:225] Handling call
0903 20:05:36.188474 (+3680093us) ts_tablet_manager.cc:1145] Deleting tablet 653eade3172344d7aa801edf906ae1d4
0903 20:05:36.188485 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.189644 (+ 1159us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3679860}
W20250903 20:05:36.190260 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 653eade3172344d7aa801edf906ae1d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 653eade3172344d7aa801edf906ae1d4 already in progress: creating tablet
I20250903 20:05:36.190517 27489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e285057d32914e789ac1efacebbd91e2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:36.190850 27183 leader_election.cc:343] T a0f79831866b40c49c5b27ce2014c3f6 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: a0f79831866b40c49c5b27ce2014c3f6
I20250903 20:05:36.190953 27183 leader_election.cc:304] T a0f79831866b40c49c5b27ce2014c3f6 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:36.191320 29285 raft_consensus.cc:2747] T a0f79831866b40c49c5b27ce2014c3f6 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:36.169335 27347 tablet_service.cc:1468] Processing CreateTablet for tablet 069a9e21c417467fa68bac210dd64e4f (DEFAULT_TABLE table=default.Table_258f358903dd4f6fa7e310d72562a570 [id=ae5e27c159a9462482d09a79e148f795]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.191798 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 069a9e21c417467fa68bac210dd64e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.192752 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 979) took 3624 ms. Trace:
I20250903 20:05:36.192827 27685 rpcz_store.cc:276] 0903 20:05:32.567908 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.568053 (+ 145us) service_pool.cc:225] Handling call
0903 20:05:36.192644 (+3624591us) ts_tablet_manager.cc:1145] Deleting tablet a09771c026a9474c9f61ed259d843cca
0903 20:05:36.192652 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.192736 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":511,"tablet-delete.queue_time_us":3623715}
I20250903 20:05:36.167532 27349 tablet_service.cc:1468] Processing CreateTablet for tablet 876f753f95ad4c94a34dea9edeeb4233 (DEFAULT_TABLE table=default.Table_0a09cc2674c449fda58086db253699a6 [id=fe8c40362bc54598bc7ed163c503b771]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.193292 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 876f753f95ad4c94a34dea9edeeb4233. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:36.193275 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet a09771c026a9474c9f61ed259d843cca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a09771c026a9474c9f61ed259d843cca already in progress: creating tablet
I20250903 20:05:36.197727 27216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e971126750214c15be004ade8c3b41d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.199421 27495 tablet_service.cc:1468] Processing CreateTablet for tablet 069a9e21c417467fa68bac210dd64e4f (DEFAULT_TABLE table=default.Table_258f358903dd4f6fa7e310d72562a570 [id=ae5e27c159a9462482d09a79e148f795]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.199604 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea6c245fe0904a3099d21b542d50a0ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.199810 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 069a9e21c417467fa68bac210dd64e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.199941 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 980) took 3631 ms. Trace:
I20250903 20:05:36.200042 27685 rpcz_store.cc:276] 0903 20:05:32.568757 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.570128 (+ 1371us) service_pool.cc:225] Handling call
0903 20:05:36.197638 (+3627510us) ts_tablet_manager.cc:1145] Deleting tablet 8330ab3e4b6f4560b9e41a9df280c211
0903 20:05:36.199820 (+ 2182us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.199925 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3627073}
W20250903 20:05:36.200464 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 8330ab3e4b6f4560b9e41a9df280c211 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8330ab3e4b6f4560b9e41a9df280c211 already in progress: creating tablet
I20250903 20:05:36.200599 27684 ts_tablet_manager.cc:1918] T bf459db3327640f88574fca7b5f2ca98 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:36.200687 27684 log.cc:1199] T bf459db3327640f88574fca7b5f2ca98 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/bf459db3327640f88574fca7b5f2ca98
I20250903 20:05:36.200804 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 981) took 3593 ms. Trace:
I20250903 20:05:36.200870 27685 rpcz_store.cc:276] 0903 20:05:32.607858 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.607922 (+ 64us) service_pool.cc:225] Handling call
0903 20:05:36.200707 (+3592785us) ts_tablet_manager.cc:1145] Deleting tablet 931404d9b3844d908baaed3190d57b56
0903 20:05:36.200716 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.200790 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3581830}
I20250903 20:05:36.201004 27684 ts_tablet_manager.cc:1939] T bf459db3327640f88574fca7b5f2ca98 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:36.201383 27685 tablet_replica.cc:331] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:36.201601 27685 raft_consensus.cc:2241] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:36.201716 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1086) took 1804 ms. Trace:
I20250903 20:05:36.201768 27685 raft_consensus.cc:2270] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:36.201797 27684 rpcz_store.cc:276] 0903 20:05:34.397217 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.397308 (+ 91us) service_pool.cc:225] Handling call
0903 20:05:36.039832 (+1642524us) ts_tablet_manager.cc:1145] Deleting tablet bf459db3327640f88574fca7b5f2ca98
0903 20:05:36.039842 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.103292 (+ 63450us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.200509 (+ 97217us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.201696 (+ 1187us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1642245}
I20250903 20:05:36.202332 27685 ts_tablet_manager.cc:1905] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:36.203161 27695 tablet_bootstrap.cc:492] T 47c1287bff1145a5b2ab6d14dbd608c7 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:36.203370 27695 ts_tablet_manager.cc:1397] T 47c1287bff1145a5b2ab6d14dbd608c7 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.001s
I20250903 20:05:36.204079 27695 raft_consensus.cc:357] T 47c1287bff1145a5b2ab6d14dbd608c7 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:36.204339 27695 raft_consensus.cc:383] T 47c1287bff1145a5b2ab6d14dbd608c7 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:36.204435 27695 raft_consensus.cc:738] T 47c1287bff1145a5b2ab6d14dbd608c7 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:36.204668 27695 consensus_queue.cc:260] T 47c1287bff1145a5b2ab6d14dbd608c7 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:36.205240 27695 ts_tablet_manager.cc:1428] T 47c1287bff1145a5b2ab6d14dbd608c7 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:36.205806 27695 tablet_bootstrap.cc:492] T 7824a0bccdc9485abde124572e502791 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:36.207289 27695 tablet_bootstrap.cc:654] T 7824a0bccdc9485abde124572e502791 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:36.209566 27692 tablet_bootstrap.cc:492] T 30cc0d4ba8d444f1b98232f906ea0c7b P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
W20250903 20:05:36.210188 27183 consensus_peers.cc:489] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 548b50819f694fa3ad508da9f442a99c. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:36.211189 27692 tablet_bootstrap.cc:654] T 30cc0d4ba8d444f1b98232f906ea0c7b P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:36.211588 29285 raft_consensus.cc:491] T 4e34d6d5d68e4395aed78821ea077c1b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:36.211731 29285 raft_consensus.cc:513] T 4e34d6d5d68e4395aed78821ea077c1b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.212393 29285 leader_election.cc:290] T 4e34d6d5d68e4395aed78821ea077c1b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:36.213037 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e34d6d5d68e4395aed78821ea077c1b" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:36.213281 27379 raft_consensus.cc:2466] T 4e34d6d5d68e4395aed78821ea077c1b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:36.213845 27184 leader_election.cc:304] T 4e34d6d5d68e4395aed78821ea077c1b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:36.214433 29285 raft_consensus.cc:2802] T 4e34d6d5d68e4395aed78821ea077c1b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:36.214522 29285 raft_consensus.cc:491] T 4e34d6d5d68e4395aed78821ea077c1b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:36.214582 29285 raft_consensus.cc:3058] T 4e34d6d5d68e4395aed78821ea077c1b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:36.217730 29285 raft_consensus.cc:513] T 4e34d6d5d68e4395aed78821ea077c1b P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.219102 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e34d6d5d68e4395aed78821ea077c1b" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:36.219319 27379 raft_consensus.cc:3058] T 4e34d6d5d68e4395aed78821ea077c1b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:36.219947 29285 leader_election.cc:290] T 4e34d6d5d68e4395aed78821ea077c1b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:36.222003 27379 raft_consensus.cc:2466] T 4e34d6d5d68e4395aed78821ea077c1b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:36.219204 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e34d6d5d68e4395aed78821ea077c1b" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:36.222507 27511 raft_consensus.cc:2466] T 4e34d6d5d68e4395aed78821ea077c1b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:36.219913 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e34d6d5d68e4395aed78821ea077c1b" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:36.222877 27516 raft_consensus.cc:3058] T 4e34d6d5d68e4395aed78821ea077c1b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:36.223449 27685 ts_tablet_manager.cc:1918] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 20:05:36.223557 27685 log.cc:1199] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/9367b3a22f624c4087848099de32c080
I20250903 20:05:36.223922 27685 ts_tablet_manager.cc:1939] T 9367b3a22f624c4087848099de32c080 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:36.224740 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 982) took 3510 ms. Trace:
I20250903 20:05:36.224824 27685 rpcz_store.cc:276] 0903 20:05:32.714098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.714269 (+ 171us) service_pool.cc:225] Handling call
0903 20:05:36.201344 (+3487075us) ts_tablet_manager.cc:1145] Deleting tablet 9367b3a22f624c4087848099de32c080
0903 20:05:36.201352 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.220151 (+ 18799us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.223342 (+ 3191us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.224716 (+ 1374us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3486838}
I20250903 20:05:36.225265 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 983) took 3488 ms. Trace:
I20250903 20:05:36.225339 27685 rpcz_store.cc:276] 0903 20:05:32.736893 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.737685 (+ 792us) service_pool.cc:225] Handling call
0903 20:05:36.225131 (+3487446us) ts_tablet_manager.cc:1145] Deleting tablet df9d0f3662a245d7bc7cc184bc9ef3db
0903 20:05:36.225141 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.225248 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3487220}
I20250903 20:05:36.225574 27516 raft_consensus.cc:2466] T 4e34d6d5d68e4395aed78821ea077c1b P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:36.225672 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 984) took 3479 ms. Trace:
I20250903 20:05:36.225739 27685 rpcz_store.cc:276] 0903 20:05:32.746045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.746665 (+ 620us) service_pool.cc:225] Handling call
0903 20:05:36.225574 (+3478909us) ts_tablet_manager.cc:1145] Deleting tablet df9d0f3662a245d7bc7cc184bc9ef3db
0903 20:05:36.225582 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.225657 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3478712}
I20250903 20:05:36.225987 27685 tablet_replica.cc:331] T a9f43781299e47a8aecc37a0b73d7a64 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:36.227298 27184 leader_election.cc:304] T 4e34d6d5d68e4395aed78821ea077c1b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:36.228487 29285 raft_consensus.cc:2802] T 4e34d6d5d68e4395aed78821ea077c1b P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:36.229334 29285 raft_consensus.cc:695] T 4e34d6d5d68e4395aed78821ea077c1b P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:36.229657 29285 consensus_queue.cc:237] T 4e34d6d5d68e4395aed78821ea077c1b P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.230777 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_258d201c94be4e2aa3dce9385816da45" } modify_external_catalogs: true
I20250903 20:05:36.230964 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_258d201c94be4e2aa3dce9385816da45" } modify_external_catalogs: true
I20250903 20:05:36.211432 27500 tablet_service.cc:1468] Processing CreateTablet for tablet 876f753f95ad4c94a34dea9edeeb4233 (DEFAULT_TABLE table=default.Table_0a09cc2674c449fda58086db253699a6 [id=fe8c40362bc54598bc7ed163c503b771]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.231602 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 876f753f95ad4c94a34dea9edeeb4233. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.233654 29221 raft_consensus.cc:491] T 64c4dd2645034a7eaf243f5474603f10 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:36.233839 29221 raft_consensus.cc:513] T 64c4dd2645034a7eaf243f5474603f10 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.234715 29221 leader_election.cc:290] T 64c4dd2645034a7eaf243f5474603f10 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:36.235558 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet bf459db3327640f88574fca7b5f2ca98 (table default.Table_f6ddd620f29a4333a8f71f99316e1a8d [id=b6932927b05f46be967419e342532b3f]) successfully deleted
I20250903 20:05:36.235759 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "64c4dd2645034a7eaf243f5474603f10" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:36.236487 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "64c4dd2645034a7eaf243f5474603f10" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:36.236721 27462 leader_election.cc:343] T 64c4dd2645034a7eaf243f5474603f10 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:36.237125 27462 leader_election.cc:343] T 64c4dd2645034a7eaf243f5474603f10 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:36.237232 27462 leader_election.cc:304] T 64c4dd2645034a7eaf243f5474603f10 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:36.237823 29221 raft_consensus.cc:2747] T 64c4dd2645034a7eaf243f5474603f10 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:36.243292 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_1b1798fd0780489a935aa84d7ea2995a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:36.243580 27685 raft_consensus.cc:2241] T a9f43781299e47a8aecc37a0b73d7a64 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:36.243826 27685 raft_consensus.cc:2270] T a9f43781299e47a8aecc37a0b73d7a64 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:36.243973 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b1798fd0780489a935aa84d7ea2995a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:36.244557 27685 ts_tablet_manager.cc:1905] T a9f43781299e47a8aecc37a0b73d7a64 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:36.250833 27495 tablet_service.cc:1468] Processing CreateTablet for tablet 9956bd3b85e042ab8625d7f9021e363a (DEFAULT_TABLE table=default.Table_9d4f2aec0e3f4f858affaa9a001f0b88 [id=bb7b2ca6500d48e29d28c4b4dde22f4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.251441 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9956bd3b85e042ab8625d7f9021e363a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.252547 27687 ts_tablet_manager.cc:1918] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:36.255412 27692 tablet_bootstrap.cc:492] T 30cc0d4ba8d444f1b98232f906ea0c7b P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:36.255581 27692 ts_tablet_manager.cc:1397] T 30cc0d4ba8d444f1b98232f906ea0c7b P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.000s
I20250903 20:05:36.256366 27692 raft_consensus.cc:357] T 30cc0d4ba8d444f1b98232f906ea0c7b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.256562 27692 raft_consensus.cc:383] T 30cc0d4ba8d444f1b98232f906ea0c7b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:36.256619 27692 raft_consensus.cc:738] T 30cc0d4ba8d444f1b98232f906ea0c7b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:36.256824 27692 consensus_queue.cc:260] T 30cc0d4ba8d444f1b98232f906ea0c7b P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.257417 27692 ts_tablet_manager.cc:1428] T 30cc0d4ba8d444f1b98232f906ea0c7b P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:36.258457 29199 raft_consensus.cc:491] T 8ba2636641654d8596fd2a5b0644d073 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:36.258594 29199 raft_consensus.cc:513] T 8ba2636641654d8596fd2a5b0644d073 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.259820 29199 leader_election.cc:290] T 8ba2636641654d8596fd2a5b0644d073 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:36.260191 27692 tablet_bootstrap.cc:492] T 13496efe34944b7280e7a684f180e6e0 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:36.265574 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ba2636641654d8596fd2a5b0644d073" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:36.268049 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_1c303bbec4ab48e48656cc3a86cffea0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:36.268843 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c303bbec4ab48e48656cc3a86cffea0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:36.272473 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1087) took 1864 ms. Trace:
I20250903 20:05:36.272678 27684 rpcz_store.cc:276] 0903 20:05:34.408427 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.415508 (+ 7081us) service_pool.cc:225] Handling call
0903 20:05:36.257629 (+1842121us) ts_tablet_manager.cc:1145] Deleting tablet 200c06f296b94eab94f35f312c5a443b
0903 20:05:36.271723 (+ 14094us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.272448 (+ 725us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1841593}
I20250903 20:05:36.273133 29221 raft_consensus.cc:491] T 3f303233e6c347289650c6cf83f8127f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:36.274394 29221 raft_consensus.cc:513] T 3f303233e6c347289650c6cf83f8127f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.275096 29221 leader_election.cc:290] T 3f303233e6c347289650c6cf83f8127f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:36.275488 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1096) took 1728 ms. Trace:
I20250903 20:05:36.275696 27684 rpcz_store.cc:276] 0903 20:05:34.547468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.547546 (+ 78us) service_pool.cc:225] Handling call
0903 20:05:36.275365 (+1727819us) ts_tablet_manager.cc:1145] Deleting tablet 13137053365a428e9d1e8d8852a53b6a
0903 20:05:36.275376 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.275472 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1727504}
I20250903 20:05:36.275748 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f303233e6c347289650c6cf83f8127f" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:36.276134 27684 tablet_replica.cc:331] T 5557dfdd953a41e9bd3a8be4cd689604 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
W20250903 20:05:36.280193 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2770e6b0f07d468b98fc431c81a417b0 consensus state has no leader
I20250903 20:05:36.279728 27685 ts_tablet_manager.cc:1918] T a9f43781299e47a8aecc37a0b73d7a64 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 20:05:36.281688 27685 log.cc:1199] T a9f43781299e47a8aecc37a0b73d7a64 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/a9f43781299e47a8aecc37a0b73d7a64
I20250903 20:05:36.282387 27685 ts_tablet_manager.cc:1939] T a9f43781299e47a8aecc37a0b73d7a64 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:36.282902 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 9956bd3b85e042ab8625d7f9021e363a (DEFAULT_TABLE table=default.Table_9d4f2aec0e3f4f858affaa9a001f0b88 [id=bb7b2ca6500d48e29d28c4b4dde22f4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.260610 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ba2636641654d8596fd2a5b0644d073" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:36.283296 29199 raft_consensus.cc:491] T 5557dfdd953a41e9bd3a8be4cd689604 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:36.283345 27381 raft_consensus.cc:2391] T 8ba2636641654d8596fd2a5b0644d073 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89f8f0ee68b940db9ce250c0604f934a in current term 1: Already voted for candidate e72ef1dbcc5143cf9f04a808f101c80f in this term.
I20250903 20:05:36.283435 29199 raft_consensus.cc:513] T 5557dfdd953a41e9bd3a8be4cd689604 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.276582 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f303233e6c347289650c6cf83f8127f" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:36.283888 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_61d7e38771d54f1fb1f75280d225c5cc" } modify_external_catalogs: true
I20250903 20:05:36.284106 27687 log.cc:1199] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/bf459db3327640f88574fca7b5f2ca98
I20250903 20:05:36.284075 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_61d7e38771d54f1fb1f75280d225c5cc" } modify_external_catalogs: true
I20250903 20:05:36.284513 27184 leader_election.cc:304] T 8ba2636641654d8596fd2a5b0644d073 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:36.284876 29196 raft_consensus.cc:3058] T 8ba2636641654d8596fd2a5b0644d073 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:36.275964 27692 tablet_bootstrap.cc:654] T 13496efe34944b7280e7a684f180e6e0 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:36.277035 27462 leader_election.cc:343] T 3f303233e6c347289650c6cf83f8127f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:36.286882 27462 leader_election.cc:343] T 3f303233e6c347289650c6cf83f8127f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:36.287042 27462 leader_election.cc:304] T 3f303233e6c347289650c6cf83f8127f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:36.287382 29196 raft_consensus.cc:2747] T 8ba2636641654d8596fd2a5b0644d073 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:36.287374 27684 raft_consensus.cc:2241] T 5557dfdd953a41e9bd3a8be4cd689604 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:36.287528 27684 raft_consensus.cc:2270] T 5557dfdd953a41e9bd3a8be4cd689604 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:36.287783 29220 raft_consensus.cc:2747] T 3f303233e6c347289650c6cf83f8127f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:36.288098 29199 leader_election.cc:290] T 5557dfdd953a41e9bd3a8be4cd689604 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
W20250903 20:05:36.288470 27184 leader_election.cc:343] T 5557dfdd953a41e9bd3a8be4cd689604 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 5557dfdd953a41e9bd3a8be4cd689604
I20250903 20:05:36.288522 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5557dfdd953a41e9bd3a8be4cd689604" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:36.281028 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 9956bd3b85e042ab8625d7f9021e363a (DEFAULT_TABLE table=default.Table_9d4f2aec0e3f4f858affaa9a001f0b88 [id=bb7b2ca6500d48e29d28c4b4dde22f4c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:36.288810 27462 consensus_peers.cc:489] T 472f0fb01263468a828ec13795656624 P e72ef1dbcc5143cf9f04a808f101c80f -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:36.289161 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9956bd3b85e042ab8625d7f9021e363a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.289227 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9956bd3b85e042ab8625d7f9021e363a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.292630 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet fa544dfdfff04a1e97676e562aeb0f40
I20250903 20:05:36.293280 27065 master_service.cc:948] ReplaceTablet: received request to replace tablet 120ca61b3da742dc9a2efa483ead7e97 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:36.294932 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 993) took 3462 ms. Trace:
I20250903 20:05:36.295186 27685 rpcz_store.cc:276] 0903 20:05:32.832072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.832512 (+ 440us) service_pool.cc:225] Handling call
0903 20:05:36.225939 (+3393427us) ts_tablet_manager.cc:1145] Deleting tablet a9f43781299e47a8aecc37a0b73d7a64
0903 20:05:36.225950 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.257538 (+ 31588us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.279602 (+ 22064us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.294905 (+ 15303us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3393268}
I20250903 20:05:36.295706 27685 tablet_replica.cc:331] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:36.296026 27685 raft_consensus.cc:2241] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:36.296546 27685 raft_consensus.cc:2270] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:36.297124 27687 ts_tablet_manager.cc:1939] T bf459db3327640f88574fca7b5f2ca98 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:36.297888 27065 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c9ba436d242a498582b9009fa5d299e4" } modify_external_catalogs: true
I20250903 20:05:36.298087 27065 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c9ba436d242a498582b9009fa5d299e4" } modify_external_catalogs: true
I20250903 20:05:36.303678 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 9367b3a22f624c4087848099de32c080 (table default.Table_08918e8b515e4487bf4aee0a33234bd5 [id=891b259c1c6b44e2a82dc982c24e6cb6]) successfully deleted
W20250903 20:05:36.304006 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet df9d0f3662a245d7bc7cc184bc9ef3db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df9d0f3662a245d7bc7cc184bc9ef3db already in progress: creating tablet
W20250903 20:05:36.304389 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet df9d0f3662a245d7bc7cc184bc9ef3db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df9d0f3662a245d7bc7cc184bc9ef3db already in progress: creating tablet
I20250903 20:05:36.304720 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet a9f43781299e47a8aecc37a0b73d7a64 (table default.Table_73a4ac2afbad4654ac623cd581a45eb2 [id=f70d7ea93e154e6199f6766316353862]) successfully deleted
I20250903 20:05:36.308300 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1124) took 1603 ms. Trace:
I20250903 20:05:36.303275 27347 tablet_service.cc:1468] Processing CreateTablet for tablet 2eba38dd30454b26b2c0754e5fa05c16 (DEFAULT_TABLE table=default.Table_258d201c94be4e2aa3dce9385816da45 [id=e4934d788707475d9adf8912390182a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.308882 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2eba38dd30454b26b2c0754e5fa05c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.309091 27059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 7c6780770f104f669102b051ee7faeb5
I20250903 20:05:36.309490 27684 ts_tablet_manager.cc:1905] T 5557dfdd953a41e9bd3a8be4cd689604 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:36.313154 29221 raft_consensus.cc:491] T de20d18d7f264811a0631aa1144aef2c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:36.313311 29221 raft_consensus.cc:513] T de20d18d7f264811a0631aa1144aef2c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.313881 29221 leader_election.cc:290] T de20d18d7f264811a0631aa1144aef2c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:36.315333 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de20d18d7f264811a0631aa1144aef2c" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:36.315922 27462 leader_election.cc:343] T de20d18d7f264811a0631aa1144aef2c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:36.318651 27061 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 793d6157927143d4b9d8dbafd7cf0fcb (table default.Table_e79f52615fe142b9823a2a99b8b9c406 [id=3432f6ac076b4e49b9fe63bf5fce8591]). Expected version 1 got 0
I20250903 20:05:36.322937 27489 tablet_service.cc:1468] Processing CreateTablet for tablet 7bb0da31c68440788dc296e67b38ec7c (DEFAULT_TABLE table=default.Table_6b71f7836da34d58a17dacbfb7bcac07 [id=d62014ebcbe1451b832b1f38a036418a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.323859 27489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bb0da31c68440788dc296e67b38ec7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.305999 27349 tablet_service.cc:1468] Processing CreateTablet for tablet 7bb0da31c68440788dc296e67b38ec7c (DEFAULT_TABLE table=default.Table_6b71f7836da34d58a17dacbfb7bcac07 [id=d62014ebcbe1451b832b1f38a036418a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.325522 29330 tablet.cc:1722] T 2059ccbd6d884085886d7d564afe03aa P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:36.325012 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bb0da31c68440788dc296e67b38ec7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.307000 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet fa544dfdfff04a1e97676e562aeb0f40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:36.332263 29220 raft_consensus.cc:491] T 7824a0bccdc9485abde124572e502791 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:36.332432 29220 raft_consensus.cc:513] T 7824a0bccdc9485abde124572e502791 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.332705 27503 tablet_service.cc:1468] Processing CreateTablet for tablet 6e1479acb1c04d3d9d07bbef26e77970 (DEFAULT_TABLE table=default.Table_b54f1ddd84a4489a88fa3ec0bc3dc6a3 [id=a308894c11e5416b929388e6beccd96b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.333081 29220 leader_election.cc:290] T 7824a0bccdc9485abde124572e502791 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:36.333268 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e1479acb1c04d3d9d07bbef26e77970. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.333487 27496 tablet_service.cc:1515] Processing DeleteTablet for tablet fa544dfdfff04a1e97676e562aeb0f40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:36.333760 27496 tablet_service.cc:1515] Processing DeleteTablet for tablet 7c6780770f104f669102b051ee7faeb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:36.334172 27496 tablet_service.cc:1515] Processing DeleteTablet for tablet 39c2ebe4ada5426590ec3769c8662644 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c772ab7c50c4994aacde5a05bb0f08e) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:36.334429 27496 tablet_service.cc:1515] Processing DeleteTablet for tablet d4c703dafbca4e669dbb18946ffac699 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:36.335847 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7824a0bccdc9485abde124572e502791" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:36.336978 27462 leader_election.cc:343] T 7824a0bccdc9485abde124572e502791 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:36.337256 29329 raft_consensus.cc:491] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:36.337433 29329 raft_consensus.cc:513] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:36.338076 27051 catalog_manager.cc:4895] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 200c06f296b94eab94f35f312c5a443b because the tablet was not found. No further retry: Not found: Tablet not found: 200c06f296b94eab94f35f312c5a443b
I20250903 20:05:36.338569 27488 tablet_service.cc:1468] Processing CreateTablet for tablet 2eba38dd30454b26b2c0754e5fa05c16 (DEFAULT_TABLE table=default.Table_258d201c94be4e2aa3dce9385816da45 [id=e4934d788707475d9adf8912390182a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.338975 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2eba38dd30454b26b2c0754e5fa05c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.339563 29329 leader_election.cc:290] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:36.302167 27357 tablet_service.cc:1468] Processing CreateTablet for tablet 6e1479acb1c04d3d9d07bbef26e77970 (DEFAULT_TABLE table=default.Table_b54f1ddd84a4489a88fa3ec0bc3dc6a3 [id=a308894c11e5416b929388e6beccd96b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.340035 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e1479acb1c04d3d9d07bbef26e77970. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:36.347050 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 13137053365a428e9d1e8d8852a53b6a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13137053365a428e9d1e8d8852a53b6a already in progress: creating tablet
I20250903 20:05:36.303256 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 6e1479acb1c04d3d9d07bbef26e77970 (DEFAULT_TABLE table=default.Table_b54f1ddd84a4489a88fa3ec0bc3dc6a3 [id=a308894c11e5416b929388e6beccd96b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.347944 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e1479acb1c04d3d9d07bbef26e77970. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.350416 27692 tablet_bootstrap.cc:492] T 13496efe34944b7280e7a684f180e6e0 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:36.305559 29199 raft_consensus.cc:491] T 02b3e262dd7841dfbb0520377eb94743 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e72ef1dbcc5143cf9f04a808f101c80f)
I20250903 20:05:36.350596 27692 ts_tablet_manager.cc:1397] T 13496efe34944b7280e7a684f180e6e0 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.091s user 0.002s sys 0.001s
I20250903 20:05:36.351286 29199 raft_consensus.cc:513] T 02b3e262dd7841dfbb0520377eb94743 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.351451 27692 raft_consensus.cc:357] T 13496efe34944b7280e7a684f180e6e0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:36.351615 27692 raft_consensus.cc:383] T 13496efe34944b7280e7a684f180e6e0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:36.351677 27692 raft_consensus.cc:738] T 13496efe34944b7280e7a684f180e6e0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:36.351940 27692 consensus_queue.cc:260] T 13496efe34944b7280e7a684f180e6e0 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:36.352681 27692 ts_tablet_manager.cc:1428] T 13496efe34944b7280e7a684f180e6e0 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:36.353152 27692 tablet_bootstrap.cc:492] T e2441aa3c2c748bba5360259fb663c81 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:36.355309 29199 leader_election.cc:290] T 02b3e262dd7841dfbb0520377eb94743 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:36.305815 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 2eba38dd30454b26b2c0754e5fa05c16 (DEFAULT_TABLE table=default.Table_258d201c94be4e2aa3dce9385816da45 [id=e4934d788707475d9adf8912390182a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.306648 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet fa544dfdfff04a1e97676e562aeb0f40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:36.314805 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de20d18d7f264811a0631aa1144aef2c" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:36.323398 29363 tablet.cc:1722] T 2059ccbd6d884085886d7d564afe03aa P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:36.356595 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2eba38dd30454b26b2c0754e5fa05c16. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:36.328794 27183 leader_election.cc:343] T 5557dfdd953a41e9bd3a8be4cd689604 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:36.356738 27183 leader_election.cc:304] T 5557dfdd953a41e9bd3a8be4cd689604 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:36.357287 27462 leader_election.cc:343] T de20d18d7f264811a0631aa1144aef2c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:36.357414 27462 leader_election.cc:304] T de20d18d7f264811a0631aa1144aef2c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:36.357934 29220 raft_consensus.cc:2747] T de20d18d7f264811a0631aa1144aef2c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:36.332705 29196 raft_consensus.cc:491] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:36.336709 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7824a0bccdc9485abde124572e502791" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:36.358242 29196 raft_consensus.cc:513] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.339375 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3103e1595ec4c38ba65dca26fe1167c" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:36.358489 27240 raft_consensus.cc:2466] T a3103e1595ec4c38ba65dca26fe1167c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:36.358894 29196 leader_election.cc:290] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:36.358954 27321 leader_election.cc:304] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:36.359038 29196 raft_consensus.cc:2747] T 5557dfdd953a41e9bd3a8be4cd689604 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:36.359491 27462 leader_election.cc:343] T 7824a0bccdc9485abde124572e502791 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:36.359642 27462 leader_election.cc:304] T 7824a0bccdc9485abde124572e502791 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:36.359896 29220 raft_consensus.cc:2747] T 7824a0bccdc9485abde124572e502791 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:36.361799 29227 raft_consensus.cc:2802] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:36.361903 29227 raft_consensus.cc:491] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:36.361990 29227 raft_consensus.cc:3058] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:36.364773 29227 raft_consensus.cc:513] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.365460 29227 leader_election.cc:290] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:36.368054 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3103e1595ec4c38ba65dca26fe1167c" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:36.368278 27240 raft_consensus.cc:3058] T a3103e1595ec4c38ba65dca26fe1167c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:36.374819 27509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f1d22e76ea72441db51f7e9db26eb520" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:36.375128 27509 raft_consensus.cc:2466] T f1d22e76ea72441db51f7e9db26eb520 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:36.375288 29331 tablet.cc:1722] T 2059ccbd6d884085886d7d564afe03aa P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:36.374401 27240 raft_consensus.cc:2466] T a3103e1595ec4c38ba65dca26fe1167c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:36.376324 27321 leader_election.cc:304] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:36.376643 28567 raft_consensus.cc:2802] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:36.376742 28567 raft_consensus.cc:695] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:36.376996 28567 consensus_queue.cc:237] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:36.378428 27183 leader_election.cc:343] T 02b3e262dd7841dfbb0520377eb94743 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 02b3e262dd7841dfbb0520377eb94743
I20250903 20:05:36.378800 27183 leader_election.cc:304] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:36.379297 29196 raft_consensus.cc:2802] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:36.379380 29196 raft_consensus.cc:491] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:36.379468 29196 raft_consensus.cc:3058] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:36.382356 27320 consensus_peers.cc:489] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 950a6fcd35f64f2d889831fe9cdf503a. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:36.383227 27509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3103e1595ec4c38ba65dca26fe1167c" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:36.383448 27509 raft_consensus.cc:3058] T a3103e1595ec4c38ba65dca26fe1167c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:36.384326 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 2eba38dd30454b26b2c0754e5fa05c16
I20250903 20:05:36.386408 29220 consensus_queue.cc:1035] T 8ba2636641654d8596fd2a5b0644d073 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:36.387246 29196 raft_consensus.cc:513] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.387925 29196 leader_election.cc:290] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:36.386152 27509 raft_consensus.cc:2466] T a3103e1595ec4c38ba65dca26fe1167c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:36.388794 27509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f1d22e76ea72441db51f7e9db26eb520" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:36.388974 27509 raft_consensus.cc:3058] T f1d22e76ea72441db51f7e9db26eb520 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:36.389434 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 2dc0f714cdb44ece90615de5a11bc71f
I20250903 20:05:36.390733 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 17f16e9c8f794e03a2f8b3b6b35f8e84
W20250903 20:05:36.390815 27184 leader_election.cc:343] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: f1d22e76ea72441db51f7e9db26eb520
W20250903 20:05:36.391459 27184 leader_election.cc:343] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: f1d22e76ea72441db51f7e9db26eb520
I20250903 20:05:36.391806 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_d0aec832c86d4e868563056c6100c6b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:36.392522 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0aec832c86d4e868563056c6100c6b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:36.392959 27509 raft_consensus.cc:2466] T f1d22e76ea72441db51f7e9db26eb520 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:36.394483 27183 leader_election.cc:304] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:36.394879 29199 raft_consensus.cc:2802] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:36.394999 29199 raft_consensus.cc:695] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:36.395269 29199 consensus_queue.cc:237] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.397404 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_53b5ec8ef7854180bc6c9d983cbf216d" } new_table_name: "default.Table_eb85ccd3e82044c6b7760fa2b6a3422c" modify_external_catalogs: true
I20250903 20:05:36.398452 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_48a0a980c12f40128907043e0ad8a7a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:36.399053 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_48a0a980c12f40128907043e0ad8a7a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:36.402616 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_351c22f0442944f889808350e50d2887"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:36.403241 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_351c22f0442944f889808350e50d2887 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:36.404165 27184 consensus_peers.cc:489] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c18a98d5c0f4a43ae91c9130047b681. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:36.406661 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 7c6780770f104f669102b051ee7faeb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:36.407027 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e34d6d5d68e4395aed78821ea077c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:36.407655 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 2eba38dd30454b26b2c0754e5fa05c16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:36.408242 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 2eba38dd30454b26b2c0754e5fa05c16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:36.408610 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 2dc0f714cdb44ece90615de5a11bc71f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:36.408963 27503 tablet_service.cc:1515] Processing DeleteTablet for tablet 2dc0f714cdb44ece90615de5a11bc71f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:36.409355 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 2dc0f714cdb44ece90615de5a11bc71f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:36.409559 27500 tablet_service.cc:1515] Processing DeleteTablet for tablet 2eba38dd30454b26b2c0754e5fa05c16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:36.409953 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 17f16e9c8f794e03a2f8b3b6b35f8e84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:36.409610 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet 7c6780770f104f669102b051ee7faeb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:36.410823 27692 tablet_bootstrap.cc:654] T e2441aa3c2c748bba5360259fb663c81 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:36.411540 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 17f16e9c8f794e03a2f8b3b6b35f8e84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:36.411633 29220 consensus_queue.cc:1035] T 8ba2636641654d8596fd2a5b0644d073 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:36.412107 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet e2441aa3c2c748bba5360259fb663c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:36.414371 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 47c1287bff1145a5b2ab6d14dbd608c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:36.415055 27184 leader_election.cc:343] T 02b3e262dd7841dfbb0520377eb94743 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 02b3e262dd7841dfbb0520377eb94743
I20250903 20:05:36.415200 27184 leader_election.cc:304] T 02b3e262dd7841dfbb0520377eb94743 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:36.415577 29199 raft_consensus.cc:2747] T 02b3e262dd7841dfbb0520377eb94743 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:36.433261 27503 tablet_service.cc:1515] Processing DeleteTablet for tablet 30cc0d4ba8d444f1b98232f906ea0c7b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:36.434509 27500 tablet_service.cc:1515] Processing DeleteTablet for tablet 17f16e9c8f794e03a2f8b3b6b35f8e84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:36.441031 29371 raft_consensus.cc:491] T 39c2ebe4ada5426590ec3769c8662644 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:36.441439 29371 raft_consensus.cc:513] T 39c2ebe4ada5426590ec3769c8662644 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.442173 29371 leader_election.cc:290] T 39c2ebe4ada5426590ec3769c8662644 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:36.444485 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39c2ebe4ada5426590ec3769c8662644" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:36.445236 27462 leader_election.cc:343] T 39c2ebe4ada5426590ec3769c8662644 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:36.446630 27684 ts_tablet_manager.cc:1918] T 5557dfdd953a41e9bd3a8be4cd689604 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:36.446812 27684 log.cc:1199] T 5557dfdd953a41e9bd3a8be4cd689604 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/5557dfdd953a41e9bd3a8be4cd689604
I20250903 20:05:36.447216 27684 ts_tablet_manager.cc:1939] T 5557dfdd953a41e9bd3a8be4cd689604 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:36.449079 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1097) took 1901 ms. Trace:
I20250903 20:05:36.449528 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39c2ebe4ada5426590ec3769c8662644" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:36.450335 27684 rpcz_store.cc:276] 0903 20:05:34.547847 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.547915 (+ 68us) service_pool.cc:225] Handling call
0903 20:05:36.276093 (+1728178us) ts_tablet_manager.cc:1145] Deleting tablet 5557dfdd953a41e9bd3a8be4cd689604
0903 20:05:36.276101 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.312566 (+ 36465us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.446503 (+133937us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.449047 (+ 2544us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1728006}
I20250903 20:05:36.451165 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1098) took 1901 ms. Trace:
I20250903 20:05:36.451295 27684 rpcz_store.cc:276] 0903 20:05:34.550071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.550132 (+ 61us) service_pool.cc:225] Handling call
0903 20:05:36.451038 (+1900906us) ts_tablet_manager.cc:1145] Deleting tablet 2035344c7f6d478b89a89981d89f22ca
0903 20:05:36.451050 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.451150 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1900675}
I20250903 20:05:36.451776 27684 tablet_replica.cc:331] T f1d22e76ea72441db51f7e9db26eb520 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:36.452569 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e027444cbd5545a38f6e4f35e1ed90c3" } new_table_name: "default.Table_03a2ab4b166743fab42d4ec94c2f5ea3" modify_external_catalogs: true
I20250903 20:05:36.455823 27058 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 8ba2636641654d8596fd2a5b0644d073 (table default.Table_e7148e16606e4cd3908c86b439f75b78 [id=4bffa29ea98049718ac3ce35de9813c5]). Expected version 1 got 0
W20250903 20:05:36.456425 26890 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 92537.5us
I20250903 20:05:36.458026 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_37b6f939fca548ada1f45f908c602ca9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:36.458626 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37b6f939fca548ada1f45f908c602ca9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:36.460039 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 5557dfdd953a41e9bd3a8be4cd689604 (table default.Table_672b60689df64b1d8c9ebd27468d68d4 [id=6d5615e3adc24110a36e1486e69cde7b]) successfully deleted
W20250903 20:05:36.460325 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 2035344c7f6d478b89a89981d89f22ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2035344c7f6d478b89a89981d89f22ca already in progress: creating tablet
I20250903 20:05:36.460743 27685 ts_tablet_manager.cc:1905] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:36.462013 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_051d82784c114e9f96c5717143bbc65d" } new_table_name: "default.Table_95fbad40228140c9b99261d911521769" modify_external_catalogs: true
I20250903 20:05:36.463472 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_f0fc60c750344a2d96326556708d661e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:36.463992 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0fc60c750344a2d96326556708d661e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:36.465160 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_741f03359c5343268a5ebe2c70552ed2" } new_table_name: "default.Table_5d2480e7b40245e488dedd3c6a5add50" modify_external_catalogs: true
I20250903 20:05:36.466339 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_43b6194256e1461cb34be4e7642889f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:36.466835 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43b6194256e1461cb34be4e7642889f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:36.472892 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7be2526b81554322893aa9de732205ad consensus state has no leader
I20250903 20:05:36.474345 29285 raft_consensus.cc:491] T 200c06f296b94eab94f35f312c5a443b P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:36.474534 29285 raft_consensus.cc:513] T 200c06f296b94eab94f35f312c5a443b P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.475812 29285 leader_election.cc:290] T 200c06f296b94eab94f35f312c5a443b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:36.480224 27684 raft_consensus.cc:2241] T f1d22e76ea72441db51f7e9db26eb520 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
W20250903 20:05:36.456218 27462 leader_election.cc:343] T 39c2ebe4ada5426590ec3769c8662644 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:36.482402 27183 leader_election.cc:343] T 200c06f296b94eab94f35f312c5a443b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 200c06f296b94eab94f35f312c5a443b
W20250903 20:05:36.484445 27184 leader_election.cc:343] T 200c06f296b94eab94f35f312c5a443b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 200c06f296b94eab94f35f312c5a443b
I20250903 20:05:36.484773 27184 leader_election.cc:304] T 200c06f296b94eab94f35f312c5a443b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:36.480686 27462 leader_election.cc:304] T 39c2ebe4ada5426590ec3769c8662644 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:36.487529 27695 tablet_bootstrap.cc:492] T 7824a0bccdc9485abde124572e502791 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:36.487721 27695 ts_tablet_manager.cc:1397] T 7824a0bccdc9485abde124572e502791 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.282s user 0.022s sys 0.039s
I20250903 20:05:36.487751 29220 raft_consensus.cc:2747] T 39c2ebe4ada5426590ec3769c8662644 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:36.488497 27695 raft_consensus.cc:357] T 7824a0bccdc9485abde124572e502791 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.488775 27695 raft_consensus.cc:383] T 7824a0bccdc9485abde124572e502791 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:36.488869 27695 raft_consensus.cc:738] T 7824a0bccdc9485abde124572e502791 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:36.486104 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3103e1595ec4c38ba65dca26fe1167c" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:36.489193 27695 consensus_queue.cc:260] T 7824a0bccdc9485abde124572e502791 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.489360 27516 raft_consensus.cc:2374] T a3103e1595ec4c38ba65dca26fe1167c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1. Re-sending same reply.
I20250903 20:05:36.489889 27695 ts_tablet_manager.cc:1428] T 7824a0bccdc9485abde124572e502791 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:36.480486 27684 raft_consensus.cc:2270] T f1d22e76ea72441db51f7e9db26eb520 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:36.491288 29196 raft_consensus.cc:2747] T 200c06f296b94eab94f35f312c5a443b P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:36.491828 27684 ts_tablet_manager.cc:1905] T f1d22e76ea72441db51f7e9db26eb520 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:36.493579 27687 rpcz_store.cc:276] 0903 20:05:34.704540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.704587 (+ 47us) service_pool.cc:225] Handling call
0903 20:05:36.177959 (+1473372us) ts_tablet_manager.cc:1145] Deleting tablet bf459db3327640f88574fca7b5f2ca98
0903 20:05:36.177968 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.213664 (+ 35696us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.252435 (+ 38771us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.308276 (+ 55841us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1473213}
I20250903 20:05:36.494140 27692 tablet_bootstrap.cc:492] T e2441aa3c2c748bba5360259fb663c81 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:36.494182 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet bf459db3327640f88574fca7b5f2ca98 (table default.Table_f6ddd620f29a4333a8f71f99316e1a8d [id=b6932927b05f46be967419e342532b3f]) successfully deleted
I20250903 20:05:36.494314 27692 ts_tablet_manager.cc:1397] T e2441aa3c2c748bba5360259fb663c81 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.141s user 0.003s sys 0.001s
I20250903 20:05:36.494644 27687 tablet_replica.cc:331] T a38f65e992884635aae9942681c0dc4f P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:36.494993 27687 raft_consensus.cc:2241] T a38f65e992884635aae9942681c0dc4f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:36.495086 27692 raft_consensus.cc:357] T e2441aa3c2c748bba5360259fb663c81 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.495198 27687 raft_consensus.cc:2270] T a38f65e992884635aae9942681c0dc4f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:36.495280 27692 raft_consensus.cc:383] T e2441aa3c2c748bba5360259fb663c81 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:36.495347 27692 raft_consensus.cc:738] T e2441aa3c2c748bba5360259fb663c81 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:36.495553 27692 consensus_queue.cc:260] T e2441aa3c2c748bba5360259fb663c81 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.496098 27687 ts_tablet_manager.cc:1905] T a38f65e992884635aae9942681c0dc4f P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:36.496129 27692 ts_tablet_manager.cc:1428] T e2441aa3c2c748bba5360259fb663c81 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:36.504412 27692 tablet_bootstrap.cc:492] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:36.510295 27222 tablet_service.cc:1468] Processing CreateTablet for tablet 7bb0da31c68440788dc296e67b38ec7c (DEFAULT_TABLE table=default.Table_6b71f7836da34d58a17dacbfb7bcac07 [id=d62014ebcbe1451b832b1f38a036418a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:36.514089 27066 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 88f74467a168473ab81d6fad2bddacab consensus state has no leader
I20250903 20:05:36.519301 29371 raft_consensus.cc:491] T d4c703dafbca4e669dbb18946ffac699 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:36.519604 29371 raft_consensus.cc:513] T d4c703dafbca4e669dbb18946ffac699 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.520026 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bb0da31c68440788dc296e67b38ec7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.520270 29371 leader_election.cc:290] T d4c703dafbca4e669dbb18946ffac699 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:36.521417 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4c703dafbca4e669dbb18946ffac699" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:36.521435 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4c703dafbca4e669dbb18946ffac699" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:36.522172 27462 leader_election.cc:343] T d4c703dafbca4e669dbb18946ffac699 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:36.522486 27462 leader_election.cc:343] T d4c703dafbca4e669dbb18946ffac699 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:36.522614 27695 tablet_bootstrap.cc:492] T cb2f50683e904dd39ee6eb128717d38c P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:36.522636 27462 leader_election.cc:304] T d4c703dafbca4e669dbb18946ffac699 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:36.523162 29371 raft_consensus.cc:2747] T d4c703dafbca4e669dbb18946ffac699 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:36.524199 27695 tablet_bootstrap.cc:654] T cb2f50683e904dd39ee6eb128717d38c P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:36.541749 27692 tablet_bootstrap.cc:654] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:36.548849 27066 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 8ba2636641654d8596fd2a5b0644d073 (table default.Table_e7148e16606e4cd3908c86b439f75b78 [id=4bffa29ea98049718ac3ce35de9813c5]). Expected version 1 got 0
I20250903 20:05:36.549832 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0a08d5d27dd24365b9e527123d15e771" } modify_external_catalogs: true
I20250903 20:05:36.550019 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0a08d5d27dd24365b9e527123d15e771" } modify_external_catalogs: true
I20250903 20:05:36.550472 27057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_66ee6081d8504c119f365516fc8be9f5" } new_table_name: "default.Table_32f55aced0c7467abc26722515eb5fb2" modify_external_catalogs: true
I20250903 20:05:36.551872 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_9f971775443a46128184a77aa0233d86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:36.552450 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f971775443a46128184a77aa0233d86 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:36.554664 27058 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_9f63732bf0ff4219886e8692abc459ce [id=91abc1158ea047849f398926e3695927] alter complete (version 2)
I20250903 20:05:36.555475 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e27e9a52e38149619ab852649bff1d03" } modify_external_catalogs: true
I20250903 20:05:36.555616 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e27e9a52e38149619ab852649bff1d03" } modify_external_catalogs: true
I20250903 20:05:36.556635 27695 tablet_bootstrap.cc:492] T cb2f50683e904dd39ee6eb128717d38c P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:36.556769 27695 ts_tablet_manager.cc:1397] T cb2f50683e904dd39ee6eb128717d38c P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.006s
I20250903 20:05:36.557574 27500 tablet_service.cc:1515] Processing DeleteTablet for tablet 13496efe34944b7280e7a684f180e6e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:36.557523 27695 raft_consensus.cc:357] T cb2f50683e904dd39ee6eb128717d38c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:36.557695 27695 raft_consensus.cc:383] T cb2f50683e904dd39ee6eb128717d38c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:36.557749 27695 raft_consensus.cc:738] T cb2f50683e904dd39ee6eb128717d38c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:36.557941 27695 consensus_queue.cc:260] T cb2f50683e904dd39ee6eb128717d38c P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:36.558470 27695 ts_tablet_manager.cc:1428] T cb2f50683e904dd39ee6eb128717d38c P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:36.558748 27695 tablet_bootstrap.cc:492] T 64c4dd2645034a7eaf243f5474603f10 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:36.560240 27695 tablet_bootstrap.cc:654] T 64c4dd2645034a7eaf243f5474603f10 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:36.561856 27059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fa2014bccb5e4f0a91cbc9845df4634d consensus state has no leader
I20250903 20:05:36.561897 27062 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 8ba2636641654d8596fd2a5b0644d073 (table default.Table_e7148e16606e4cd3908c86b439f75b78 [id=4bffa29ea98049718ac3ce35de9813c5]). Expected version 1 got 0
I20250903 20:05:36.562121 27059 master_service.cc:948] ReplaceTablet: received request to replace tablet e35574d360334150bb494e13f55d3656 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:36.563272 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_f55e6f36ed164f81ba558bce123e1292"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:36.563815 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f55e6f36ed164f81ba558bce123e1292 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:36.568076 27692 tablet_bootstrap.cc:492] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:36.568228 27692 ts_tablet_manager.cc:1397] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.000s
I20250903 20:05:36.568828 27059 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 8ba2636641654d8596fd2a5b0644d073 (table default.Table_e7148e16606e4cd3908c86b439f75b78 [id=4bffa29ea98049718ac3ce35de9813c5]). Expected version 1 got 0
I20250903 20:05:36.568971 27692 raft_consensus.cc:357] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.569141 27692 raft_consensus.cc:383] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:36.569200 27692 raft_consensus.cc:738] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:36.569450 27692 consensus_queue.cc:260] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.569941 27692 ts_tablet_manager.cc:1428] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:36.572590 27685 ts_tablet_manager.cc:1918] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250903 20:05:36.576989 27057 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e65eac0cf8fa4a29927db5149d39a3f6 consensus state has no leader
W20250903 20:05:36.577291 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cab87148f7b84222915bf8eefc78a5f1 consensus state has no leader
I20250903 20:05:36.580331 29371 consensus_queue.cc:1035] T 472f0fb01263468a828ec13795656624 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:36.582358 27500 tablet_service.cc:1515] Processing DeleteTablet for tablet e2441aa3c2c748bba5360259fb663c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:36.600297 27692 tablet_bootstrap.cc:492] T f4d3e1cfdfab4be1a9143b78bf3bc46d P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:36.609086 27500 tablet_service.cc:1515] Processing DeleteTablet for tablet 0fef9aa7153a4902abce522e3ff0df3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:36.617080 27684 ts_tablet_manager.cc:1918] T f1d22e76ea72441db51f7e9db26eb520 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:36.617337 27684 log.cc:1199] T f1d22e76ea72441db51f7e9db26eb520 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/f1d22e76ea72441db51f7e9db26eb520
I20250903 20:05:36.617760 27684 ts_tablet_manager.cc:1939] T f1d22e76ea72441db51f7e9db26eb520 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:36.618031 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 827d186faaa140bdab2f87cf421ca64c
I20250903 20:05:36.622763 27500 tablet_service.cc:1515] Processing DeleteTablet for tablet 827d186faaa140bdab2f87cf421ca64c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:36.624377 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6e9cf5ead5a24831a6a14d131dd8397e" } modify_external_catalogs: true
I20250903 20:05:36.624574 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6e9cf5ead5a24831a6a14d131dd8397e" } modify_external_catalogs: true
I20250903 20:05:36.626005 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_fa0af1d57fb048e7ab3e84f79c1f674d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:36.626621 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa0af1d57fb048e7ab3e84f79c1f674d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:36.628366 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_0f58bdb4d1eb4aa9bb8d36909880410b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:36.628887 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f58bdb4d1eb4aa9bb8d36909880410b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:36.634490 27692 tablet_bootstrap.cc:654] T f4d3e1cfdfab4be1a9143b78bf3bc46d P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:36.582549 27695 tablet_bootstrap.cc:492] T 64c4dd2645034a7eaf243f5474603f10 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:36.635452 27685 log.cc:1199] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/567630e2a93645e58f9bf05ffff87b32
I20250903 20:05:36.635504 27695 ts_tablet_manager.cc:1397] T 64c4dd2645034a7eaf243f5474603f10 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.077s user 0.003s sys 0.000s
I20250903 20:05:36.635900 27685 ts_tablet_manager.cc:1939] T 567630e2a93645e58f9bf05ffff87b32 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:36.636240 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 996) took 3803 ms. Trace:
I20250903 20:05:36.639103 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1099) took 2061 ms. Trace:
I20250903 20:05:36.639312 27684 rpcz_store.cc:276] 0903 20:05:34.577952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.578504 (+ 552us) service_pool.cc:225] Handling call
0903 20:05:36.451733 (+1873229us) ts_tablet_manager.cc:1145] Deleting tablet f1d22e76ea72441db51f7e9db26eb520
0903 20:05:36.451741 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.499092 (+ 47351us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.616938 (+117846us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.639079 (+ 22141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1872987}
I20250903 20:05:36.640101 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1100) took 2059 ms. Trace:
I20250903 20:05:36.619203 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 827d186faaa140bdab2f87cf421ca64c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:36.640242 27684 rpcz_store.cc:276] 0903 20:05:34.580328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.580393 (+ 65us) service_pool.cc:225] Handling call
0903 20:05:36.639889 (+2059496us) ts_tablet_manager.cc:1145] Deleting tablet a2358eac86914f3ba51bfc6c299838c3
0903 20:05:36.639899 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.640084 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2059271}
I20250903 20:05:36.636335 27695 raft_consensus.cc:357] T 64c4dd2645034a7eaf243f5474603f10 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.640686 27695 raft_consensus.cc:383] T 64c4dd2645034a7eaf243f5474603f10 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:36.640784 27695 raft_consensus.cc:738] T 64c4dd2645034a7eaf243f5474603f10 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:36.641026 27695 consensus_queue.cc:260] T 64c4dd2645034a7eaf243f5474603f10 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.641152 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1102) took 2045 ms. Trace:
I20250903 20:05:36.640743 27685 rpcz_store.cc:276] 0903 20:05:32.832374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.837273 (+ 4899us) service_pool.cc:225] Handling call
0903 20:05:36.295653 (+3458380us) ts_tablet_manager.cc:1145] Deleting tablet 567630e2a93645e58f9bf05ffff87b32
0903 20:05:36.295665 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.568119 (+272454us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.572488 (+ 4369us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.636222 (+ 63734us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3458161}
I20250903 20:05:36.641304 27684 rpcz_store.cc:276] 0903 20:05:34.595803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.599198 (+ 3395us) service_pool.cc:225] Handling call
0903 20:05:36.640740 (+2041542us) ts_tablet_manager.cc:1145] Deleting tablet 54f1ce5192f547f595dea23a9eabf8e6
0903 20:05:36.640750 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.641133 (+ 383us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2041166}
I20250903 20:05:36.641794 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1103) took 2024 ms. Trace:
I20250903 20:05:36.641918 27684 rpcz_store.cc:276] 0903 20:05:34.617285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.622863 (+ 5578us) service_pool.cc:225] Handling call
0903 20:05:36.641700 (+2018837us) ts_tablet_manager.cc:1145] Deleting tablet a785adcd0bd943fc97b201b6d29073f4
0903 20:05:36.641709 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.641781 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2005082}
I20250903 20:05:36.642589 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1104) took 2005 ms. Trace:
I20250903 20:05:36.642716 27684 rpcz_store.cc:276] 0903 20:05:34.637497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.645037 (+ 7540us) service_pool.cc:225] Handling call
0903 20:05:36.642491 (+1997454us) ts_tablet_manager.cc:1145] Deleting tablet 6800e0b6269045f29682704fd847d5d1
0903 20:05:36.642500 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.642575 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1997186}
I20250903 20:05:36.643194 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1105) took 1988 ms. Trace:
I20250903 20:05:36.643909 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a0858c440ad84423888fcb05f56ca68d" } modify_external_catalogs: true
I20250903 20:05:36.644106 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_a0858c440ad84423888fcb05f56ca68d" } modify_external_catalogs: true
I20250903 20:05:36.646453 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e997a321ad0c45dc9258399326f0830d" } modify_external_catalogs: true
I20250903 20:05:36.646621 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e997a321ad0c45dc9258399326f0830d" } modify_external_catalogs: true
I20250903 20:05:36.648099 27347 tablet_service.cc:1515] Processing DeleteTablet for tablet 827d186faaa140bdab2f87cf421ca64c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:36.097944 27693 consensus_queue.cc:260] T 2385d3006f90486890ebe43e16a53d7a P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.641606 27695 ts_tablet_manager.cc:1428] T 64c4dd2645034a7eaf243f5474603f10 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250903 20:05:36.651194 27695 tablet_bootstrap.cc:492] T 3919f7f95a954f59b68f56d0051c0109 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:36.652789 27695 tablet_bootstrap.cc:654] T 3919f7f95a954f59b68f56d0051c0109 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:36.653337 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 0f5b87c1c4a24f8abdbfd6194f3b2c4a
I20250903 20:05:36.608067 29285 raft_consensus.cc:491] T e2441aa3c2c748bba5360259fb663c81 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:36.653709 29285 raft_consensus.cc:513] T e2441aa3c2c748bba5360259fb663c81 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.654381 29285 leader_election.cc:290] T e2441aa3c2c748bba5360259fb663c81 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:36.656674 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet f1d22e76ea72441db51f7e9db26eb520 (table default.Table_258f358903dd4f6fa7e310d72562a570 [id=ae5e27c159a9462482d09a79e148f795]) successfully deleted
W20250903 20:05:36.657037 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet a2358eac86914f3ba51bfc6c299838c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2358eac86914f3ba51bfc6c299838c3 already in progress: creating tablet
W20250903 20:05:36.657327 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 54f1ce5192f547f595dea23a9eabf8e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54f1ce5192f547f595dea23a9eabf8e6 already in progress: creating tablet
W20250903 20:05:36.657605 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet a785adcd0bd943fc97b201b6d29073f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a785adcd0bd943fc97b201b6d29073f4 already in progress: creating tablet
W20250903 20:05:36.657846 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 6800e0b6269045f29682704fd847d5d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6800e0b6269045f29682704fd847d5d1 already in progress: creating tablet
W20250903 20:05:36.646525 27184 consensus_peers.cc:489] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa9f5e0964b348b7af184f60cd145666. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:36.669826 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f5b87c1c4a24f8abdbfd6194f3b2c4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:36.671160 27695 tablet_bootstrap.cc:492] T 3919f7f95a954f59b68f56d0051c0109 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:36.671404 27695 ts_tablet_manager.cc:1397] T 3919f7f95a954f59b68f56d0051c0109 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.001s
I20250903 20:05:36.672219 27695 raft_consensus.cc:357] T 3919f7f95a954f59b68f56d0051c0109 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.672585 27695 raft_consensus.cc:383] T 3919f7f95a954f59b68f56d0051c0109 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:36.672695 27695 raft_consensus.cc:738] T 3919f7f95a954f59b68f56d0051c0109 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:36.672981 27695 consensus_queue.cc:260] T 3919f7f95a954f59b68f56d0051c0109 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.677610 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_fdf6ca8eecbb4d1cb0fd5fcd4e67b5a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:36.678392 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fdf6ca8eecbb4d1cb0fd5fcd4e67b5a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:36.679044 27695 ts_tablet_manager.cc:1428] T 3919f7f95a954f59b68f56d0051c0109 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250903 20:05:36.679471 27695 tablet_bootstrap.cc:492] T 3f303233e6c347289650c6cf83f8127f P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:36.683943 27695 tablet_bootstrap.cc:654] T 3f303233e6c347289650c6cf83f8127f P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:36.656075 27500 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f5b87c1c4a24f8abdbfd6194f3b2c4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:36.664477 27508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2441aa3c2c748bba5360259fb663c81" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:36.689390 27508 raft_consensus.cc:2466] T e2441aa3c2c748bba5360259fb663c81 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:36.689854 27684 rpcz_store.cc:276] 0903 20:05:34.654893 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.654953 (+ 60us) service_pool.cc:225] Handling call
0903 20:05:36.643102 (+1988149us) ts_tablet_manager.cc:1145] Deleting tablet 8bebea9fa64e459cb24639e1f2d11ac3
0903 20:05:36.643111 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.643181 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1987812}
W20250903 20:05:36.691601 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 8bebea9fa64e459cb24639e1f2d11ac3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8bebea9fa64e459cb24639e1f2d11ac3 already in progress: creating tablet
I20250903 20:05:36.692060 27684 tablet_replica.cc:331] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:36.692565 27684 raft_consensus.cc:2241] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:36.693746 27183 leader_election.cc:304] T e2441aa3c2c748bba5360259fb663c81 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:36.694046 29199 raft_consensus.cc:2802] T e2441aa3c2c748bba5360259fb663c81 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:36.694160 29199 raft_consensus.cc:491] T e2441aa3c2c748bba5360259fb663c81 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:36.694260 29199 raft_consensus.cc:3058] T e2441aa3c2c748bba5360259fb663c81 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:36.695535 27684 raft_consensus.cc:2270] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:36.696136 27684 ts_tablet_manager.cc:1905] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:36.696529 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 567630e2a93645e58f9bf05ffff87b32 (table default.Table_f2d0d1d8bddc492dbccd2c5c0a4e1a16 [id=3348ac5023a64fe3b19aed6c9471d580]) successfully deleted
I20250903 20:05:36.696655 27685 tablet_replica.cc:331] T 5002b7e8118c49b6a247cb8dc4ca1aea P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:36.696866 27685 raft_consensus.cc:2241] T 5002b7e8118c49b6a247cb8dc4ca1aea P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:36.697075 27685 raft_consensus.cc:2270] T 5002b7e8118c49b6a247cb8dc4ca1aea P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:36.695176 27692 tablet_bootstrap.cc:492] T f4d3e1cfdfab4be1a9143b78bf3bc46d P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:36.697467 29199 raft_consensus.cc:513] T e2441aa3c2c748bba5360259fb663c81 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.697695 27685 ts_tablet_manager.cc:1905] T 5002b7e8118c49b6a247cb8dc4ca1aea P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:36.697731 27692 ts_tablet_manager.cc:1397] T f4d3e1cfdfab4be1a9143b78bf3bc46d P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.098s user 0.004s sys 0.000s
I20250903 20:05:36.698681 27692 raft_consensus.cc:357] T f4d3e1cfdfab4be1a9143b78bf3bc46d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.698861 27692 raft_consensus.cc:383] T f4d3e1cfdfab4be1a9143b78bf3bc46d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:36.663801 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2441aa3c2c748bba5360259fb663c81" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:36.698921 27692 raft_consensus.cc:738] T f4d3e1cfdfab4be1a9143b78bf3bc46d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:36.698169 29199 leader_election.cc:290] T e2441aa3c2c748bba5360259fb663c81 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:36.699680 27508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2441aa3c2c748bba5360259fb663c81" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:36.699841 27508 raft_consensus.cc:3058] T e2441aa3c2c748bba5360259fb663c81 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:36.702384 27184 leader_election.cc:343] T e2441aa3c2c748bba5360259fb663c81 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:36.702781 27508 raft_consensus.cc:2466] T e2441aa3c2c748bba5360259fb663c81 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:36.703088 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2441aa3c2c748bba5360259fb663c81" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
W20250903 20:05:36.703649 27184 leader_election.cc:343] T e2441aa3c2c748bba5360259fb663c81 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:36.703661 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7197fb56c22c40948ede1c2959901507" } new_table_name: "default.Table_2e2dd17c15234bb3836b57cd2989c12f" modify_external_catalogs: true
I20250903 20:05:36.699144 27692 consensus_queue.cc:260] T f4d3e1cfdfab4be1a9143b78bf3bc46d P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.704023 27183 leader_election.cc:304] T e2441aa3c2c748bba5360259fb663c81 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:36.704361 29199 raft_consensus.cc:2802] T e2441aa3c2c748bba5360259fb663c81 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:36.704905 29199 raft_consensus.cc:695] T e2441aa3c2c748bba5360259fb663c81 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:36.705256 29199 consensus_queue.cc:237] T e2441aa3c2c748bba5360259fb663c81 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.709093 27065 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet f4d3e1cfdfab4be1a9143b78bf3bc46d (table default.Table_5e672ed0c8c142029a79b72f799f3f59 [id=386b8d4e7c4042d1947c19ac464c75a0]). Expected version 2 got 0
I20250903 20:05:36.710788 27692 ts_tablet_manager.cc:1428] T f4d3e1cfdfab4be1a9143b78bf3bc46d P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250903 20:05:36.670512 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f5b87c1c4a24f8abdbfd6194f3b2c4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:36.716161 27695 tablet_bootstrap.cc:492] T 3f303233e6c347289650c6cf83f8127f P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:36.717142 27693 ts_tablet_manager.cc:1428] T 2385d3006f90486890ebe43e16a53d7a P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.627s user 0.018s sys 0.000s
I20250903 20:05:36.720270 27061 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet f4d3e1cfdfab4be1a9143b78bf3bc46d (table default.Table_5e672ed0c8c142029a79b72f799f3f59 [id=386b8d4e7c4042d1947c19ac464c75a0]). Expected version 2 got 0
I20250903 20:05:36.719252 27692 tablet_bootstrap.cc:492] T 6800e0b6269045f29682704fd847d5d1 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:36.730597 29371 raft_consensus.cc:491] T 30cc0d4ba8d444f1b98232f906ea0c7b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:36.730768 29371 raft_consensus.cc:513] T 30cc0d4ba8d444f1b98232f906ea0c7b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.730839 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_a20d16f1ce384a83998a23435eb76dff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:36.731285 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f55e6f36ed164f81ba558bce123e1292" } new_table_name: "default.Table_2ae7f1d891744aeca75e2ed6bf43666f" modify_external_catalogs: true
W20250903 20:05:36.731400 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a20d16f1ce384a83998a23435eb76dff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:36.732560 29371 leader_election.cc:290] T 30cc0d4ba8d444f1b98232f906ea0c7b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:36.732720 27693 ts_tablet_manager.cc:1460] T 2385d3006f90486890ebe43e16a53d7a P 51578fb7bceb4e119b3ab1e58c4f5528: Tablet startup took 642ms
W20250903 20:05:36.732789 27693 ts_tablet_manager.cc:1462] T 2385d3006f90486890ebe43e16a53d7a P 51578fb7bceb4e119b3ab1e58c4f5528: Trace:
0903 20:05:16.355950 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:16.356128 (+ 178us) service_pool.cc:225] Handling call
0903 20:05:16.369207 (+ 13079us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0903 20:05:16.369274 (+ 67us) ts_tablet_manager.cc:676] Creating new metadata...
0903 20:05:16.434482 (+ 65208us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:16.437582 (+ 3100us) tablet_replica.cc:206] Creating consensus instance
0903 20:05:16.437761 (+ 179us) inbound_call.cc:173] Queueing success response
0903 20:05:35.986540 (+19548779us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0903 20:05:36.090557 (+104017us) ts_tablet_manager.cc:1429] Starting tablet replica
0903 20:05:36.097136 (+ 6579us) tablet_replica.cc:260] Starting instrumentation
0903 20:05:36.097256 (+ 120us) tablet_replica.cc:279] Starting consensus
Metrics: {"raft.queue_time_us":51,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":92,"tablet-open.queue_time_us":19548829}
I20250903 20:05:36.732923 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_03367a125ef440d2bc8c3f1531dfb0b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:36.733393 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03367a125ef440d2bc8c3f1531dfb0b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:36.737010 27693 tablet_bootstrap.cc:492] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:36.738385 27693 tablet_bootstrap.cc:654] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:36.738871 27692 tablet_bootstrap.cc:654] T 6800e0b6269045f29682704fd847d5d1 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:36.746117 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ca722dfe38ac4e63b024508a3997cedb consensus state has no leader
I20250903 20:05:36.732388 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30cc0d4ba8d444f1b98232f906ea0c7b" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:36.759172 27462 leader_election.cc:343] T 30cc0d4ba8d444f1b98232f906ea0c7b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:36.759547 27695 ts_tablet_manager.cc:1397] T 3f303233e6c347289650c6cf83f8127f P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.080s user 0.002s sys 0.002s
I20250903 20:05:36.760483 27695 raft_consensus.cc:357] T 3f303233e6c347289650c6cf83f8127f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.760728 27695 raft_consensus.cc:383] T 3f303233e6c347289650c6cf83f8127f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:36.760836 27695 raft_consensus.cc:738] T 3f303233e6c347289650c6cf83f8127f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:36.761128 27695 consensus_queue.cc:260] T 3f303233e6c347289650c6cf83f8127f P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.761461 27684 ts_tablet_manager.cc:1918] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:36.761577 27684 log.cc:1199] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/a38f65e992884635aae9942681c0dc4f
I20250903 20:05:36.762758 27695 ts_tablet_manager.cc:1428] T 3f303233e6c347289650c6cf83f8127f P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250903 20:05:36.763314 27695 tablet_bootstrap.cc:492] T 4e9a6a6e05464b7b97ae624f51402cf0 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:36.769800 27692 tablet_bootstrap.cc:492] T 6800e0b6269045f29682704fd847d5d1 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:36.770021 27692 ts_tablet_manager.cc:1397] T 6800e0b6269045f29682704fd847d5d1 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.000s
I20250903 20:05:36.771121 27692 raft_consensus.cc:357] T 6800e0b6269045f29682704fd847d5d1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:36.771375 27692 raft_consensus.cc:383] T 6800e0b6269045f29682704fd847d5d1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:36.771452 27692 raft_consensus.cc:738] T 6800e0b6269045f29682704fd847d5d1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:36.771718 27692 consensus_queue.cc:260] T 6800e0b6269045f29682704fd847d5d1 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:36.772372 27692 ts_tablet_manager.cc:1428] T 6800e0b6269045f29682704fd847d5d1 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250903 20:05:36.772715 27692 tablet_bootstrap.cc:492] T b4b15d77d07c4fa7856d9c198cf15450 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:36.718494 27687 ts_tablet_manager.cc:1918] T a38f65e992884635aae9942681c0dc4f P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:36.731662 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30cc0d4ba8d444f1b98232f906ea0c7b" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:36.775023 27462 leader_election.cc:343] T 30cc0d4ba8d444f1b98232f906ea0c7b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:36.775182 27462 leader_election.cc:304] T 30cc0d4ba8d444f1b98232f906ea0c7b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:36.775915 27687 log.cc:1199] T a38f65e992884635aae9942681c0dc4f P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/a38f65e992884635aae9942681c0dc4f
I20250903 20:05:36.776635 27687 ts_tablet_manager.cc:1939] T a38f65e992884635aae9942681c0dc4f P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:36.777675 29371 raft_consensus.cc:2747] T 30cc0d4ba8d444f1b98232f906ea0c7b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:36.786178 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b78645d4e9574ce3b2a78eebb541fbfe has no consensus state
I20250903 20:05:36.786507 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1125) took 2081 ms. Trace:
I20250903 20:05:36.788229 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_96bef96c2061427d8b6643208730463c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:36.788769 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96bef96c2061427d8b6643208730463c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:36.788187 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_092cc68e6e774d91ba49dc1622f26b0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:36.790381 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_092cc68e6e774d91ba49dc1622f26b0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:36.793202 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_92312eea209d4177bae77b8fa25fb7a8" } new_table_name: "default.Table_ced17472d1d340a29cd08896d995cc87" modify_external_catalogs: true
I20250903 20:05:36.786614 27687 rpcz_store.cc:276] 0903 20:05:34.705133 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.705186 (+ 53us) service_pool.cc:225] Handling call
0903 20:05:36.494592 (+1789406us) ts_tablet_manager.cc:1145] Deleting tablet a38f65e992884635aae9942681c0dc4f
0903 20:05:36.494602 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.610168 (+115566us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.718363 (+108195us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.786474 (+ 68111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1789232}
I20250903 20:05:36.794021 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet a38f65e992884635aae9942681c0dc4f (table default.Table_bb8c68be79034a5c8469862c43953131 [id=f0171ee670294ce58ab84483fd71ab7e]) successfully deleted
I20250903 20:05:36.794406 27687 tablet_replica.cc:331] T 548b50819f694fa3ad508da9f442a99c P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:36.794595 27687 raft_consensus.cc:2241] T 548b50819f694fa3ad508da9f442a99c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:36.794611 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e019b2e0ac534620ad809b330da7038b" } new_table_name: "default.Table_d17ef5956d5f4744b7d7e9ebfddd612b" modify_external_catalogs: true
I20250903 20:05:36.794747 27687 raft_consensus.cc:2270] T 548b50819f694fa3ad508da9f442a99c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:36.795284 27687 ts_tablet_manager.cc:1905] T 548b50819f694fa3ad508da9f442a99c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:36.797533 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6120f4e2a1c340b989f03b20673de1a6" } new_table_name: "default.Table_d8e0617b6b194aa382092114902f3702" modify_external_catalogs: true
I20250903 20:05:36.797808 27500 tablet_service.cc:1515] Processing DeleteTablet for tablet 6800e0b6269045f29682704fd847d5d1 with delete_type TABLET_DATA_DELETED (replaced by tablet a3cf18a9a8c94a58bb739ee2f81d19c0) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:36.800704 27692 tablet_bootstrap.cc:654] T b4b15d77d07c4fa7856d9c198cf15450 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:36.801237 27684 ts_tablet_manager.cc:1939] T a38f65e992884635aae9942681c0dc4f P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:36.802234 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1106) took 2113 ms. Trace:
I20250903 20:05:36.802402 27684 rpcz_store.cc:276] 0903 20:05:34.689240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.689302 (+ 62us) service_pool.cc:225] Handling call
0903 20:05:36.691989 (+2002687us) ts_tablet_manager.cc:1145] Deleting tablet a38f65e992884635aae9942681c0dc4f
0903 20:05:36.692002 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.758199 (+ 66197us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.761354 (+ 3155us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.802186 (+ 40832us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2002462}
I20250903 20:05:36.802841 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet a38f65e992884635aae9942681c0dc4f (table default.Table_bb8c68be79034a5c8469862c43953131 [id=f0171ee670294ce58ab84483fd71ab7e]) successfully deleted
I20250903 20:05:36.803166 27684 tablet_replica.cc:331] T f326ec776ba446be8d5dda5c1dc5b6ec P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:36.803354 27684 raft_consensus.cc:2241] T f326ec776ba446be8d5dda5c1dc5b6ec P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:36.805874 27062 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 8ba2636641654d8596fd2a5b0644d073 (table default.Table_e7148e16606e4cd3908c86b439f75b78 [id=4bffa29ea98049718ac3ce35de9813c5]). Expected version 1 got 0
I20250903 20:05:36.807315 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_781c5fddf2e945b5bf4c68483f03dde5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:36.808477 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_781c5fddf2e945b5bf4c68483f03dde5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:36.810941 27684 raft_consensus.cc:2270] T f326ec776ba446be8d5dda5c1dc5b6ec P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:36.812359 27684 ts_tablet_manager.cc:1905] T f326ec776ba446be8d5dda5c1dc5b6ec P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:36.814386 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet cb2f50683e904dd39ee6eb128717d38c with delete_type TABLET_DATA_DELETED (replaced by tablet 72fcb7b0cba34313a7dc2b8c29e120fc) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:36.820091 27349 tablet_service.cc:1468] Processing CreateTablet for tablet 789f930df8804d73a30bb7e91aa0f95b (DEFAULT_TABLE table=default.Table_1c303bbec4ab48e48656cc3a86cffea0 [id=7d64e9ce87bc4e298905ea66d4e3ee20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.820498 27347 tablet_service.cc:1468] Processing CreateTablet for tablet b6ccfe45ba01444fa42ededc6381f225 (DEFAULT_TABLE table=default.Table_1b1798fd0780489a935aa84d7ea2995a [id=c35287cdb05f432399fb8aebc9a62fc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.820823 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 789f930df8804d73a30bb7e91aa0f95b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.821658 27357 tablet_service.cc:1468] Processing CreateTablet for tablet c6bf6f0182f346bdb92720f8506e9a13 (DEFAULT_TABLE table=default.Table_d99e1237827942879902dd30ea22ba38 [id=1a9347c03d734ccab96c8f864691b922]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.822072 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6bf6f0182f346bdb92720f8506e9a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.822392 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 09229109533c49ff9f9824cadb6e2e7e (DEFAULT_TABLE table=default.Table_91cfcef3088c4292bdc92b3cac245994 [id=eac9e0fb6abb46f9b8b25f2a795145b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.822439 27350 tablet_service.cc:1468] Processing CreateTablet for tablet ab380b2765b142c890c6010fe8d00948 (DEFAULT_TABLE table=default.Table_7ad5f88730c24cbbb97feda4328aa2ec [id=60842e6c321b4bf28c9ca298b0d656f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.822747 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09229109533c49ff9f9824cadb6e2e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.826826 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab380b2765b142c890c6010fe8d00948. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:36.834203 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c51076f613fa46a8bf1309efba28195f consensus state has no leader
I20250903 20:05:36.834785 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_aeea04c58a4643328ab1223efbed4a4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:36.835626 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aeea04c58a4643328ab1223efbed4a4a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:36.835729 27059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ec8598ef27cc470fa48938de8d5196fc consensus state has no leader
I20250903 20:05:36.836359 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6ccfe45ba01444fa42ededc6381f225. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:36.837502 26890 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 91678.4us
I20250903 20:05:36.840062 29220 consensus_queue.cc:1035] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:36.841296 27500 tablet_service.cc:1468] Processing CreateTablet for tablet 789f930df8804d73a30bb7e91aa0f95b (DEFAULT_TABLE table=default.Table_1c303bbec4ab48e48656cc3a86cffea0 [id=7d64e9ce87bc4e298905ea66d4e3ee20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.841660 27495 tablet_service.cc:1468] Processing CreateTablet for tablet 09229109533c49ff9f9824cadb6e2e7e (DEFAULT_TABLE table=default.Table_91cfcef3088c4292bdc92b3cac245994 [id=eac9e0fb6abb46f9b8b25f2a795145b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.841935 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 789f930df8804d73a30bb7e91aa0f95b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.842162 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09229109533c49ff9f9824cadb6e2e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.842685 27496 tablet_service.cc:1468] Processing CreateTablet for tablet d2c255209a594cea9312a3e9ede0ac6e (DEFAULT_TABLE table=default.Table_ff9d1c55236648f295a6715fd27166f7 [id=a076dbaaf3e1492ab94cc6e4dc9476d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.842989 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2c255209a594cea9312a3e9ede0ac6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.843346 27498 tablet_service.cc:1468] Processing CreateTablet for tablet 0d36f01cb01141d1a6f371ed88d5c584 (DEFAULT_TABLE table=default.Table_949650a78270483689230d33cb7c4a3b [id=561ca09258ca4da5b836254126753ea5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.843631 27498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d36f01cb01141d1a6f371ed88d5c584. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.845588 27503 tablet_service.cc:1468] Processing CreateTablet for tablet b6ccfe45ba01444fa42ededc6381f225 (DEFAULT_TABLE table=default.Table_1b1798fd0780489a935aa84d7ea2995a [id=c35287cdb05f432399fb8aebc9a62fc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.845851 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6ccfe45ba01444fa42ededc6381f225. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.850375 27489 tablet_service.cc:1468] Processing CreateTablet for tablet ab380b2765b142c890c6010fe8d00948 (DEFAULT_TABLE table=default.Table_7ad5f88730c24cbbb97feda4328aa2ec [id=60842e6c321b4bf28c9ca298b0d656f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.841296 27488 tablet_service.cc:1468] Processing CreateTablet for tablet c6bf6f0182f346bdb92720f8506e9a13 (DEFAULT_TABLE table=default.Table_d99e1237827942879902dd30ea22ba38 [id=1a9347c03d734ccab96c8f864691b922]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.850862 27489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab380b2765b142c890c6010fe8d00948. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.854971 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 5029c403fbc34dc0aa535d21018e170f (DEFAULT_TABLE table=default.Table_ec7b30148c954f9e8eff70562aed4edb [id=9d92dbfedf09485b8ad8656ed0c30d40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.855434 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5029c403fbc34dc0aa535d21018e170f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.856555 27357 tablet_service.cc:1468] Processing CreateTablet for tablet 0d36f01cb01141d1a6f371ed88d5c584 (DEFAULT_TABLE table=default.Table_949650a78270483689230d33cb7c4a3b [id=561ca09258ca4da5b836254126753ea5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.857172 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6bf6f0182f346bdb92720f8506e9a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.858892 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d36f01cb01141d1a6f371ed88d5c584. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.859213 27352 tablet_service.cc:1468] Processing CreateTablet for tablet 9942bb4b868d4a08866298d7790ecad2 (DEFAULT_TABLE table=default.Table_c86263c481ce4c5ebeb98439c682f1a2 [id=fb1b537879be4681a7f1a764a3ec6914]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.859545 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9942bb4b868d4a08866298d7790ecad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.858425 27693 tablet_bootstrap.cc:492] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:36.860302 27693 ts_tablet_manager.cc:1397] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.124s user 0.002s sys 0.001s
I20250903 20:05:36.860986 27693 raft_consensus.cc:357] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.850375 27498 tablet_service.cc:1468] Processing CreateTablet for tablet 5029c403fbc34dc0aa535d21018e170f (DEFAULT_TABLE table=default.Table_ec7b30148c954f9e8eff70562aed4edb [id=9d92dbfedf09485b8ad8656ed0c30d40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.864082 27498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5029c403fbc34dc0aa535d21018e170f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.867899 27489 tablet_service.cc:1468] Processing CreateTablet for tablet 9942bb4b868d4a08866298d7790ecad2 (DEFAULT_TABLE table=default.Table_c86263c481ce4c5ebeb98439c682f1a2 [id=fb1b537879be4681a7f1a764a3ec6914]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.868363 27489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9942bb4b868d4a08866298d7790ecad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.862552 27359 tablet_service.cc:1468] Processing CreateTablet for tablet 0383bfbf13a74bb18b25528340136950 (DEFAULT_TABLE table=default.Table_23373e831f62480fa62bf3dfe3454bef [id=251324c651924eb6b13ca766d5a919ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.878774 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0383bfbf13a74bb18b25528340136950. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.878974 27503 tablet_service.cc:1468] Processing CreateTablet for tablet 0383bfbf13a74bb18b25528340136950 (DEFAULT_TABLE table=default.Table_23373e831f62480fa62bf3dfe3454bef [id=251324c651924eb6b13ca766d5a919ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.879385 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0383bfbf13a74bb18b25528340136950. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.880817 27353 tablet_service.cc:1468] Processing CreateTablet for tablet ee07d9c4743049d184a3a0f979b04ccd (DEFAULT_TABLE table=default.Table_5bcfed1071c9400ca66b8e9b8b2f26cc [id=d4392f49a2354586a30e403b2f292a97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.881152 27353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee07d9c4743049d184a3a0f979b04ccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.861150 27693 raft_consensus.cc:383] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:36.888134 27695 tablet_bootstrap.cc:654] T 4e9a6a6e05464b7b97ae624f51402cf0 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:36.887998 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 688cfe6155784b31ad50820bc61c4c4a (DEFAULT_TABLE table=default.Table_1124e9ecb8de4091bb37e8cc347dc8ab [id=fa5aa700613942db982b126b42b87974]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.888641 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 688cfe6155784b31ad50820bc61c4c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.888577 27348 tablet_service.cc:1468] Processing CreateTablet for tablet 2a056877a93342dd9d22ff50606731ca (DEFAULT_TABLE table=default.Table_9beb5c7440aa47549f15ef79b911dfea [id=6a022518eb934096bf49e092a98907ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.888669 27362 tablet_service.cc:1468] Processing CreateTablet for tablet e4315f4fa6cf4572b09550e06f675b10 (DEFAULT_TABLE table=default.Table_9533e6ddb5a84196b359ee85dd13e42c [id=322a3056681f48d7bd428e55227f883a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.888927 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a056877a93342dd9d22ff50606731ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.897624 27693 raft_consensus.cc:738] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:36.890033 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 09229109533c49ff9f9824cadb6e2e7e (DEFAULT_TABLE table=default.Table_91cfcef3088c4292bdc92b3cac245994 [id=eac9e0fb6abb46f9b8b25f2a795145b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.890798 27227 tablet_service.cc:1468] Processing CreateTablet for tablet 9942bb4b868d4a08866298d7790ecad2 (DEFAULT_TABLE table=default.Table_c86263c481ce4c5ebeb98439c682f1a2 [id=fb1b537879be4681a7f1a764a3ec6914]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.891234 27213 tablet_service.cc:1468] Processing CreateTablet for tablet 0383bfbf13a74bb18b25528340136950 (DEFAULT_TABLE table=default.Table_23373e831f62480fa62bf3dfe3454bef [id=251324c651924eb6b13ca766d5a919ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.891669 27226 tablet_service.cc:1468] Processing CreateTablet for tablet 42e618f4f375464a87eb7fb9e4265fb6 (DEFAULT_TABLE table=default.Table_0da8589e73bd4ad693cb4d5695b755c7 [id=04d67506921a44dab3638c5e00363291]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.891718 27229 tablet_service.cc:1468] Processing CreateTablet for tablet 9dd03e34d4d74dc791a08d441c19d8bb (DEFAULT_TABLE table=default.Table_4e126e6eaed846d186337d5a589023b6 [id=2af230da612c4c30a2701b0affdf2a00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.892086 27228 tablet_service.cc:1468] Processing CreateTablet for tablet ee07d9c4743049d184a3a0f979b04ccd (DEFAULT_TABLE table=default.Table_5bcfed1071c9400ca66b8e9b8b2f26cc [id=d4392f49a2354586a30e403b2f292a97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.892484 27224 tablet_service.cc:1468] Processing CreateTablet for tablet 688cfe6155784b31ad50820bc61c4c4a (DEFAULT_TABLE table=default.Table_1124e9ecb8de4091bb37e8cc347dc8ab [id=fa5aa700613942db982b126b42b87974]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.892887 27219 tablet_service.cc:1468] Processing CreateTablet for tablet 7006b7ee1926434cb8148b454c6f422d (DEFAULT_TABLE table=default.Table_a4a91ad2f79c465cadf33585679b2708 [id=489fa4ba6fab4b05a6c86f66a3e3cacd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.893308 27223 tablet_service.cc:1515] Processing DeleteTablet for tablet 3f303233e6c347289650c6cf83f8127f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:36.899673 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4315f4fa6cf4572b09550e06f675b10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.893343 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 2a056877a93342dd9d22ff50606731ca (DEFAULT_TABLE table=default.Table_9beb5c7440aa47549f15ef79b911dfea [id=6a022518eb934096bf49e092a98907ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.893568 27220 tablet_service.cc:1515] Processing DeleteTablet for tablet 64c4dd2645034a7eaf243f5474603f10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:36.893829 27211 tablet_service.cc:1515] Processing DeleteTablet for tablet 3919f7f95a954f59b68f56d0051c0109 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:19 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:36.894014 27214 tablet_service.cc:1468] Processing CreateTablet for tablet ab380b2765b142c890c6010fe8d00948 (DEFAULT_TABLE table=default.Table_7ad5f88730c24cbbb97feda4328aa2ec [id=60842e6c321b4bf28c9ca298b0d656f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.894358 27216 tablet_service.cc:1468] Processing CreateTablet for tablet d2c255209a594cea9312a3e9ede0ac6e (DEFAULT_TABLE table=default.Table_ff9d1c55236648f295a6715fd27166f7 [id=a076dbaaf3e1492ab94cc6e4dc9476d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.894539 27221 tablet_service.cc:1468] Processing CreateTablet for tablet 0d36f01cb01141d1a6f371ed88d5c584 (DEFAULT_TABLE table=default.Table_949650a78270483689230d33cb7c4a3b [id=561ca09258ca4da5b836254126753ea5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.894639 27218 tablet_service.cc:1468] Processing CreateTablet for tablet c6bf6f0182f346bdb92720f8506e9a13 (DEFAULT_TABLE table=default.Table_d99e1237827942879902dd30ea22ba38 [id=1a9347c03d734ccab96c8f864691b922]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.894850 27212 tablet_service.cc:1468] Processing CreateTablet for tablet 5029c403fbc34dc0aa535d21018e170f (DEFAULT_TABLE table=default.Table_ec7b30148c954f9e8eff70562aed4edb [id=9d92dbfedf09485b8ad8656ed0c30d40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.903302 27352 tablet_service.cc:1468] Processing CreateTablet for tablet 9dd03e34d4d74dc791a08d441c19d8bb (DEFAULT_TABLE table=default.Table_4e126e6eaed846d186337d5a589023b6 [id=2af230da612c4c30a2701b0affdf2a00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.903733 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dd03e34d4d74dc791a08d441c19d8bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.905225 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 42e618f4f375464a87eb7fb9e4265fb6 (DEFAULT_TABLE table=default.Table_0da8589e73bd4ad693cb4d5695b755c7 [id=04d67506921a44dab3638c5e00363291]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.905633 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42e618f4f375464a87eb7fb9e4265fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.894927 27215 tablet_service.cc:1468] Processing CreateTablet for tablet b6ccfe45ba01444fa42ededc6381f225 (DEFAULT_TABLE table=default.Table_1b1798fd0780489a935aa84d7ea2995a [id=c35287cdb05f432399fb8aebc9a62fc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.909674 29196 consensus_queue.cc:1035] T 793d6157927143d4b9d8dbafd7cf0fcb P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:36.897037 27222 tablet_service.cc:1468] Processing CreateTablet for tablet 789f930df8804d73a30bb7e91aa0f95b (DEFAULT_TABLE table=default.Table_1c303bbec4ab48e48656cc3a86cffea0 [id=7d64e9ce87bc4e298905ea66d4e3ee20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.911062 27359 tablet_service.cc:1468] Processing CreateTablet for tablet 7006b7ee1926434cb8148b454c6f422d (DEFAULT_TABLE table=default.Table_a4a91ad2f79c465cadf33585679b2708 [id=489fa4ba6fab4b05a6c86f66a3e3cacd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.911276 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_716897f657da43e5986ff43425cedd20" } modify_external_catalogs: true
I20250903 20:05:36.911478 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_716897f657da43e5986ff43425cedd20" } modify_external_catalogs: true
I20250903 20:05:36.912160 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7006b7ee1926434cb8148b454c6f422d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.914103 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5d1a53d4800a4927b3de164630cc0b7a" } modify_external_catalogs: true
I20250903 20:05:36.914299 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5d1a53d4800a4927b3de164630cc0b7a" } modify_external_catalogs: true
I20250903 20:05:36.914508 27488 tablet_service.cc:1468] Processing CreateTablet for tablet 42e618f4f375464a87eb7fb9e4265fb6 (DEFAULT_TABLE table=default.Table_0da8589e73bd4ad693cb4d5695b755c7 [id=04d67506921a44dab3638c5e00363291]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.914906 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42e618f4f375464a87eb7fb9e4265fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.915637 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_9cac2ed28d064dfebdc8acfc692b00d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:36.916047 27693 consensus_queue.cc:260] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:36.916361 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9cac2ed28d064dfebdc8acfc692b00d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:36.917811 27349 tablet_service.cc:1468] Processing CreateTablet for tablet d2c255209a594cea9312a3e9ede0ac6e (DEFAULT_TABLE table=default.Table_ff9d1c55236648f295a6715fd27166f7 [id=a076dbaaf3e1492ab94cc6e4dc9476d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.921463 29371 raft_consensus.cc:491] T 13496efe34944b7280e7a684f180e6e0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:36.921623 29371 raft_consensus.cc:513] T 13496efe34944b7280e7a684f180e6e0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:36.922842 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13496efe34944b7280e7a684f180e6e0" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:36.922868 27495 tablet_service.cc:1468] Processing CreateTablet for tablet 9dd03e34d4d74dc791a08d441c19d8bb (DEFAULT_TABLE table=default.Table_4e126e6eaed846d186337d5a589023b6 [id=2af230da612c4c30a2701b0affdf2a00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.923030 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13496efe34944b7280e7a684f180e6e0" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:36.923202 29227 raft_consensus.cc:491] T 2385d3006f90486890ebe43e16a53d7a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:36.923286 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dd03e34d4d74dc791a08d441c19d8bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.923341 29227 raft_consensus.cc:513] T 2385d3006f90486890ebe43e16a53d7a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.923440 29371 leader_election.cc:290] T 13496efe34944b7280e7a684f180e6e0 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
W20250903 20:05:36.923509 27462 leader_election.cc:343] T 13496efe34944b7280e7a684f180e6e0 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:36.923684 27462 leader_election.cc:343] T 13496efe34944b7280e7a684f180e6e0 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:36.923758 27462 leader_election.cc:304] T 13496efe34944b7280e7a684f180e6e0 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:36.923938 29227 leader_election.cc:290] T 2385d3006f90486890ebe43e16a53d7a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:36.924255 29371 raft_consensus.cc:2747] T 13496efe34944b7280e7a684f180e6e0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:36.924516 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2385d3006f90486890ebe43e16a53d7a" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:36.925063 27321 leader_election.cc:343] T 2385d3006f90486890ebe43e16a53d7a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:36.926435 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ecfa891cdd9c4b52a981ca48c937bf64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:36.927366 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ecfa891cdd9c4b52a981ca48c937bf64 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:36.927560 27508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2385d3006f90486890ebe43e16a53d7a" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:36.931066 27320 leader_election.cc:343] T 2385d3006f90486890ebe43e16a53d7a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:36.931216 27320 leader_election.cc:304] T 2385d3006f90486890ebe43e16a53d7a P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:36.931723 29227 raft_consensus.cc:2747] T 2385d3006f90486890ebe43e16a53d7a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:36.932134 27693 ts_tablet_manager.cc:1428] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.072s user 0.000s sys 0.002s
I20250903 20:05:36.933024 27693 tablet_bootstrap.cc:492] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:36.933645 27243 raft_consensus.cc:3058] T cb2f50683e904dd39ee6eb128717d38c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:36.938755 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 4978cd831361454d86b23c1a52ab1c3b
I20250903 20:05:36.938925 29221 consensus_queue.cc:1035] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:36.940501 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 04094bb7dd324b53a865980c53d4a17a
I20250903 20:05:36.940838 27496 tablet_service.cc:1468] Processing CreateTablet for tablet ee07d9c4743049d184a3a0f979b04ccd (DEFAULT_TABLE table=default.Table_5bcfed1071c9400ca66b8e9b8b2f26cc [id=d4392f49a2354586a30e403b2f292a97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.941315 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee07d9c4743049d184a3a0f979b04ccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.941422 27497 tablet_service.cc:1468] Processing CreateTablet for tablet 7006b7ee1926434cb8148b454c6f422d (DEFAULT_TABLE table=default.Table_a4a91ad2f79c465cadf33585679b2708 [id=489fa4ba6fab4b05a6c86f66a3e3cacd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.941844 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7006b7ee1926434cb8148b454c6f422d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.942548 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2c255209a594cea9312a3e9ede0ac6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.945086 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 4978cd831361454d86b23c1a52ab1c3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:36.945475 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 2a056877a93342dd9d22ff50606731ca (DEFAULT_TABLE table=default.Table_9beb5c7440aa47549f15ef79b911dfea [id=6a022518eb934096bf49e092a98907ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.945660 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 688cfe6155784b31ad50820bc61c4c4a (DEFAULT_TABLE table=default.Table_1124e9ecb8de4091bb37e8cc347dc8ab [id=fa5aa700613942db982b126b42b87974]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.945851 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a056877a93342dd9d22ff50606731ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.946125 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 688cfe6155784b31ad50820bc61c4c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.946396 27493 tablet_service.cc:1468] Processing CreateTablet for tablet e4315f4fa6cf4572b09550e06f675b10 (DEFAULT_TABLE table=default.Table_9533e6ddb5a84196b359ee85dd13e42c [id=322a3056681f48d7bd428e55227f883a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.946733 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4315f4fa6cf4572b09550e06f675b10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.949591 27693 tablet_bootstrap.cc:654] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:36.949991 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 32164b4e88b94e1fbca2e714e6bddfc4
I20250903 20:05:36.953580 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 04094bb7dd324b53a865980c53d4a17a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:36.953787 27695 tablet_bootstrap.cc:492] T 4e9a6a6e05464b7b97ae624f51402cf0 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:36.953979 27695 ts_tablet_manager.cc:1397] T 4e9a6a6e05464b7b97ae624f51402cf0 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.191s user 0.002s sys 0.002s
I20250903 20:05:36.954635 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 32164b4e88b94e1fbca2e714e6bddfc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:36.954697 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_019bea9491a242768b3d8863a9e0194b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:36.955382 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_019bea9491a242768b3d8863a9e0194b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:36.960311 27498 tablet_service.cc:1468] Processing CreateTablet for tablet 7fda4c6cb0504d89bb94f1dc78ffc59b (DEFAULT_TABLE table=default.Table_312cd54d2d6f4ae5bbcd5fb11ab7700e [id=716994dd8a9f4e0d857f8a88e95048a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.960697 27498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fda4c6cb0504d89bb94f1dc78ffc59b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.961758 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_017a71fa3a044b73bd2ced468094acac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:36.961819 27695 raft_consensus.cc:357] T 4e9a6a6e05464b7b97ae624f51402cf0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.962128 27695 raft_consensus.cc:383] T 4e9a6a6e05464b7b97ae624f51402cf0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:36.962198 27695 raft_consensus.cc:738] T 4e9a6a6e05464b7b97ae624f51402cf0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
W20250903 20:05:36.962288 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_017a71fa3a044b73bd2ced468094acac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:36.964071 27685 ts_tablet_manager.cc:1918] T 5002b7e8118c49b6a247cb8dc4ca1aea P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:36.969056 27066 master_service.cc:948] ReplaceTablet: received request to replace tablet 4a4198b4841b4c2585e6c2f5c6a655d4 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:36.969696 27685 log.cc:1199] T 5002b7e8118c49b6a247cb8dc4ca1aea P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/5002b7e8118c49b6a247cb8dc4ca1aea
I20250903 20:05:36.969709 27489 tablet_service.cc:1468] Processing CreateTablet for tablet 3e4b186af5e34540be249780b52f120d (DEFAULT_TABLE table=default.Table_83abf243847040dc8165106073aaa138 [id=ce072908f8364ed180ff9dbedbc4778e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.970337 27685 ts_tablet_manager.cc:1939] T 5002b7e8118c49b6a247cb8dc4ca1aea P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
W20250903 20:05:36.970026 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4fa886f5be0f489290921e61dcaa4916 consensus state has no leader
I20250903 20:05:36.970640 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_89d23e941ec94830ade8433eb65b5e66" } new_table_name: "default.Table_2906ad3e354d42d1ae223bbb75233d98" modify_external_catalogs: true
I20250903 20:05:36.970507 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 19c012cef4864d97add4fcf7993fbba5 (DEFAULT_TABLE table=default.Table_1b90687e7c414e00ba0897296f8c4ea5 [id=1dea2627e025433b83bc68e84236d7a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:36.973212 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19c012cef4864d97add4fcf7993fbba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.976342 27695 consensus_queue.cc:260] T 4e9a6a6e05464b7b97ae624f51402cf0 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:36.978098 27695 ts_tablet_manager.cc:1428] T 4e9a6a6e05464b7b97ae624f51402cf0 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.024s user 0.003s sys 0.000s
I20250903 20:05:36.980567 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09229109533c49ff9f9824cadb6e2e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.983866 27687 ts_tablet_manager.cc:1918] T 548b50819f694fa3ad508da9f442a99c P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:36.984016 27687 log.cc:1199] T 548b50819f694fa3ad508da9f442a99c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/548b50819f694fa3ad508da9f442a99c
I20250903 20:05:36.984408 27687 ts_tablet_manager.cc:1939] T 548b50819f694fa3ad508da9f442a99c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:36.985769 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9942bb4b868d4a08866298d7790ecad2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:36.989295 27053 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 4fa886f5be0f489290921e61dcaa4916 (table default.Table_d8e0617b6b194aa382092114902f3702 [id=b7afa73399c94853a86ca78575d42726], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4fa886f5be0f489290921e61dcaa4916 consensus state has no leader
I20250903 20:05:36.990835 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0383bfbf13a74bb18b25528340136950. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.992133 27693 tablet_bootstrap.cc:492] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:36.992295 27693 ts_tablet_manager.cc:1397] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.059s user 0.002s sys 0.000s
I20250903 20:05:36.993049 27226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42e618f4f375464a87eb7fb9e4265fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.992995 27693 raft_consensus.cc:357] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:36.993149 27693 raft_consensus.cc:383] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:36.993196 27693 raft_consensus.cc:738] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:36.993433 27693 consensus_queue.cc:260] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:36.996654 27183 consensus_peers.cc:489] T f326ec776ba446be8d5dda5c1dc5b6ec P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:36.997148 27693 ts_tablet_manager.cc:1428] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250903 20:05:36.998462 27229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dd03e34d4d74dc791a08d441c19d8bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:36.998857 27693 tablet_bootstrap.cc:492] T 472f0fb01263468a828ec13795656624 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:37.000365 27693 tablet_bootstrap.cc:654] T 472f0fb01263468a828ec13795656624 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:37.001001 27065 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 7a9e55f0811345f7afa71645f38ebdaf (table default.Table_adf05e59a7be478b89666c3162ac83f1 [id=e69321e60a3a4e71a463a0011c0c6d65]). Expected version 2 got 0
I20250903 20:05:37.001343 27065 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 793d6157927143d4b9d8dbafd7cf0fcb (table default.Table_e79f52615fe142b9823a2a99b8b9c406 [id=3432f6ac076b4e49b9fe63bf5fce8591]). Expected version 1 got 0
I20250903 20:05:37.001595 27065 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet fc634116a78f45dd833034ddda90e9bf (table default.Table_78e7a5b76ebc469396d8a3d670ab63ee [id=fb43441ca8f2415d99a628e81ef1642f]). Expected version 2 got 0
I20250903 20:05:37.002454 27228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee07d9c4743049d184a3a0f979b04ccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.003451 29227 consensus_queue.cc:1035] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:37.006467 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 688cfe6155784b31ad50820bc61c4c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.007078 29196 raft_consensus.cc:491] T 3919f7f95a954f59b68f56d0051c0109 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:37.007256 29196 raft_consensus.cc:513] T 3919f7f95a954f59b68f56d0051c0109 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.007874 29196 leader_election.cc:290] T 3919f7f95a954f59b68f56d0051c0109 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:37.009492 27508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3919f7f95a954f59b68f56d0051c0109" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:37.010046 27183 leader_election.cc:343] T 3919f7f95a954f59b68f56d0051c0109 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:37.010735 27219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7006b7ee1926434cb8148b454c6f422d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.011463 29227 consensus_queue.cc:1035] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:37.014473 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a056877a93342dd9d22ff50606731ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.014665 27216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2c255209a594cea9312a3e9ede0ac6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.018577 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d36f01cb01141d1a6f371ed88d5c584. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.020896 29227 consensus_queue.cc:1035] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:37.021765 29285 raft_consensus.cc:491] T 64c4dd2645034a7eaf243f5474603f10 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:37.021898 29285 raft_consensus.cc:513] T 64c4dd2645034a7eaf243f5474603f10 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.022466 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6bf6f0182f346bdb92720f8506e9a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.022523 29285 leader_election.cc:290] T 64c4dd2645034a7eaf243f5474603f10 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:37.024022 28567 consensus_queue.cc:1035] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:37.024880 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 32164b4e88b94e1fbca2e714e6bddfc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:37.025074 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet 04094bb7dd324b53a865980c53d4a17a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:37.026003 29285 raft_consensus.cc:491] T ee40566e497740789575275d0cdca0db P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:37.026126 29285 raft_consensus.cc:513] T ee40566e497740789575275d0cdca0db P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.026511 27212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5029c403fbc34dc0aa535d21018e170f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.026801 29285 leader_election.cc:290] T ee40566e497740789575275d0cdca0db P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:37.026732 27211 tablet_service.cc:1468] Processing CreateTablet for tablet e4315f4fa6cf4572b09550e06f675b10 (DEFAULT_TABLE table=default.Table_9533e6ddb5a84196b359ee85dd13e42c [id=322a3056681f48d7bd428e55227f883a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.027204 27211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4315f4fa6cf4572b09550e06f675b10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.027253 27220 tablet_service.cc:1468] Processing CreateTablet for tablet 7fda4c6cb0504d89bb94f1dc78ffc59b (DEFAULT_TABLE table=default.Table_312cd54d2d6f4ae5bbcd5fb11ab7700e [id=716994dd8a9f4e0d857f8a88e95048a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.027631 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fda4c6cb0504d89bb94f1dc78ffc59b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.030601 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 4978cd831361454d86b23c1a52ab1c3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:37.031100 27348 tablet_service.cc:1468] Processing CreateTablet for tablet 3e4b186af5e34540be249780b52f120d (DEFAULT_TABLE table=default.Table_83abf243847040dc8165106073aaa138 [id=ce072908f8364ed180ff9dbedbc4778e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.031592 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e4b186af5e34540be249780b52f120d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.033291 27213 tablet_service.cc:1515] Processing DeleteTablet for tablet 32164b4e88b94e1fbca2e714e6bddfc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:37.033512 27227 tablet_service.cc:1515] Processing DeleteTablet for tablet 04094bb7dd324b53a865980c53d4a17a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:37.034416 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3919f7f95a954f59b68f56d0051c0109" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:37.034514 27350 tablet_service.cc:1468] Processing CreateTablet for tablet 19c012cef4864d97add4fcf7993fbba5 (DEFAULT_TABLE table=default.Table_1b90687e7c414e00ba0897296f8c4ea5 [id=1dea2627e025433b83bc68e84236d7a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.034775 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "64c4dd2645034a7eaf243f5474603f10" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:37.034888 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19c012cef4864d97add4fcf7993fbba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.039966 27347 tablet_service.cc:1468] Processing CreateTablet for tablet 7fda4c6cb0504d89bb94f1dc78ffc59b (DEFAULT_TABLE table=default.Table_312cd54d2d6f4ae5bbcd5fb11ab7700e [id=716994dd8a9f4e0d857f8a88e95048a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.040441 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fda4c6cb0504d89bb94f1dc78ffc59b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.045882 27509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "64c4dd2645034a7eaf243f5474603f10" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:37.046137 27509 raft_consensus.cc:2466] T 64c4dd2645034a7eaf243f5474603f10 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:37.046339 27517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ee40566e497740789575275d0cdca0db" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:37.046522 27517 raft_consensus.cc:2466] T ee40566e497740789575275d0cdca0db P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:37.049261 29199 consensus_queue.cc:1035] T 4e34d6d5d68e4395aed78821ea077c1b P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:37.051371 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 5bff3d77088c4b69a90365950641e300
I20250903 20:05:37.053107 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet cc71346bf84f4a708bf17b5aba174582
I20250903 20:05:37.060621 27489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e4b186af5e34540be249780b52f120d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.060631 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 997) took 4228 ms. Trace:
I20250903 20:05:37.061002 27695 tablet_bootstrap.cc:492] T 30cc0d4ba8d444f1b98232f906ea0c7b P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:37.065373 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_d9f8debc2d944aa295651daee0dcb326" } new_table_name: "default.Table_7567eeb166a44c7ab14d8bb83c7a6809" modify_external_catalogs: true
I20250903 20:05:37.066020 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_357d7af9ec6f493582c7c7c5a3b28414"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:37.066604 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_357d7af9ec6f493582c7c7c5a3b28414 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:37.068534 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_12c3415978494757b57a3c08462fa742"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:37.069038 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12c3415978494757b57a3c08462fa742 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:37.070662 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_a204d49d7f1e4eeabb2f685a5ee69a40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:37.071159 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a204d49d7f1e4eeabb2f685a5ee69a40 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:37.072468 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_1828d17a24374724b13ff20d37532044"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:37.072942 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1828d17a24374724b13ff20d37532044 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:37.074877 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_762b9caaaec44868b1816efca7d186cf" } new_table_name: "default.Table_c34fba2b5127426c8d3c2cd426ca7da5" modify_external_catalogs: true
I20250903 20:05:37.076743 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 4a4198b4841b4c2585e6c2f5c6a655d4
I20250903 20:05:37.077481 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab380b2765b142c890c6010fe8d00948. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.077870 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6ccfe45ba01444fa42ededc6381f225. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.078447 27066 catalog_manager.cc:6652] ReplaceTablet: tablet 4a4198b4841b4c2585e6c2f5c6a655d4 deleted and replaced by tablet 0c2ea06c87b243d9a1aecf4a81841dfd
I20250903 20:05:37.079047 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 789f930df8804d73a30bb7e91aa0f95b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.081377 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 5bff3d77088c4b69a90365950641e300 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:37.081565 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet 4978cd831361454d86b23c1a52ab1c3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:37.081830 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet cc71346bf84f4a708bf17b5aba174582 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:37.082559 27362 tablet_service.cc:1468] Processing CreateTablet for tablet f963ef50ecf64a0fa336d84783bc9fd9 (DEFAULT_TABLE table=default.Table_2f4fdec99f854b22bc5262d859311c48 [id=73766a120cbe41adb2a9c51fd5697d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.082978 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f963ef50ecf64a0fa336d84783bc9fd9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:37.085048 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0ab15dd783ff49868d6f9707369a5d3a consensus state has no leader
I20250903 20:05:37.085186 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 3e4b186af5e34540be249780b52f120d (DEFAULT_TABLE table=default.Table_83abf243847040dc8165106073aaa138 [id=ce072908f8364ed180ff9dbedbc4778e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.086994 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a4198b4841b4c2585e6c2f5c6a655d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c2ea06c87b243d9a1aecf4a81841dfd) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:37.088231 27353 tablet_service.cc:1468] Processing CreateTablet for tablet 8df79821c6aa43b69d4f4e2ad00272df (DEFAULT_TABLE table=default.Table_badd71a0f177471b805d80d3e5652d50 [id=fe4325e8c68640a2872f44a264e8a5c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.088675 27353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8df79821c6aa43b69d4f4e2ad00272df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.089236 27693 tablet_bootstrap.cc:492] T 472f0fb01263468a828ec13795656624 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:37.089551 27693 ts_tablet_manager.cc:1397] T 472f0fb01263468a828ec13795656624 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.091s user 0.002s sys 0.004s
I20250903 20:05:37.091055 27693 raft_consensus.cc:357] T 472f0fb01263468a828ec13795656624 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.091924 27693 raft_consensus.cc:383] T 472f0fb01263468a828ec13795656624 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:37.092125 27693 raft_consensus.cc:738] T 472f0fb01263468a828ec13795656624 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
W20250903 20:05:37.093024 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4732783236f44467a04fe573653333f9 consensus state has no leader
I20250903 20:05:37.092676 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 9ef10e3c467a4966ad936a384e26acb9 (DEFAULT_TABLE table=default.Table_584007f8f96047bfa933decba8b5d048 [id=2055e46a1fab4ab087ffe4206ea7322c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.094678 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ef10e3c467a4966ad936a384e26acb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.095088 29371 raft_consensus.cc:491] T 6800e0b6269045f29682704fd847d5d1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:37.095274 29371 raft_consensus.cc:513] T 6800e0b6269045f29682704fd847d5d1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:37.095896 29371 leader_election.cc:290] T 6800e0b6269045f29682704fd847d5d1 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:37.095963 27692 tablet_bootstrap.cc:492] T b4b15d77d07c4fa7856d9c198cf15450 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:37.096091 27692 ts_tablet_manager.cc:1397] T b4b15d77d07c4fa7856d9c198cf15450 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.323s user 0.000s sys 0.003s
I20250903 20:05:37.096231 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e4b186af5e34540be249780b52f120d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.096735 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6800e0b6269045f29682704fd847d5d1" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:37.096846 27692 raft_consensus.cc:357] T b4b15d77d07c4fa7856d9c198cf15450 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.097003 27692 raft_consensus.cc:383] T b4b15d77d07c4fa7856d9c198cf15450 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:37.097054 27692 raft_consensus.cc:738] T b4b15d77d07c4fa7856d9c198cf15450 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:37.097275 27692 consensus_queue.cc:260] T b4b15d77d07c4fa7856d9c198cf15450 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:37.097378 27462 leader_election.cc:343] T 6800e0b6269045f29682704fd847d5d1 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:37.097859 27692 ts_tablet_manager.cc:1428] T b4b15d77d07c4fa7856d9c198cf15450 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20250903 20:05:37.098148 27692 tablet_bootstrap.cc:492] T 259bc73ca9df47e9874692725a60bb1c P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:37.099200 27693 consensus_queue.cc:260] T 472f0fb01263468a828ec13795656624 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.099699 27183 leader_election.cc:304] T 64c4dd2645034a7eaf243f5474603f10 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:37.099877 27692 tablet_bootstrap.cc:654] T 259bc73ca9df47e9874692725a60bb1c P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:37.099975 27183 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 43726.8us
I20250903 20:05:37.100083 29199 raft_consensus.cc:2802] T 64c4dd2645034a7eaf243f5474603f10 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:37.100203 29199 raft_consensus.cc:491] T 64c4dd2645034a7eaf243f5474603f10 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:37.100235 27183 leader_election.cc:304] T ee40566e497740789575275d0cdca0db P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:37.100333 29199 raft_consensus.cc:3058] T 64c4dd2645034a7eaf243f5474603f10 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:37.100724 29196 raft_consensus.cc:2802] T ee40566e497740789575275d0cdca0db P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:37.100836 29196 raft_consensus.cc:491] T ee40566e497740789575275d0cdca0db P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:37.100917 29196 raft_consensus.cc:3058] T ee40566e497740789575275d0cdca0db P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:37.102051 29285 consensus_queue.cc:1035] T 793d6157927143d4b9d8dbafd7cf0fcb P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:37.102802 29227 raft_consensus.cc:491] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:37.102670 27685 rpcz_store.cc:276] 0903 20:05:32.832465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.837868 (+ 5403us) service_pool.cc:225] Handling call
0903 20:05:36.696576 (+3858708us) ts_tablet_manager.cc:1145] Deleting tablet 5002b7e8118c49b6a247cb8dc4ca1aea
0903 20:05:36.696600 (+ 24us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.800376 (+103776us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.963924 (+163548us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.060611 (+ 96687us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3858551}
I20250903 20:05:37.102918 29227 raft_consensus.cc:513] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:37.103092 27685 tablet_replica.cc:331] T 02b3e262dd7841dfbb0520377eb94743 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:37.103307 27685 raft_consensus.cc:2241] T 02b3e262dd7841dfbb0520377eb94743 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:37.103475 27685 raft_consensus.cc:2270] T 02b3e262dd7841dfbb0520377eb94743 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:37.103530 29227 leader_election.cc:290] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:37.104107 27685 ts_tablet_manager.cc:1905] T 02b3e262dd7841dfbb0520377eb94743 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:37.104265 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47c1287bff1145a5b2ab6d14dbd608c7" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:37.104445 27511 raft_consensus.cc:2466] T 47c1287bff1145a5b2ab6d14dbd608c7 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:37.104774 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47c1287bff1145a5b2ab6d14dbd608c7" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:37.104955 27243 raft_consensus.cc:2466] T 47c1287bff1145a5b2ab6d14dbd608c7 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:37.105226 27320 leader_election.cc:304] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:37.105607 29227 raft_consensus.cc:2802] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:37.105700 29227 raft_consensus.cc:491] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:37.105775 29227 raft_consensus.cc:3058] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:37.110077 27693 ts_tablet_manager.cc:1428] T 472f0fb01263468a828ec13795656624 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20250903 20:05:37.114465 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1126) took 2408 ms. Trace:
I20250903 20:05:37.114575 27687 rpcz_store.cc:276] 0903 20:05:34.705623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.705675 (+ 52us) service_pool.cc:225] Handling call
0903 20:05:36.794347 (+2088672us) ts_tablet_manager.cc:1145] Deleting tablet 548b50819f694fa3ad508da9f442a99c
0903 20:05:36.794359 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.955028 (+160669us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:36.983748 (+ 28720us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.114441 (+130693us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2088508}
I20250903 20:05:37.114925 27687 tablet_replica.cc:331] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:37.115095 27687 raft_consensus.cc:2241] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:37.115461 27687 raft_consensus.cc:2270] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:37.118468 27687 ts_tablet_manager.cc:1905] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:37.119254 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0aab16e37f6e433aa4834dd6606f78a1 consensus state has no leader
I20250903 20:05:37.085068 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_b677e454454e465f8717a74bb5a4596b" } new_table_name: "default.Table_e25de49e58cc41b0a3e800c9f258361d" modify_external_catalogs: true
I20250903 20:05:37.123248 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 548b50819f694fa3ad508da9f442a99c (table default.Table_b838fda30d4043a5ad578ec527c1d105 [id=1eb6b859ca0049699466cc72e5a2b6f7]) successfully deleted
I20250903 20:05:37.124082 27684 ts_tablet_manager.cc:1918] T f326ec776ba446be8d5dda5c1dc5b6ec P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:37.126505 27352 tablet_service.cc:1468] Processing CreateTablet for tablet d41de82304bf453fab8b15d96cf32a74 (DEFAULT_TABLE table=default.Table_0276baa7ef3b41c994e5d131569d5955 [id=4ff425ad3eeb4abca1887f5127cda1cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.126940 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d41de82304bf453fab8b15d96cf32a74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.135109 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 36932534c9094a1980cf2ecb6074ebd2 (DEFAULT_TABLE table=default.Table_694c90191216407784c7fea5d5379e2d [id=1261af79c2f845c58d6de2bae4b2d8ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.135670 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36932534c9094a1980cf2ecb6074ebd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.135993 27359 tablet_service.cc:1468] Processing CreateTablet for tablet c6d7f237dfce419c83e1d5c9fa12814e (DEFAULT_TABLE table=default.Table_32800b753a384355979ff8944551c12c [id=ac2188e4478c4956bc105df3aa19920d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.136384 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6d7f237dfce419c83e1d5c9fa12814e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:37.136881 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8292ae4c5bdd469681f3a3b295b24349 consensus state has no leader
I20250903 20:05:37.138964 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6800e0b6269045f29682704fd847d5d1" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:37.143366 27684 log.cc:1199] T f326ec776ba446be8d5dda5c1dc5b6ec P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/f326ec776ba446be8d5dda5c1dc5b6ec
I20250903 20:05:37.145550 27692 tablet_bootstrap.cc:492] T 259bc73ca9df47e9874692725a60bb1c P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:37.145830 27692 ts_tablet_manager.cc:1397] T 259bc73ca9df47e9874692725a60bb1c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.048s user 0.000s sys 0.003s
I20250903 20:05:37.146966 27692 raft_consensus.cc:357] T 259bc73ca9df47e9874692725a60bb1c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.147217 27692 raft_consensus.cc:383] T 259bc73ca9df47e9874692725a60bb1c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:37.147336 27692 raft_consensus.cc:738] T 259bc73ca9df47e9874692725a60bb1c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:37.147596 27692 consensus_queue.cc:260] T 259bc73ca9df47e9874692725a60bb1c P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.148213 27692 ts_tablet_manager.cc:1428] T 259bc73ca9df47e9874692725a60bb1c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250903 20:05:37.148577 27692 tablet_bootstrap.cc:492] T 5b46bb34ac214b1ba23fba1a33f746ac P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
W20250903 20:05:37.149292 27462 leader_election.cc:343] T 6800e0b6269045f29682704fd847d5d1 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:37.149420 27462 leader_election.cc:304] T 6800e0b6269045f29682704fd847d5d1 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:37.149832 29371 raft_consensus.cc:2747] T 6800e0b6269045f29682704fd847d5d1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:37.150207 27684 ts_tablet_manager.cc:1939] T f326ec776ba446be8d5dda5c1dc5b6ec P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:37.151010 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1107) took 2461 ms. Trace:
I20250903 20:05:37.151154 27684 rpcz_store.cc:276] 0903 20:05:34.689420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.689569 (+ 149us) service_pool.cc:225] Handling call
0903 20:05:36.803114 (+2113545us) ts_tablet_manager.cc:1145] Deleting tablet f326ec776ba446be8d5dda5c1dc5b6ec
0903 20:05:36.803124 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:36.816120 (+ 12996us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.123976 (+307856us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.150992 (+ 27016us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2113404}
I20250903 20:05:37.151233 27350 tablet_service.cc:1468] Processing CreateTablet for tablet 740b7d1b3c5545c78aff4629858ac7f1 (DEFAULT_TABLE table=default.Table_d73ecd1c981c4cfdbebd10521a2b86eb [id=562751cb22694d6caa06088c3f7490c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.151340 29227 raft_consensus.cc:513] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:37.151603 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 740b7d1b3c5545c78aff4629858ac7f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.151628 27684 tablet_replica.cc:331] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:37.151876 27684 raft_consensus.cc:2241] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:37.151927 29227 leader_election.cc:290] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:37.152256 27684 raft_consensus.cc:2270] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:37.152498 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47c1287bff1145a5b2ab6d14dbd608c7" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:37.152662 27511 raft_consensus.cc:3058] T 47c1287bff1145a5b2ab6d14dbd608c7 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:37.152892 27684 ts_tablet_manager.cc:1905] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:37.155375 27511 raft_consensus.cc:2466] T 47c1287bff1145a5b2ab6d14dbd608c7 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:37.156459 27320 leader_election.cc:304] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:37.157335 27349 tablet_service.cc:1468] Processing CreateTablet for tablet ce87a2b6f3a44d3c8ae3b39d158ddb33 (DEFAULT_TABLE table=default.Table_6d5082ee3fd94074a33157509c0b26c5 [id=9da1236f96c24c029aae89acadf41154]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.157765 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce87a2b6f3a44d3c8ae3b39d158ddb33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.162462 29371 raft_consensus.cc:491] T 259bc73ca9df47e9874692725a60bb1c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:37.162572 27692 tablet_bootstrap.cc:654] T 5b46bb34ac214b1ba23fba1a33f746ac P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:37.162616 29371 raft_consensus.cc:513] T 259bc73ca9df47e9874692725a60bb1c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.163208 29371 leader_election.cc:290] T 259bc73ca9df47e9874692725a60bb1c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:37.164070 27357 tablet_service.cc:1468] Processing CreateTablet for tablet 25ce665c6ee3451aae2b8281131435ae (DEFAULT_TABLE table=default.Table_8c7c15fd52a0453ca91d7fef1fed6adb [id=f77c17022eae4823a94a80694f054b47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.164462 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25ce665c6ee3451aae2b8281131435ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.169070 29227 raft_consensus.cc:2802] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:37.169297 29227 raft_consensus.cc:695] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:37.169660 29227 consensus_queue.cc:237] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:37.176405 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "259bc73ca9df47e9874692725a60bb1c" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:37.176896 27489 tablet_service.cc:1515] Processing DeleteTablet for tablet 5bff3d77088c4b69a90365950641e300 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:37.177357 27489 tablet_service.cc:1515] Processing DeleteTablet for tablet cc71346bf84f4a708bf17b5aba174582 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:37.177343 27462 leader_election.cc:343] T 259bc73ca9df47e9874692725a60bb1c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:37.185786 29329 raft_consensus.cc:491] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:37.186103 29329 raft_consensus.cc:513] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.187150 29329 leader_election.cc:290] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:37.187973 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ee40566e497740789575275d0cdca0db" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:37.188263 27511 raft_consensus.cc:2466] T ee40566e497740789575275d0cdca0db P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:37.188706 27320 leader_election.cc:304] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:37.189062 29329 raft_consensus.cc:2802] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:37.189177 29329 raft_consensus.cc:491] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:37.189292 29329 raft_consensus.cc:3058] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:37.190768 27684 ts_tablet_manager.cc:1918] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:37.192377 29329 raft_consensus.cc:513] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.194300 27684 log.cc:1199] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/886b8783aa9d4618aad498d4a1115843
I20250903 20:05:37.194738 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "259bc73ca9df47e9874692725a60bb1c" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:37.194813 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47c1287bff1145a5b2ab6d14dbd608c7" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:37.195106 27243 raft_consensus.cc:3058] T 47c1287bff1145a5b2ab6d14dbd608c7 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:37.195462 27462 leader_election.cc:343] T 259bc73ca9df47e9874692725a60bb1c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:37.195601 27462 leader_election.cc:304] T 259bc73ca9df47e9874692725a60bb1c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:37.195989 29371 raft_consensus.cc:2747] T 259bc73ca9df47e9874692725a60bb1c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:37.196298 27684 ts_tablet_manager.cc:1939] T 886b8783aa9d4618aad498d4a1115843 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:37.197506 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1108) took 2508 ms. Trace:
I20250903 20:05:37.201843 27684 rpcz_store.cc:276] 0903 20:05:34.689510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.689710 (+ 200us) service_pool.cc:225] Handling call
0903 20:05:37.151586 (+2461876us) ts_tablet_manager.cc:1145] Deleting tablet 886b8783aa9d4618aad498d4a1115843
0903 20:05:37.151594 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:37.181335 (+ 29741us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.190627 (+ 9292us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.197479 (+ 6852us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2461733}
I20250903 20:05:37.203197 27684 tablet_replica.cc:331] T 238f2f30377b42c2852b016018aa3245 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:37.204324 27684 raft_consensus.cc:2241] T 238f2f30377b42c2852b016018aa3245 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:37.204811 27684 raft_consensus.cc:2270] T 238f2f30377b42c2852b016018aa3245 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:37.205752 27684 ts_tablet_manager.cc:1905] T 238f2f30377b42c2852b016018aa3245 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:37.220793 27695 tablet_bootstrap.cc:654] T 30cc0d4ba8d444f1b98232f906ea0c7b P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:37.219308 27184 leader_election.cc:343] T 3919f7f95a954f59b68f56d0051c0109 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:37.230556 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ee40566e497740789575275d0cdca0db" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:37.231628 27684 ts_tablet_manager.cc:1918] T 238f2f30377b42c2852b016018aa3245 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:37.231899 27184 leader_election.cc:304] T 3919f7f95a954f59b68f56d0051c0109 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:37.232203 27684 log.cc:1199] T 238f2f30377b42c2852b016018aa3245 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/238f2f30377b42c2852b016018aa3245
I20250903 20:05:37.232831 27684 ts_tablet_manager.cc:1939] T 238f2f30377b42c2852b016018aa3245 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:37.193050 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 2bf6c73765534237ba3f1ccb751192b2 (DEFAULT_TABLE table=default.Table_e02c690220fb4ab2a949d23f2230eeca [id=9ffa3b31524942c4b07a8e11956d02fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.203987 27359 tablet_service.cc:1468] Processing CreateTablet for tablet bf35e907252244118b8ee9e99e0f0415 (DEFAULT_TABLE table=default.Table_1828d17a24374724b13ff20d37532044 [id=4bb622e4d8174344b74bcfa8ba0e8f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.205376 27362 tablet_service.cc:1468] Processing CreateTablet for tablet 7b801bf27b5a4e66812d127d1c5a47e3 (DEFAULT_TABLE table=default.Table_a204d49d7f1e4eeabb2f685a5ee69a40 [id=dd3c6590d6fe4fe78fafd7c236e8a811]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.206945 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 98d89282fc684dceb735e58ded9ea000 (DEFAULT_TABLE table=default.Table_12c3415978494757b57a3c08462fa742 [id=9963800b602f4c5c934854b044d6bd0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.209338 27350 tablet_service.cc:1468] Processing CreateTablet for tablet 071f2a97aed640ddad8119ec4fc452e5 (DEFAULT_TABLE table=default.Table_357d7af9ec6f493582c7c7c5a3b28414 [id=25a9593fadf44b65bcb534989f669fa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.210891 27348 tablet_service.cc:1468] Processing CreateTablet for tablet 0a5e2cc1df3d46b9b34feaa1fdfb7ed7 (DEFAULT_TABLE table=default.Table_019bea9491a242768b3d8863a9e0194b [id=df13920c040e49118af46fd13d812bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.211649 27352 tablet_service.cc:1468] Processing CreateTablet for tablet 1d5e686dcd754132b9d7ba8ce808bbbe (DEFAULT_TABLE table=default.Table_aeea04c58a4643328ab1223efbed4a4a [id=d18967040fc54e09a7fc8c7eb4997a8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.233954 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bf6c73765534237ba3f1ccb751192b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.234542 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf35e907252244118b8ee9e99e0f0415. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.234858 29196 raft_consensus.cc:513] T ee40566e497740789575275d0cdca0db P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.235669 27239 raft_consensus.cc:2391] T ee40566e497740789575275d0cdca0db P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 51578fb7bceb4e119b3ab1e58c4f5528 in current term 1: Already voted for candidate 89f8f0ee68b940db9ce250c0604f934a in this term.
I20250903 20:05:37.235777 29199 raft_consensus.cc:513] T 64c4dd2645034a7eaf243f5474603f10 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.236287 29199 leader_election.cc:290] T 64c4dd2645034a7eaf243f5474603f10 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:37.236363 27363 tablet_service.cc:1468] Processing CreateTablet for tablet a0890b5be4cd4c8995da37b47e52b820 (DEFAULT_TABLE table=default.Table_017a71fa3a044b73bd2ced468094acac [id=b0e506ed02a44adbb79969b2accf9816]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.236780 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0890b5be4cd4c8995da37b47e52b820. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.236750 29199 raft_consensus.cc:2747] T 3919f7f95a954f59b68f56d0051c0109 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:37.238564 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98d89282fc684dceb735e58ded9ea000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.239634 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 55711d641a664e3996230d7c6a256ec1 (DEFAULT_TABLE table=default.Table_9f971775443a46128184a77aa0233d86 [id=3a23a6f1ed57437bbf971f3bc4393e0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.239986 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55711d641a664e3996230d7c6a256ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.240245 27350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 071f2a97aed640ddad8119ec4fc452e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.240736 27693 tablet_bootstrap.cc:492] T 30cc0d4ba8d444f1b98232f906ea0c7b P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:37.212574 27687 ts_tablet_manager.cc:1918] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:37.213840 27354 tablet_service.cc:1468] Processing CreateTablet for tablet 37071b15494b4c5ea450e34f7b93aafa (DEFAULT_TABLE table=default.Table_96bef96c2061427d8b6643208730463c [id=7a5292efb3fa4041ad3f50fa37a449b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.215605 27358 tablet_service.cc:1468] Processing CreateTablet for tablet 831d2bfa62e2439396fa53576201413d (DEFAULT_TABLE table=default.Table_03367a125ef440d2bc8c3f1531dfb0b9 [id=9e14de0e4aeb473ea230c57fdd29889f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.243005 27687 log.cc:1199] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/1223743918814d88b401782ca4b521dd
I20250903 20:05:37.243467 27687 ts_tablet_manager.cc:1939] T 1223743918814d88b401782ca4b521dd P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
W20250903 20:05:37.123570 27051 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 56008.5us
I20250903 20:05:37.244871 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet f326ec776ba446be8d5dda5c1dc5b6ec (table default.Table_b2a82a2c310140caaeac5d0dcb4bb61a [id=e2548ad469e54a41af7c0181357d2467]) successfully deleted
I20250903 20:05:37.248184 27243 raft_consensus.cc:2466] T 47c1287bff1145a5b2ab6d14dbd608c7 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:37.251446 29196 leader_election.cc:290] T ee40566e497740789575275d0cdca0db P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:37.216627 27351 tablet_service.cc:1468] Processing CreateTablet for tablet 1fd11faff1f54553accb793aa910303d (DEFAULT_TABLE table=default.Table_0f58bdb4d1eb4aa9bb8d36909880410b [id=9b40fbee58ce4d71810c2ab96a6fee11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.218130 27364 tablet_service.cc:1468] Processing CreateTablet for tablet 34eadb8f690d41c0877eeeb00d4784a8 (DEFAULT_TABLE table=default.Table_fa0af1d57fb048e7ab3e84f79c1f674d [id=1ffea37ca73649fcb8d8ea85f1b34c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.219666 27366 tablet_service.cc:1468] Processing CreateTablet for tablet 64af6482653844ea93abb850b86ae2ea (DEFAULT_TABLE table=default.Table_43b6194256e1461cb34be4e7642889f2 [id=8d86a912eeb54b99a494678329a80799]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.230466 27355 tablet_service.cc:1468] Processing CreateTablet for tablet 7f909c505e774b86ad107dc03a592180 (DEFAULT_TABLE table=default.Table_f0fc60c750344a2d96326556708d661e [id=ee453f94b790498f84e3488ce8b0d809]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.212841 27356 tablet_service.cc:1468] Processing CreateTablet for tablet 6b0c7097d19a412bbe5720126621b79d (DEFAULT_TABLE table=default.Table_781c5fddf2e945b5bf4c68483f03dde5 [id=aac5db259297453fa9fe0785d33f8a6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.243602 29329 leader_election.cc:290] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:37.243666 27354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37071b15494b4c5ea450e34f7b93aafa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.260380 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 831d2bfa62e2439396fa53576201413d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.260562 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a5e2cc1df3d46b9b34feaa1fdfb7ed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.260738 27351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fd11faff1f54553accb793aa910303d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.260907 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b801bf27b5a4e66812d127d1c5a47e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.261117 27356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b0c7097d19a412bbe5720126621b79d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.263504 27685 ts_tablet_manager.cc:1918] T 02b3e262dd7841dfbb0520377eb94743 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:37.263625 27685 log.cc:1199] T 02b3e262dd7841dfbb0520377eb94743 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/02b3e262dd7841dfbb0520377eb94743
I20250903 20:05:37.263996 27685 ts_tablet_manager.cc:1939] T 02b3e262dd7841dfbb0520377eb94743 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:37.264326 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 998) took 4426 ms. Trace:
I20250903 20:05:37.264400 27685 rpcz_store.cc:276] 0903 20:05:32.838190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.838256 (+ 66us) service_pool.cc:225] Handling call
0903 20:05:37.103014 (+4264758us) ts_tablet_manager.cc:1145] Deleting tablet 02b3e262dd7841dfbb0520377eb94743
0903 20:05:37.103027 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:37.213711 (+110684us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.263361 (+ 49650us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.264310 (+ 949us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4264605}
I20250903 20:05:37.264756 27685 tablet_replica.cc:331] T a0f79831866b40c49c5b27ce2014c3f6 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:37.264986 27685 raft_consensus.cc:2241] T a0f79831866b40c49c5b27ce2014c3f6 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:37.265174 27685 raft_consensus.cc:2270] T a0f79831866b40c49c5b27ce2014c3f6 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:37.265743 27685 ts_tablet_manager.cc:1905] T a0f79831866b40c49c5b27ce2014c3f6 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:37.265795 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d5e686dcd754132b9d7ba8ce808bbbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.267832 27355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f909c505e774b86ad107dc03a592180. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.270153 27348 tablet_service.cc:1468] Processing CreateTablet for tablet 0c2ea06c87b243d9a1aecf4a81841dfd (DEFAULT_TABLE table=default.Table_36f5d67acb72437493d261988d5ccb28 [id=8c216327759c4647b9cabb512e8cba6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.270530 27366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64af6482653844ea93abb850b86ae2ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.271555 27363 tablet_service.cc:1468] Processing CreateTablet for tablet 701a652bc9d74992aabd6603cc93d0be (DEFAULT_TABLE table=default.Table_351c22f0442944f889808350e50d2887 [id=7e80b43d1ad04c998e1e47257cc44a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.271936 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 701a652bc9d74992aabd6603cc93d0be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.215566 27353 tablet_service.cc:1468] Processing CreateTablet for tablet a655abd16c9c4f50b86fb9c3d12b3805 (DEFAULT_TABLE table=default.Table_ecfa891cdd9c4b52a981ca48c937bf64 [id=b991811dc93547cdab83ffaeda7c51b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.273033 27353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a655abd16c9c4f50b86fb9c3d12b3805. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.274356 27364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34eadb8f690d41c0877eeeb00d4784a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.276559 29371 raft_consensus.cc:491] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:37.276715 29371 raft_consensus.cc:513] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.277343 29371 leader_election.cc:290] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:37.278630 27348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c2ea06c87b243d9a1aecf4a81841dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.283368 27359 tablet_service.cc:1468] Processing CreateTablet for tablet a307376a81904146ae8eeb39f11d9235 (DEFAULT_TABLE table=default.Table_9cac2ed28d064dfebdc8acfc692b00d9 [id=76cde1597b644ba28f701714955dfdae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.283737 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a307376a81904146ae8eeb39f11d9235. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.287722 27352 tablet_service.cc:1468] Processing CreateTablet for tablet f1843a04837d4b76a80a38660ea8cbc5 (DEFAULT_TABLE table=default.Table_fdf6ca8eecbb4d1cb0fd5fcd4e67b5a7 [id=19263ce3f63d41c49677bce476201881]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.288285 27352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1843a04837d4b76a80a38660ea8cbc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.291481 27362 tablet_service.cc:1468] Processing CreateTablet for tablet 6d614ed6d2fe4217bc2d289c12554efa (DEFAULT_TABLE table=default.Table_092cc68e6e774d91ba49dc1622f26b0b [id=b8645b0a55534fbb82818616e31b3b96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.257741 27360 tablet_service.cc:1468] Processing CreateTablet for tablet b78645d4e9574ce3b2a78eebb541fbfe (DEFAULT_TABLE table=default.Table_2ae7f1d891744aeca75e2ed6bf43666f [id=af2de94e42d44e389fac2030cde1f623]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.292032 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d614ed6d2fe4217bc2d289c12554efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.292577 27363 tablet_service.cc:1468] Processing CreateTablet for tablet ccb671d7e67444ac8e5b78055271d02a (DEFAULT_TABLE table=default.Table_d0aec832c86d4e868563056c6100c6b8 [id=8d69d1671a82411da646bc38ae0a5109]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.292933 27363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccb671d7e67444ac8e5b78055271d02a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.296175 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b78645d4e9574ce3b2a78eebb541fbfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.296583 27358 tablet_service.cc:1468] Processing CreateTablet for tablet 658ee69734254d33af0dce3b4407a41f (DEFAULT_TABLE table=default.Table_37b6f939fca548ada1f45f908c602ca9 [id=e43eca3152194fd5a1ce3311b4e55b0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.297061 27358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 658ee69734254d33af0dce3b4407a41f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:37.299173 27183 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 198569us
I20250903 20:05:37.301267 29196 raft_consensus.cc:491] T 4e9a6a6e05464b7b97ae624f51402cf0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:37.301443 29196 raft_consensus.cc:513] T 4e9a6a6e05464b7b97ae624f51402cf0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.302068 29196 leader_election.cc:290] T 4e9a6a6e05464b7b97ae624f51402cf0 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:37.319546 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ee40566e497740789575275d0cdca0db" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:37.319844 27511 raft_consensus.cc:3058] T ee40566e497740789575275d0cdca0db P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:37.319934 27509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ee40566e497740789575275d0cdca0db" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:37.320896 27513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e9a6a6e05464b7b97ae624f51402cf0" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:37.321908 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fef9aa7153a4902abce522e3ff0df3f" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:37.323052 27462 leader_election.cc:343] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:37.323424 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fef9aa7153a4902abce522e3ff0df3f" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:37.324030 27462 leader_election.cc:343] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:37.324195 27462 leader_election.cc:304] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:37.324497 29371 raft_consensus.cc:2747] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:37.325073 27508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "64c4dd2645034a7eaf243f5474603f10" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:37.325237 27508 raft_consensus.cc:3058] T 64c4dd2645034a7eaf243f5474603f10 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:37.326593 26890 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 81208.3us
I20250903 20:05:37.326807 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1109) took 2637 ms. Trace:
I20250903 20:05:37.326912 27684 rpcz_store.cc:276] 0903 20:05:34.689626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.689731 (+ 105us) service_pool.cc:225] Handling call
0903 20:05:37.203123 (+2513392us) ts_tablet_manager.cc:1145] Deleting tablet 238f2f30377b42c2852b016018aa3245
0903 20:05:37.203139 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:37.218332 (+ 15193us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.231495 (+ 13163us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.326782 (+ 95287us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2513241}
I20250903 20:05:37.327262 27684 tablet_replica.cc:331] T 09fa32312c9648fbbd6fd327e9738066 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:37.327633 27684 raft_consensus.cc:2241] T 09fa32312c9648fbbd6fd327e9738066 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:37.327803 27684 raft_consensus.cc:2270] T 09fa32312c9648fbbd6fd327e9738066 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:37.328577 27684 ts_tablet_manager.cc:1905] T 09fa32312c9648fbbd6fd327e9738066 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:37.330596 27357 tablet_service.cc:1468] Processing CreateTablet for tablet 1ed519e2e9c94fa3a134dcfc8d5364e6 (DEFAULT_TABLE table=default.Table_48a0a980c12f40128907043e0ad8a7a8 [id=6c8ffb4fd82644f6a75a57bb196c6d6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.331748 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ed519e2e9c94fa3a134dcfc8d5364e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.332082 27508 raft_consensus.cc:2466] T 64c4dd2645034a7eaf243f5474603f10 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:37.331027 27349 tablet_service.cc:1468] Processing CreateTablet for tablet 1c75bfd1a2f1489393fed76d6e2eb31c (DEFAULT_TABLE table=default.Table_a20d16f1ce384a83998a23435eb76dff [id=a4cb747dbb9c495e85cc2497b4f0397f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.332235 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1127) took 2626 ms. Trace:
I20250903 20:05:37.333274 27687 rpcz_store.cc:276] 0903 20:05:34.706109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.706159 (+ 50us) service_pool.cc:225] Handling call
0903 20:05:37.114845 (+2408686us) ts_tablet_manager.cc:1145] Deleting tablet 1223743918814d88b401782ca4b521dd
0903 20:05:37.114857 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:37.121723 (+ 6866us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.212411 (+ 90688us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.332195 (+119784us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2408458}
I20250903 20:05:37.334146 27511 raft_consensus.cc:2466] T ee40566e497740789575275d0cdca0db P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:37.335367 27692 tablet_bootstrap.cc:492] T 5b46bb34ac214b1ba23fba1a33f746ac P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:37.335580 27692 ts_tablet_manager.cc:1397] T 5b46bb34ac214b1ba23fba1a33f746ac P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.187s user 0.008s sys 0.012s
I20250903 20:05:37.334755 27383 raft_consensus.cc:3058] T 472f0fb01263468a828ec13795656624 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:37.336437 27692 raft_consensus.cc:357] T 5b46bb34ac214b1ba23fba1a33f746ac P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:37.336616 27692 raft_consensus.cc:383] T 5b46bb34ac214b1ba23fba1a33f746ac P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:37.336663 27692 raft_consensus.cc:738] T 5b46bb34ac214b1ba23fba1a33f746ac P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:37.336856 27692 consensus_queue.cc:260] T 5b46bb34ac214b1ba23fba1a33f746ac P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:37.337312 27692 ts_tablet_manager.cc:1428] T 5b46bb34ac214b1ba23fba1a33f746ac P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:37.337615 27692 tablet_bootstrap.cc:492] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:37.340992 27695 tablet_bootstrap.cc:492] T 30cc0d4ba8d444f1b98232f906ea0c7b P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:37.341180 27695 ts_tablet_manager.cc:1397] T 30cc0d4ba8d444f1b98232f906ea0c7b P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.280s user 0.011s sys 0.006s
I20250903 20:05:37.341953 27695 raft_consensus.cc:357] T 30cc0d4ba8d444f1b98232f906ea0c7b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.342180 27695 raft_consensus.cc:383] T 30cc0d4ba8d444f1b98232f906ea0c7b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:37.342299 27695 raft_consensus.cc:738] T 30cc0d4ba8d444f1b98232f906ea0c7b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:37.342550 27695 consensus_queue.cc:260] T 30cc0d4ba8d444f1b98232f906ea0c7b P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.343129 27695 ts_tablet_manager.cc:1428] T 30cc0d4ba8d444f1b98232f906ea0c7b P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:37.334789 27320 leader_election.cc:304] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:37.343411 27695 tablet_bootstrap.cc:492] T d4c703dafbca4e669dbb18946ffac699 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:37.345036 27695 tablet_bootstrap.cc:654] T d4c703dafbca4e669dbb18946ffac699 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:37.345678 29405 raft_consensus.cc:2802] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:37.345863 29405 raft_consensus.cc:695] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:37.346331 29405 consensus_queue.cc:237] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.348024 27692 tablet_bootstrap.cc:654] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:37.348156 27695 tablet_bootstrap.cc:492] T d4c703dafbca4e669dbb18946ffac699 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:37.348373 27695 ts_tablet_manager.cc:1397] T d4c703dafbca4e669dbb18946ffac699 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20250903 20:05:37.349082 27695 raft_consensus.cc:357] T d4c703dafbca4e669dbb18946ffac699 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.349380 27695 raft_consensus.cc:383] T d4c703dafbca4e669dbb18946ffac699 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:37.349453 27695 raft_consensus.cc:738] T d4c703dafbca4e669dbb18946ffac699 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:37.349650 27695 consensus_queue.cc:260] T d4c703dafbca4e669dbb18946ffac699 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.350394 27695 ts_tablet_manager.cc:1428] T d4c703dafbca4e669dbb18946ffac699 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:37.350678 27695 tablet_bootstrap.cc:492] T 259bc73ca9df47e9874692725a60bb1c P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:37.352293 27695 tablet_bootstrap.cc:654] T 259bc73ca9df47e9874692725a60bb1c P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:37.258561 27347 tablet_service.cc:1468] Processing CreateTablet for tablet 3fd1cfb6e3264e9db2a1c75d468d6186 (DEFAULT_TABLE table=default.Table_e57f2570b5f3401cbadd48277ed8993e [id=a9fe04b7569c441c93c00b922cab4a17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.356056 29199 consensus_queue.cc:1035] T 4e34d6d5d68e4395aed78821ea077c1b P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250903 20:05:37.357161 27183 leader_election.cc:343] T 4e9a6a6e05464b7b97ae624f51402cf0 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:37.357488 27183 leader_election.cc:304] T 64c4dd2645034a7eaf243f5474603f10 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:37.357791 29199 raft_consensus.cc:2802] T 64c4dd2645034a7eaf243f5474603f10 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:37.357911 29199 raft_consensus.cc:695] T 64c4dd2645034a7eaf243f5474603f10 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:37.358196 29199 consensus_queue.cc:237] T 64c4dd2645034a7eaf243f5474603f10 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.358736 29227 consensus_queue.cc:1035] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:37.359552 27065 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 793d6157927143d4b9d8dbafd7cf0fcb (table default.Table_e79f52615fe142b9823a2a99b8b9c406 [id=3432f6ac076b4e49b9fe63bf5fce8591]). Expected version 1 got 0
I20250903 20:05:37.335564 27693 tablet_bootstrap.cc:654] T 30cc0d4ba8d444f1b98232f906ea0c7b P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:37.370839 27684 ts_tablet_manager.cc:1918] T 09fa32312c9648fbbd6fd327e9738066 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:37.376622 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 1223743918814d88b401782ca4b521dd (table default.Table_f038423efb3f4fd5bba9911713436b97 [id=22a7d6c72c2e4dbfadd44a229bb175e3]) successfully deleted
I20250903 20:05:37.377111 27687 tablet_replica.cc:331] T 886b8783aa9d4618aad498d4a1115843 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:37.377374 27687 raft_consensus.cc:2241] T 886b8783aa9d4618aad498d4a1115843 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:37.377616 27687 raft_consensus.cc:2270] T 886b8783aa9d4618aad498d4a1115843 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:37.377691 27184 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 327709us
W20250903 20:05:37.377940 27184 leader_election.cc:343] T 64c4dd2645034a7eaf243f5474603f10 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:37.378378 27684 log.cc:1199] T 09fa32312c9648fbbd6fd327e9738066 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/09fa32312c9648fbbd6fd327e9738066
I20250903 20:05:37.379000 27684 ts_tablet_manager.cc:1939] T 09fa32312c9648fbbd6fd327e9738066 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:37.380053 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1112) took 2638 ms. Trace:
I20250903 20:05:37.380250 27684 rpcz_store.cc:276] 0903 20:05:34.742018 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.742092 (+ 74us) service_pool.cc:225] Handling call
0903 20:05:37.327213 (+2585121us) ts_tablet_manager.cc:1145] Deleting tablet 09fa32312c9648fbbd6fd327e9738066
0903 20:05:37.327223 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:37.332709 (+ 5486us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.370687 (+ 37978us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.380026 (+ 9339us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2584730}
I20250903 20:05:37.380702 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ee40566e497740789575275d0cdca0db" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:37.380963 27684 tablet_replica.cc:331] T a3103e1595ec4c38ba65dca26fe1167c P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:37.381011 27240 raft_consensus.cc:2391] T ee40566e497740789575275d0cdca0db P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 51578fb7bceb4e119b3ab1e58c4f5528 in current term 1: Already voted for candidate 89f8f0ee68b940db9ce250c0604f934a in this term.
W20250903 20:05:37.333694 27051 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 88819.8us
I20250903 20:05:37.383502 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 886b8783aa9d4618aad498d4a1115843 (table default.Table_a540eea45dfb46ec92b02a3f9a2b3aed [id=5fe055478b724744a65fcaa2f25bef0d]) successfully deleted
I20250903 20:05:37.383790 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 238f2f30377b42c2852b016018aa3245 (table default.Table_a22a6e16b90a461ba0c890fb7c5791e9 [id=8fe9f7413b124e39bd3c5645312415e0]) successfully deleted
I20250903 20:05:37.388164 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd1cfb6e3264e9db2a1c75d468d6186. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.388885 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c75bfd1a2f1489393fed76d6e2eb31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.390731 27684 raft_consensus.cc:2241] T a3103e1595ec4c38ba65dca26fe1167c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:37.391608 27684 pending_rounds.cc:70] T a3103e1595ec4c38ba65dca26fe1167c P e72ef1dbcc5143cf9f04a808f101c80f: Trying to abort 1 pending ops.
I20250903 20:05:37.391716 27684 pending_rounds.cc:77] T a3103e1595ec4c38ba65dca26fe1167c P e72ef1dbcc5143cf9f04a808f101c80f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196385019402506240 op_type: NO_OP noop_request { }
I20250903 20:05:37.391893 27684 raft_consensus.cc:2887] T a3103e1595ec4c38ba65dca26fe1167c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:37.392062 27684 raft_consensus.cc:2270] T a3103e1595ec4c38ba65dca26fe1167c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:37.393014 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ee40566e497740789575275d0cdca0db" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:37.384018 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 09fa32312c9648fbbd6fd327e9738066 (table default.Table_61ee151cc2184644b05a60d8feb9e3c4 [id=f459194547904acd99011de4c21c4e33]) successfully deleted
I20250903 20:05:37.406538 27489 tablet_service.cc:1468] Processing CreateTablet for tablet f963ef50ecf64a0fa336d84783bc9fd9 (DEFAULT_TABLE table=default.Table_2f4fdec99f854b22bc5262d859311c48 [id=73766a120cbe41adb2a9c51fd5697d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.406993 27684 ts_tablet_manager.cc:1905] T a3103e1595ec4c38ba65dca26fe1167c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:37.407047 27489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f963ef50ecf64a0fa336d84783bc9fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.407896 27498 tablet_service.cc:1468] Processing CreateTablet for tablet 9ef10e3c467a4966ad936a384e26acb9 (DEFAULT_TABLE table=default.Table_584007f8f96047bfa933decba8b5d048 [id=2055e46a1fab4ab087ffe4206ea7322c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:37.407946 27320 consensus_peers.cc:489] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:37.408267 27498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ef10e3c467a4966ad936a384e26acb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.408555 27687 ts_tablet_manager.cc:1905] T 886b8783aa9d4618aad498d4a1115843 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:37.410885 27495 tablet_service.cc:1468] Processing CreateTablet for tablet 8df79821c6aa43b69d4f4e2ad00272df (DEFAULT_TABLE table=default.Table_badd71a0f177471b805d80d3e5652d50 [id=fe4325e8c68640a2872f44a264e8a5c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.411269 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8df79821c6aa43b69d4f4e2ad00272df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.415858 27695 tablet_bootstrap.cc:492] T 259bc73ca9df47e9874692725a60bb1c P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:37.416129 27695 ts_tablet_manager.cc:1397] T 259bc73ca9df47e9874692725a60bb1c P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.066s user 0.013s sys 0.020s
I20250903 20:05:37.425166 27684 ts_tablet_manager.cc:1918] T a3103e1595ec4c38ba65dca26fe1167c P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:37.436676 29405 consensus_queue.cc:1035] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:37.439846 27693 tablet_bootstrap.cc:492] T 30cc0d4ba8d444f1b98232f906ea0c7b P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:37.448695 27693 ts_tablet_manager.cc:1397] T 30cc0d4ba8d444f1b98232f906ea0c7b P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.208s user 0.005s sys 0.000s
I20250903 20:05:37.449788 27693 raft_consensus.cc:357] T 30cc0d4ba8d444f1b98232f906ea0c7b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.450153 27693 raft_consensus.cc:383] T 30cc0d4ba8d444f1b98232f906ea0c7b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:37.450317 27693 raft_consensus.cc:738] T 30cc0d4ba8d444f1b98232f906ea0c7b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:37.450670 27693 consensus_queue.cc:260] T 30cc0d4ba8d444f1b98232f906ea0c7b P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.429008 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a4198b4841b4c2585e6c2f5c6a655d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c2ea06c87b243d9a1aecf4a81841dfd) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:37.456463 27495 tablet_service.cc:1468] Processing CreateTablet for tablet 34eadb8f690d41c0877eeeb00d4784a8 (DEFAULT_TABLE table=default.Table_fa0af1d57fb048e7ab3e84f79c1f674d [id=1ffea37ca73649fcb8d8ea85f1b34c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.456940 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34eadb8f690d41c0877eeeb00d4784a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.407974 27692 tablet_bootstrap.cc:492] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:37.441377 29371 raft_consensus.cc:491] T f4d3e1cfdfab4be1a9143b78bf3bc46d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:37.447521 27489 tablet_service.cc:1468] Processing CreateTablet for tablet 3fd1cfb6e3264e9db2a1c75d468d6186 (DEFAULT_TABLE table=default.Table_e57f2570b5f3401cbadd48277ed8993e [id=a9fe04b7569c441c93c00b922cab4a17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.447521 27503 tablet_service.cc:1468] Processing CreateTablet for tablet d41de82304bf453fab8b15d96cf32a74 (DEFAULT_TABLE table=default.Table_0276baa7ef3b41c994e5d131569d5955 [id=4ff425ad3eeb4abca1887f5127cda1cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.447918 27504 tablet_service.cc:1468] Processing CreateTablet for tablet bf35e907252244118b8ee9e99e0f0415 (DEFAULT_TABLE table=default.Table_1828d17a24374724b13ff20d37532044 [id=4bb622e4d8174344b74bcfa8ba0e8f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.465942 29371 raft_consensus.cc:513] T f4d3e1cfdfab4be1a9143b78bf3bc46d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.466315 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf35e907252244118b8ee9e99e0f0415. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.448093 27497 tablet_service.cc:1468] Processing CreateTablet for tablet ce87a2b6f3a44d3c8ae3b39d158ddb33 (DEFAULT_TABLE table=default.Table_6d5082ee3fd94074a33157509c0b26c5 [id=9da1236f96c24c029aae89acadf41154]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.466583 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce87a2b6f3a44d3c8ae3b39d158ddb33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.448383 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 25ce665c6ee3451aae2b8281131435ae (DEFAULT_TABLE table=default.Table_8c7c15fd52a0453ca91d7fef1fed6adb [id=f77c17022eae4823a94a80694f054b47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.466820 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d41de82304bf453fab8b15d96cf32a74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.466848 29371 leader_election.cc:290] T f4d3e1cfdfab4be1a9143b78bf3bc46d P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:37.467041 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25ce665c6ee3451aae2b8281131435ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.468228 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4d3e1cfdfab4be1a9143b78bf3bc46d" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:37.469187 27462 leader_election.cc:343] T f4d3e1cfdfab4be1a9143b78bf3bc46d P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:37.469527 27692 ts_tablet_manager.cc:1397] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.132s user 0.004s sys 0.000s
I20250903 20:05:37.448637 27493 tablet_service.cc:1468] Processing CreateTablet for tablet 2bf6c73765534237ba3f1ccb751192b2 (DEFAULT_TABLE table=default.Table_e02c690220fb4ab2a949d23f2230eeca [id=9ffa3b31524942c4b07a8e11956d02fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.470151 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bf6c73765534237ba3f1ccb751192b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.470567 27692 raft_consensus.cc:357] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.470786 27692 raft_consensus.cc:383] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:37.470847 27692 raft_consensus.cc:738] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:37.471087 27692 consensus_queue.cc:260] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.471822 27692 ts_tablet_manager.cc:1428] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:37.473138 27684 log.cc:1199] T a3103e1595ec4c38ba65dca26fe1167c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/a3103e1595ec4c38ba65dca26fe1167c
I20250903 20:05:37.449189 27501 tablet_service.cc:1468] Processing CreateTablet for tablet 1fd11faff1f54553accb793aa910303d (DEFAULT_TABLE table=default.Table_0f58bdb4d1eb4aa9bb8d36909880410b [id=9b40fbee58ce4d71810c2ab96a6fee11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.447918 27488 tablet_service.cc:1468] Processing CreateTablet for tablet 36932534c9094a1980cf2ecb6074ebd2 (DEFAULT_TABLE table=default.Table_694c90191216407784c7fea5d5379e2d [id=1261af79c2f845c58d6de2bae4b2d8ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.450241 27498 tablet_service.cc:1468] Processing CreateTablet for tablet c6d7f237dfce419c83e1d5c9fa12814e (DEFAULT_TABLE table=default.Table_32800b753a384355979ff8944551c12c [id=ac2188e4478c4956bc105df3aa19920d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.451025 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 740b7d1b3c5545c78aff4629858ac7f1 (DEFAULT_TABLE table=default.Table_d73ecd1c981c4cfdbebd10521a2b86eb [id=562751cb22694d6caa06088c3f7490c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.451701 27500 tablet_service.cc:1468] Processing CreateTablet for tablet 7b801bf27b5a4e66812d127d1c5a47e3 (DEFAULT_TABLE table=default.Table_a204d49d7f1e4eeabb2f685a5ee69a40 [id=dd3c6590d6fe4fe78fafd7c236e8a811]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.452047 27499 tablet_service.cc:1468] Processing CreateTablet for tablet a655abd16c9c4f50b86fb9c3d12b3805 (DEFAULT_TABLE table=default.Table_ecfa891cdd9c4b52a981ca48c937bf64 [id=b991811dc93547cdab83ffaeda7c51b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.452415 27494 tablet_service.cc:1468] Processing CreateTablet for tablet 071f2a97aed640ddad8119ec4fc452e5 (DEFAULT_TABLE table=default.Table_357d7af9ec6f493582c7c7c5a3b28414 [id=25a9593fadf44b65bcb534989f669fa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.452749 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 0a5e2cc1df3d46b9b34feaa1fdfb7ed7 (DEFAULT_TABLE table=default.Table_019bea9491a242768b3d8863a9e0194b [id=df13920c040e49118af46fd13d812bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.453100 27502 tablet_service.cc:1468] Processing CreateTablet for tablet 1d5e686dcd754132b9d7ba8ce808bbbe (DEFAULT_TABLE table=default.Table_aeea04c58a4643328ab1223efbed4a4a [id=d18967040fc54e09a7fc8c7eb4997a8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.453411 27490 tablet_service.cc:1468] Processing CreateTablet for tablet 6b0c7097d19a412bbe5720126621b79d (DEFAULT_TABLE table=default.Table_781c5fddf2e945b5bf4c68483f03dde5 [id=aac5db259297453fa9fe0785d33f8a6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.474047 27489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd1cfb6e3264e9db2a1c75d468d6186. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.454027 27492 tablet_service.cc:1468] Processing CreateTablet for tablet 831d2bfa62e2439396fa53576201413d (DEFAULT_TABLE table=default.Table_03367a125ef440d2bc8c3f1531dfb0b9 [id=9e14de0e4aeb473ea230c57fdd29889f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.474550 27684 ts_tablet_manager.cc:1939] T a3103e1595ec4c38ba65dca26fe1167c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:37.448894 27496 tablet_service.cc:1468] Processing CreateTablet for tablet 98d89282fc684dceb735e58ded9ea000 (DEFAULT_TABLE table=default.Table_12c3415978494757b57a3c08462fa742 [id=9963800b602f4c5c934854b044d6bd0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.453697 27491 tablet_service.cc:1468] Processing CreateTablet for tablet 37071b15494b4c5ea450e34f7b93aafa (DEFAULT_TABLE table=default.Table_96bef96c2061427d8b6643208730463c [id=7a5292efb3fa4041ad3f50fa37a449b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.491050 27693 ts_tablet_manager.cc:1428] T 30cc0d4ba8d444f1b98232f906ea0c7b P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.042s user 0.002s sys 0.000s
I20250903 20:05:37.472880 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4d3e1cfdfab4be1a9143b78bf3bc46d" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:37.491904 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 5002b7e8118c49b6a247cb8dc4ca1aea (table default.Table_b20633c3b14e463aac0b7bd917c3ab9b [id=c6a148f02792496e8d8f2bb2dfc5e69b]) successfully deleted
I20250903 20:05:37.494457 27501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fd11faff1f54553accb793aa910303d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.494781 27503 tablet_service.cc:1468] Processing CreateTablet for tablet 64af6482653844ea93abb850b86ae2ea (DEFAULT_TABLE table=default.Table_43b6194256e1461cb34be4e7642889f2 [id=8d86a912eeb54b99a494678329a80799]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.495152 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64af6482653844ea93abb850b86ae2ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.496026 27378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e9a6a6e05464b7b97ae624f51402cf0" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:37.497288 29411 raft_consensus.cc:491] T 7824a0bccdc9485abde124572e502791 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:37.497423 29411 raft_consensus.cc:513] T 7824a0bccdc9485abde124572e502791 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.497956 29411 leader_election.cc:290] T 7824a0bccdc9485abde124572e502791 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:37.498382 27225 tablet_service.cc:1515] Processing DeleteTablet for tablet 5bff3d77088c4b69a90365950641e300 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:37.498409 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ee40566e497740789575275d0cdca0db" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:37.498620 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet cc71346bf84f4a708bf17b5aba174582 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:37.499044 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 19c012cef4864d97add4fcf7993fbba5 (DEFAULT_TABLE table=default.Table_1b90687e7c414e00ba0897296f8c4ea5 [id=1dea2627e025433b83bc68e84236d7a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.499454 27225 tablet_service.cc:1468] Processing CreateTablet for tablet f963ef50ecf64a0fa336d84783bc9fd9 (DEFAULT_TABLE table=default.Table_2f4fdec99f854b22bc5262d859311c48 [id=73766a120cbe41adb2a9c51fd5697d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.500684 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 1ed519e2e9c94fa3a134dcfc8d5364e6 (DEFAULT_TABLE table=default.Table_48a0a980c12f40128907043e0ad8a7a8 [id=6c8ffb4fd82644f6a75a57bb196c6d6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.501037 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ed519e2e9c94fa3a134dcfc8d5364e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.501390 27497 tablet_service.cc:1468] Processing CreateTablet for tablet 55711d641a664e3996230d7c6a256ec1 (DEFAULT_TABLE table=default.Table_9f971775443a46128184a77aa0233d86 [id=3a23a6f1ed57437bbf971f3bc4393e0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.501662 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55711d641a664e3996230d7c6a256ec1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:37.501832 27462 leader_election.cc:343] T f4d3e1cfdfab4be1a9143b78bf3bc46d P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:37.501995 27462 leader_election.cc:304] T f4d3e1cfdfab4be1a9143b78bf3bc46d P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:37.502494 29371 raft_consensus.cc:2747] T f4d3e1cfdfab4be1a9143b78bf3bc46d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:37.504658 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36932534c9094a1980cf2ecb6074ebd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.504834 27492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 831d2bfa62e2439396fa53576201413d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.505254 27489 tablet_service.cc:1468] Processing CreateTablet for tablet a0890b5be4cd4c8995da37b47e52b820 (DEFAULT_TABLE table=default.Table_017a71fa3a044b73bd2ced468094acac [id=b0e506ed02a44adbb79969b2accf9816]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.506055 27693 tablet_bootstrap.cc:492] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:37.500988 27507 tablet_service.cc:1468] Processing CreateTablet for tablet 1c75bfd1a2f1489393fed76d6e2eb31c (DEFAULT_TABLE table=default.Table_a20d16f1ce384a83998a23435eb76dff [id=a4cb747dbb9c495e85cc2497b4f0397f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.509294 27693 tablet_bootstrap.cc:654] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:37.500684 27493 tablet_service.cc:1468] Processing CreateTablet for tablet 7f909c505e774b86ad107dc03a592180 (DEFAULT_TABLE table=default.Table_f0fc60c750344a2d96326556708d661e [id=ee453f94b790498f84e3488ce8b0d809]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.510998 27495 tablet_service.cc:1468] Processing CreateTablet for tablet b78645d4e9574ce3b2a78eebb541fbfe (DEFAULT_TABLE table=default.Table_2ae7f1d891744aeca75e2ed6bf43666f [id=af2de94e42d44e389fac2030cde1f623]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.514257 27501 tablet_service.cc:1468] Processing CreateTablet for tablet a307376a81904146ae8eeb39f11d9235 (DEFAULT_TABLE table=default.Table_9cac2ed28d064dfebdc8acfc692b00d9 [id=76cde1597b644ba28f701714955dfdae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.517470 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7824a0bccdc9485abde124572e502791" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:37.517549 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "64c4dd2645034a7eaf243f5474603f10" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:37.517684 27516 raft_consensus.cc:2466] T 7824a0bccdc9485abde124572e502791 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:37.518038 27692 tablet_bootstrap.cc:492] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:37.519968 27498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6d7f237dfce419c83e1d5c9fa12814e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.520045 27692 tablet_bootstrap.cc:654] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:37.522476 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 740b7d1b3c5545c78aff4629858ac7f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.522667 27685 ts_tablet_manager.cc:1918] T a0f79831866b40c49c5b27ce2014c3f6 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:37.522785 27685 log.cc:1199] T a0f79831866b40c49c5b27ce2014c3f6 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/a0f79831866b40c49c5b27ce2014c3f6
I20250903 20:05:37.523125 27685 ts_tablet_manager.cc:1939] T a0f79831866b40c49c5b27ce2014c3f6 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:37.482102 27687 ts_tablet_manager.cc:1918] T 886b8783aa9d4618aad498d4a1115843 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:37.527784 27503 tablet_service.cc:1468] Processing CreateTablet for tablet 0c2ea06c87b243d9a1aecf4a81841dfd (DEFAULT_TABLE table=default.Table_36f5d67acb72437493d261988d5ccb28 [id=8c216327759c4647b9cabb512e8cba6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.527874 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 701a652bc9d74992aabd6603cc93d0be (DEFAULT_TABLE table=default.Table_351c22f0442944f889808350e50d2887 [id=7e80b43d1ad04c998e1e47257cc44a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.527957 27687 log.cc:1199] T 886b8783aa9d4618aad498d4a1115843 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/886b8783aa9d4618aad498d4a1115843
I20250903 20:05:37.528749 27497 tablet_service.cc:1468] Processing CreateTablet for tablet 658ee69734254d33af0dce3b4407a41f (DEFAULT_TABLE table=default.Table_37b6f939fca548ada1f45f908c602ca9 [id=e43eca3152194fd5a1ce3311b4e55b0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.529037 27492 tablet_service.cc:1468] Processing CreateTablet for tablet f1843a04837d4b76a80a38660ea8cbc5 (DEFAULT_TABLE table=default.Table_fdf6ca8eecbb4d1cb0fd5fcd4e67b5a7 [id=19263ce3f63d41c49677bce476201881]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.529268 27183 leader_election.cc:304] T 7824a0bccdc9485abde124572e502791 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:37.529979 27496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98d89282fc684dceb735e58ded9ea000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.530158 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f909c505e774b86ad107dc03a592180. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.530369 27490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b0c7097d19a412bbe5720126621b79d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.530632 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 658ee69734254d33af0dce3b4407a41f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.531080 27494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 071f2a97aed640ddad8119ec4fc452e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.531298 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1113) took 2788 ms. Trace:
I20250903 20:05:37.531455 27507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c75bfd1a2f1489393fed76d6e2eb31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.531867 27489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0890b5be4cd4c8995da37b47e52b820. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.531782 27695 raft_consensus.cc:357] T 259bc73ca9df47e9874692725a60bb1c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.531971 27695 raft_consensus.cc:383] T 259bc73ca9df47e9874692725a60bb1c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:37.532023 27695 raft_consensus.cc:738] T 259bc73ca9df47e9874692725a60bb1c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:37.532232 29196 raft_consensus.cc:2802] T 7824a0bccdc9485abde124572e502791 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:37.532348 29196 raft_consensus.cc:491] T 7824a0bccdc9485abde124572e502791 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:37.532423 29196 raft_consensus.cc:3058] T 7824a0bccdc9485abde124572e502791 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:37.532249 27695 consensus_queue.cc:260] T 259bc73ca9df47e9874692725a60bb1c P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.532582 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d5e686dcd754132b9d7ba8ce808bbbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.533017 27491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37071b15494b4c5ea450e34f7b93aafa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.533051 27695 ts_tablet_manager.cc:1428] T 259bc73ca9df47e9874692725a60bb1c P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.117s user 0.001s sys 0.001s
I20250903 20:05:37.533366 27499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a655abd16c9c4f50b86fb9c3d12b3805. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.533812 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a5e2cc1df3d46b9b34feaa1fdfb7ed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.534142 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b801bf27b5a4e66812d127d1c5a47e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.534400 29371 raft_consensus.cc:491] T b4b15d77d07c4fa7856d9c198cf15450 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:37.534528 29371 raft_consensus.cc:513] T b4b15d77d07c4fa7856d9c198cf15450 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.535240 29371 leader_election.cc:290] T b4b15d77d07c4fa7856d9c198cf15450 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:37.536525 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b4b15d77d07c4fa7856d9c198cf15450" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:37.537328 27226 consensus_meta.cc:287] T 42e618f4f375464a87eb7fb9e4265fb6 P 89f8f0ee68b940db9ce250c0604f934a: Time spent flushing consensus metadata: real 0.541s user 0.022s sys 0.000s
W20250903 20:05:37.537388 27462 leader_election.cc:343] T b4b15d77d07c4fa7856d9c198cf15450 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:37.537997 29196 raft_consensus.cc:513] T 7824a0bccdc9485abde124572e502791 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.539077 29196 leader_election.cc:290] T 7824a0bccdc9485abde124572e502791 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:37.539737 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b4b15d77d07c4fa7856d9c198cf15450" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:37.539932 27222 tablet_service.cc:1468] Processing CreateTablet for tablet 9ef10e3c467a4966ad936a384e26acb9 (DEFAULT_TABLE table=default.Table_584007f8f96047bfa933decba8b5d048 [id=2055e46a1fab4ab087ffe4206ea7322c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.540642 27492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1843a04837d4b76a80a38660ea8cbc5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:37.540583 27462 leader_election.cc:343] T b4b15d77d07c4fa7856d9c198cf15450 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:37.540741 27462 leader_election.cc:304] T b4b15d77d07c4fa7856d9c198cf15450 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:37.541206 29371 raft_consensus.cc:2747] T b4b15d77d07c4fa7856d9c198cf15450 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:37.541194 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a4198b4841b4c2585e6c2f5c6a655d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c2ea06c87b243d9a1aecf4a81841dfd) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:37.541800 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 6d614ed6d2fe4217bc2d289c12554efa (DEFAULT_TABLE table=default.Table_092cc68e6e774d91ba49dc1622f26b0b [id=b8645b0a55534fbb82818616e31b3b96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.543673 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 701a652bc9d74992aabd6603cc93d0be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.543596 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 8df79821c6aa43b69d4f4e2ad00272df (DEFAULT_TABLE table=default.Table_badd71a0f177471b805d80d3e5652d50 [id=fe4325e8c68640a2872f44a264e8a5c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.544332 27217 tablet_service.cc:1468] Processing CreateTablet for tablet c6d7f237dfce419c83e1d5c9fa12814e (DEFAULT_TABLE table=default.Table_32800b753a384355979ff8944551c12c [id=ac2188e4478c4956bc105df3aa19920d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.544417 27220 tablet_service.cc:1468] Processing CreateTablet for tablet 3fd1cfb6e3264e9db2a1c75d468d6186 (DEFAULT_TABLE table=default.Table_e57f2570b5f3401cbadd48277ed8993e [id=a9fe04b7569c441c93c00b922cab4a17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.544863 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7824a0bccdc9485abde124572e502791" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
W20250903 20:05:37.545106 27183 consensus_peers.cc:489] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1d22e76ea72441db51f7e9db26eb520. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:37.545145 27511 raft_consensus.cc:3058] T 7824a0bccdc9485abde124572e502791 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:37.545428 27501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a307376a81904146ae8eeb39f11d9235. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.548449 27218 tablet_service.cc:1468] Processing CreateTablet for tablet d41de82304bf453fab8b15d96cf32a74 (DEFAULT_TABLE table=default.Table_0276baa7ef3b41c994e5d131569d5955 [id=4ff425ad3eeb4abca1887f5127cda1cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.549095 27221 tablet_service.cc:1468] Processing CreateTablet for tablet 36932534c9094a1980cf2ecb6074ebd2 (DEFAULT_TABLE table=default.Table_694c90191216407784c7fea5d5379e2d [id=1261af79c2f845c58d6de2bae4b2d8ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.549937 27511 raft_consensus.cc:2466] T 7824a0bccdc9485abde124572e502791 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:37.549919 27212 tablet_service.cc:1468] Processing CreateTablet for tablet ce87a2b6f3a44d3c8ae3b39d158ddb33 (DEFAULT_TABLE table=default.Table_6d5082ee3fd94074a33157509c0b26c5 [id=9da1236f96c24c029aae89acadf41154]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.550388 27216 tablet_service.cc:1468] Processing CreateTablet for tablet 2bf6c73765534237ba3f1ccb751192b2 (DEFAULT_TABLE table=default.Table_e02c690220fb4ab2a949d23f2230eeca [id=9ffa3b31524942c4b07a8e11956d02fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.550688 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19c012cef4864d97add4fcf7993fbba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.550930 27216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bf6c73765534237ba3f1ccb751192b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.551160 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f963ef50ecf64a0fa336d84783bc9fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.551800 27228 tablet_service.cc:1468] Processing CreateTablet for tablet 25ce665c6ee3451aae2b8281131435ae (DEFAULT_TABLE table=default.Table_8c7c15fd52a0453ca91d7fef1fed6adb [id=f77c17022eae4823a94a80694f054b47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.552141 27228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25ce665c6ee3451aae2b8281131435ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.558051 27216 tablet_service.cc:1468] Processing CreateTablet for tablet 740b7d1b3c5545c78aff4629858ac7f1 (DEFAULT_TABLE table=default.Table_d73ecd1c981c4cfdbebd10521a2b86eb [id=562751cb22694d6caa06088c3f7490c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.558589 27216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 740b7d1b3c5545c78aff4629858ac7f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.565371 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b78645d4e9574ce3b2a78eebb541fbfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.565788 27488 tablet_service.cc:1468] Processing CreateTablet for tablet ccb671d7e67444ac8e5b78055271d02a (DEFAULT_TABLE table=default.Table_d0aec832c86d4e868563056c6100c6b8 [id=8d69d1671a82411da646bc38ae0a5109]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.566162 27488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccb671d7e67444ac8e5b78055271d02a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.531383 27684 rpcz_store.cc:276] 0903 20:05:34.743231 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.743291 (+ 60us) service_pool.cc:225] Handling call
0903 20:05:37.380899 (+2637608us) ts_tablet_manager.cc:1145] Deleting tablet a3103e1595ec4c38ba65dca26fe1167c
0903 20:05:37.380912 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:37.391503 (+ 10591us) spinlock_profiling.cc:243] Waited 9.52 ms on lock 0x6160000f8e54. stack: 00007f71bc6f7dc6 00007f71bc6f7b80 00007f71bb6df3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f71c8181600 00007f71c9494b13 00007f71cbf772b7 00007f71cbfa9344 00007f71cbf8ada0 00007f71cbf8ab2d 0000000000a774dc 00007f71bc73e14f 00007f71bc74141b 00007f71bc741298
0903 20:05:37.421468 (+ 29965us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.425055 (+ 3587us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.531278 (+106223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21904256,"tablet-delete.queue_time_us":2637311}
I20250903 20:05:37.568960 27695 tablet_bootstrap.cc:492] T de20d18d7f264811a0631aa1144aef2c P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:37.569924 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1115) took 2804 ms. Trace:
I20250903 20:05:37.570254 27183 leader_election.cc:304] T 7824a0bccdc9485abde124572e502791 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:37.570459 27684 rpcz_store.cc:276] 0903 20:05:34.765790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.765857 (+ 67us) service_pool.cc:225] Handling call
0903 20:05:37.569775 (+2803918us) ts_tablet_manager.cc:1145] Deleting tablet 52ee6ba19a734329acde7608327f2a78
0903 20:05:37.569784 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:37.569905 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2803721}
I20250903 20:05:37.570868 27684 tablet_replica.cc:331] T 24eadc70e10d4d02bcb643d45c9e0e36 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:37.570979 27695 tablet_bootstrap.cc:654] T de20d18d7f264811a0631aa1144aef2c P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:37.571125 27684 raft_consensus.cc:2241] T 24eadc70e10d4d02bcb643d45c9e0e36 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:37.571313 27684 raft_consensus.cc:2270] T 24eadc70e10d4d02bcb643d45c9e0e36 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:37.571556 29411 raft_consensus.cc:2802] T 7824a0bccdc9485abde124572e502791 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:37.571692 29411 raft_consensus.cc:695] T 7824a0bccdc9485abde124572e502791 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:37.571938 29411 consensus_queue.cc:237] T 7824a0bccdc9485abde124572e502791 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.573892 27684 ts_tablet_manager.cc:1905] T 24eadc70e10d4d02bcb643d45c9e0e36 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:37.576297 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c2ea06c87b243d9a1aecf4a81841dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.586604 27211 tablet_service.cc:1468] Processing CreateTablet for tablet bf35e907252244118b8ee9e99e0f0415 (DEFAULT_TABLE table=default.Table_1828d17a24374724b13ff20d37532044 [id=4bb622e4d8174344b74bcfa8ba0e8f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.587533 27687 ts_tablet_manager.cc:1939] T 886b8783aa9d4618aad498d4a1115843 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:37.589763 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1128) took 2883 ms. Trace:
I20250903 20:05:37.589767 27065 catalog_manager.cc:5582] T 7824a0bccdc9485abde124572e502791 P 89f8f0ee68b940db9ce250c0604f934a reported cstate change: term changed from 0 to 1, leader changed from <none> to 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65). New cstate: current_term: 1 leader_uuid: "89f8f0ee68b940db9ce250c0604f934a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: HEALTHY } } }
I20250903 20:05:37.592116 27684 ts_tablet_manager.cc:1918] T 24eadc70e10d4d02bcb643d45c9e0e36 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:37.592738 27687 rpcz_store.cc:276] 0903 20:05:34.706715 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.706763 (+ 48us) service_pool.cc:225] Handling call
0903 20:05:37.377053 (+2670290us) ts_tablet_manager.cc:1145] Deleting tablet 886b8783aa9d4618aad498d4a1115843
0903 20:05:37.377064 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:37.478026 (+100962us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.481973 (+ 3947us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.589738 (+107765us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2670116}
I20250903 20:05:37.593108 27214 tablet_service.cc:1468] Processing CreateTablet for tablet 98d89282fc684dceb735e58ded9ea000 (DEFAULT_TABLE table=default.Table_12c3415978494757b57a3c08462fa742 [id=9963800b602f4c5c934854b044d6bd0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.593612 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 886b8783aa9d4618aad498d4a1115843 (table default.Table_a540eea45dfb46ec92b02a3f9a2b3aed [id=5fe055478b724744a65fcaa2f25bef0d]) successfully deleted
I20250903 20:05:37.594101 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 071f2a97aed640ddad8119ec4fc452e5 (DEFAULT_TABLE table=default.Table_357d7af9ec6f493582c7c7c5a3b28414 [id=25a9593fadf44b65bcb534989f669fa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.594645 27687 tablet_replica.cc:331] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:37.595116 27687 raft_consensus.cc:2241] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:37.595811 27687 raft_consensus.cc:2270] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:37.595971 27227 tablet_service.cc:1468] Processing CreateTablet for tablet 0a5e2cc1df3d46b9b34feaa1fdfb7ed7 (DEFAULT_TABLE table=default.Table_019bea9491a242768b3d8863a9e0194b [id=df13920c040e49118af46fd13d812bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.596987 27213 tablet_service.cc:1468] Processing CreateTablet for tablet 1d5e686dcd754132b9d7ba8ce808bbbe (DEFAULT_TABLE table=default.Table_aeea04c58a4643328ab1223efbed4a4a [id=d18967040fc54e09a7fc8c7eb4997a8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.598151 27219 tablet_service.cc:1468] Processing CreateTablet for tablet 6b0c7097d19a412bbe5720126621b79d (DEFAULT_TABLE table=default.Table_781c5fddf2e945b5bf4c68483f03dde5 [id=aac5db259297453fa9fe0785d33f8a6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.598337 27223 tablet_service.cc:1468] Processing CreateTablet for tablet 1fd11faff1f54553accb793aa910303d (DEFAULT_TABLE table=default.Table_0f58bdb4d1eb4aa9bb8d36909880410b [id=9b40fbee58ce4d71810c2ab96a6fee11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.598519 27229 tablet_service.cc:1468] Processing CreateTablet for tablet 37071b15494b4c5ea450e34f7b93aafa (DEFAULT_TABLE table=default.Table_96bef96c2061427d8b6643208730463c [id=7a5292efb3fa4041ad3f50fa37a449b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.598868 27224 tablet_service.cc:1468] Processing CreateTablet for tablet 831d2bfa62e2439396fa53576201413d (DEFAULT_TABLE table=default.Table_03367a125ef440d2bc8c3f1531dfb0b9 [id=9e14de0e4aeb473ea230c57fdd29889f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.598914 27210 tablet_service.cc:1468] Processing CreateTablet for tablet 34eadb8f690d41c0877eeeb00d4784a8 (DEFAULT_TABLE table=default.Table_fa0af1d57fb048e7ab3e84f79c1f674d [id=1ffea37ca73649fcb8d8ea85f1b34c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.601605 27228 tablet_service.cc:1468] Processing CreateTablet for tablet 7b801bf27b5a4e66812d127d1c5a47e3 (DEFAULT_TABLE table=default.Table_a204d49d7f1e4eeabb2f685a5ee69a40 [id=dd3c6590d6fe4fe78fafd7c236e8a811]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.602710 27211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf35e907252244118b8ee9e99e0f0415. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.606488 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ef10e3c467a4966ad936a384e26acb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.610443 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8df79821c6aa43b69d4f4e2ad00272df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.614450 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6d7f237dfce419c83e1d5c9fa12814e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.615780 27687 ts_tablet_manager.cc:1905] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:37.618443 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd1cfb6e3264e9db2a1c75d468d6186. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.618028 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d614ed6d2fe4217bc2d289c12554efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.622478 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d41de82304bf453fab8b15d96cf32a74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.626338 27684 log.cc:1199] T 24eadc70e10d4d02bcb643d45c9e0e36 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/24eadc70e10d4d02bcb643d45c9e0e36
I20250903 20:05:37.626732 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36932534c9094a1980cf2ecb6074ebd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.629004 27684 ts_tablet_manager.cc:1939] T 24eadc70e10d4d02bcb643d45c9e0e36 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:37.629773 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1116) took 2862 ms. Trace:
I20250903 20:05:37.629854 27684 rpcz_store.cc:276] 0903 20:05:34.767510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.767566 (+ 56us) service_pool.cc:225] Handling call
0903 20:05:37.570816 (+2803250us) ts_tablet_manager.cc:1145] Deleting tablet 24eadc70e10d4d02bcb643d45c9e0e36
0903 20:05:37.570827 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:37.576968 (+ 6141us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.591986 (+ 15018us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.629753 (+ 37767us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2803043}
I20250903 20:05:37.630461 27212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce87a2b6f3a44d3c8ae3b39d158ddb33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.632733 27065 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 793d6157927143d4b9d8dbafd7cf0fcb (table default.Table_e79f52615fe142b9823a2a99b8b9c406 [id=3432f6ac076b4e49b9fe63bf5fce8591]). Expected version 1 got 0
I20250903 20:05:37.635797 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1000) took 4796 ms. Trace:
I20250903 20:05:37.635782 27065 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_739bc0d4c1d4473caac21639a4ade167" } modify_external_catalogs: true
I20250903 20:05:37.636045 27065 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_739bc0d4c1d4473caac21639a4ade167" } modify_external_catalogs: true
I20250903 20:05:37.636056 27685 rpcz_store.cc:276] 0903 20:05:32.839183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.839228 (+ 45us) service_pool.cc:225] Handling call
0903 20:05:37.264704 (+4425476us) ts_tablet_manager.cc:1145] Deleting tablet a0f79831866b40c49c5b27ce2014c3f6
0903 20:05:37.264715 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:37.293395 (+ 28680us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.522524 (+229129us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.635774 (+113250us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4425308}
I20250903 20:05:37.636592 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1002) took 4796 ms. Trace:
I20250903 20:05:37.636721 27685 rpcz_store.cc:276] 0903 20:05:32.839648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.839699 (+ 51us) service_pool.cc:225] Handling call
0903 20:05:37.636494 (+4796795us) ts_tablet_manager.cc:1145] Deleting tablet 1bcb9d5f010f4297bab78c0183889238
0903 20:05:37.636505 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:37.636579 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4796609}
I20250903 20:05:37.637027 27685 tablet_replica.cc:331] T 8fff36d5a80e45af8a256c3908555e08 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:37.637249 27685 raft_consensus.cc:2241] T 8fff36d5a80e45af8a256c3908555e08 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:37.637467 27685 raft_consensus.cc:2270] T 8fff36d5a80e45af8a256c3908555e08 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:37.638722 27685 ts_tablet_manager.cc:1905] T 8fff36d5a80e45af8a256c3908555e08 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:37.639155 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5cbc8c64bdd34f028ab716135624ab49" } modify_external_catalogs: true
I20250903 20:05:37.639353 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5cbc8c64bdd34f028ab716135624ab49" } modify_external_catalogs: true
I20250903 20:05:37.640229 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_127e4ebeb1d54cb48599c668f7c2aa49" } modify_external_catalogs: true
I20250903 20:05:37.640384 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_127e4ebeb1d54cb48599c668f7c2aa49" } modify_external_catalogs: true
I20250903 20:05:37.640908 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_4277890a8cb0439db05edde0b26bd8bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:37.641448 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4277890a8cb0439db05edde0b26bd8bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:37.641716 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ab328ca9be0d4484a401d9fdfd047d15" } new_table_name: "default.Table_eff548f7227c4ef1b557962035b4ba9e" modify_external_catalogs: true
I20250903 20:05:37.641991 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet ee40566e497740789575275d0cdca0db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:37.642323 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet 47c1287bff1145a5b2ab6d14dbd608c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:37.642865 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_bbe7d51c5b5140b68deb8ad68216a05e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:37.643388 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbe7d51c5b5140b68deb8ad68216a05e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:37.644672 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_dec07ab3580e47e1a0be5bb8eb0dd841"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:37.645272 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dec07ab3580e47e1a0be5bb8eb0dd841 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:37.646410 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_48ce0908f41148948ef0a0d8907a8b45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:37.646857 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_48ce0908f41148948ef0a0d8907a8b45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:37.648054 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_8d0b63b0e60f4b2ba191abfd7f8680b4" } modify_external_catalogs: true
I20250903 20:05:37.648198 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_8d0b63b0e60f4b2ba191abfd7f8680b4" } modify_external_catalogs: true
I20250903 20:05:37.648871 27693 tablet_bootstrap.cc:492] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:37.649262 27693 ts_tablet_manager.cc:1397] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.143s user 0.005s sys 0.000s
I20250903 20:05:37.642890 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_10a7df6626f24353ae6661cbbdf3efb0" } modify_external_catalogs: true
I20250903 20:05:37.649710 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_10a7df6626f24353ae6661cbbdf3efb0" } modify_external_catalogs: true
I20250903 20:05:37.650082 27693 raft_consensus.cc:357] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.650360 27693 raft_consensus.cc:383] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:37.650429 27693 raft_consensus.cc:738] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:37.650739 27693 consensus_queue.cc:260] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.654660 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 503dba32175c4459892c62d3f80693fb
I20250903 20:05:37.655525 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet e4ec4cc1ef914cffa2d304cdf701cf33
I20250903 20:05:37.655997 27057 master_service.cc:948] ReplaceTablet: received request to replace tablet a0f79831866b40c49c5b27ce2014c3f6 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:37.663074 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet e4ec4cc1ef914cffa2d304cdf701cf33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:37 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:37.663226 29371 raft_consensus.cc:491] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:37.663447 29371 raft_consensus.cc:513] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.664325 29371 leader_election.cc:290] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:37.665699 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 74eab6ad2fe64b18bd00c0d5524428c6
I20250903 20:05:37.667984 27222 tablet_service.cc:1468] Processing CreateTablet for tablet 64af6482653844ea93abb850b86ae2ea (DEFAULT_TABLE table=default.Table_43b6194256e1461cb34be4e7642889f2 [id=8d86a912eeb54b99a494678329a80799]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.668409 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64af6482653844ea93abb850b86ae2ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.669546 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3919f7f95a954f59b68f56d0051c0109" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:37.670011 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 74eab6ad2fe64b18bd00c0d5524428c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:37 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:37.671409 27462 leader_election.cc:343] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:37.671955 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3919f7f95a954f59b68f56d0051c0109" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:37.672235 27239 raft_consensus.cc:2466] T 3919f7f95a954f59b68f56d0051c0109 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:37.672887 27462 leader_election.cc:304] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:37.673547 29371 raft_consensus.cc:2802] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:37.673650 29371 raft_consensus.cc:491] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:37.673729 29371 raft_consensus.cc:3058] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:37.674659 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 14f97a610f25488eb510ba0ce966470d consensus state has no leader
I20250903 20:05:37.677613 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ca622380edab4739afde521d34a1cd81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:37.677836 29371 raft_consensus.cc:513] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.678015 27057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1feef4303b4e43809a0a3f1a79d68b30" } new_table_name: "default.Table_9863ce9347594da49566eb066e9436d9" modify_external_catalogs: true
W20250903 20:05:37.678344 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca622380edab4739afde521d34a1cd81 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:37.678517 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet 503dba32175c4459892c62d3f80693fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:37 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:37.679804 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3919f7f95a954f59b68f56d0051c0109" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
W20250903 20:05:37.680526 27462 leader_election.cc:343] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:37.681576 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3919f7f95a954f59b68f56d0051c0109" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:37.681826 27239 raft_consensus.cc:3058] T 3919f7f95a954f59b68f56d0051c0109 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:37.682945 29412 raft_consensus.cc:491] T 3f303233e6c347289650c6cf83f8127f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:37.683086 29412 raft_consensus.cc:513] T 3f303233e6c347289650c6cf83f8127f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.683005 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_dd4b90aea2be4292899c22877f734f3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:37.683660 29412 leader_election.cc:290] T 3f303233e6c347289650c6cf83f8127f P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
W20250903 20:05:37.683858 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd4b90aea2be4292899c22877f734f3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:37.684195 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_041cccd0d04d430c8c06d135afd3b91e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:37.684614 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_041cccd0d04d430c8c06d135afd3b91e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:37.685046 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet d7d37849132d4420bb5fcb57c11cfb6b
I20250903 20:05:37.685732 27239 raft_consensus.cc:2466] T 3919f7f95a954f59b68f56d0051c0109 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:37.686028 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_74c673867bf9445195f037ede94c26b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:37.686534 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74c673867bf9445195f037ede94c26b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:37.687068 29412 raft_consensus.cc:491] T 259bc73ca9df47e9874692725a60bb1c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:37.687213 29412 raft_consensus.cc:513] T 259bc73ca9df47e9874692725a60bb1c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.687870 29412 leader_election.cc:290] T 259bc73ca9df47e9874692725a60bb1c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:37.688683 27462 leader_election.cc:304] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:37.688714 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_9b20469406c94986ab496ea3e678b586" } new_table_name: "default.Table_c1a71b1082b54337a2d52b446d1d4258" modify_external_catalogs: true
I20250903 20:05:37.689422 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet d7d37849132d4420bb5fcb57c11cfb6b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:37 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:37.690315 29371 leader_election.cc:290] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:37.690761 29371 raft_consensus.cc:2802] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:37.691159 29371 raft_consensus.cc:695] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:37.691642 29371 consensus_queue.cc:237] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.692098 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f573c7cc8d7240c0bf60622441b33a69" } new_table_name: "default.Table_511ad95d56b5416c89bced96468410a4" modify_external_catalogs: true
I20250903 20:05:37.691934 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "259bc73ca9df47e9874692725a60bb1c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:37.692593 27516 raft_consensus.cc:2466] T 259bc73ca9df47e9874692725a60bb1c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:37.693014 27183 leader_election.cc:304] T 259bc73ca9df47e9874692725a60bb1c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:37.694396 27693 ts_tablet_manager.cc:1428] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.045s user 0.002s sys 0.000s
I20250903 20:05:37.694417 29412 raft_consensus.cc:2802] T 259bc73ca9df47e9874692725a60bb1c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:37.694504 29412 raft_consensus.cc:491] T 259bc73ca9df47e9874692725a60bb1c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:37.694581 29412 raft_consensus.cc:3058] T 259bc73ca9df47e9874692725a60bb1c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:37.694669 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_8a7162fd4e2540d5a4b2a7bccac7af08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:37.695289 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a7162fd4e2540d5a4b2a7bccac7af08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:37.696157 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ab6863cab5e04e57b58bc14d9a3d379d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:37.696677 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab6863cab5e04e57b58bc14d9a3d379d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:37.697360 29412 raft_consensus.cc:513] T 259bc73ca9df47e9874692725a60bb1c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.698549 29412 leader_election.cc:290] T 259bc73ca9df47e9874692725a60bb1c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:37.698675 27692 tablet_bootstrap.cc:492] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:37.698822 27692 ts_tablet_manager.cc:1397] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.207s user 0.007s sys 0.000s
I20250903 20:05:37.698916 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f303233e6c347289650c6cf83f8127f" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:37.699123 27511 raft_consensus.cc:2466] T 3f303233e6c347289650c6cf83f8127f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:37.699352 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "259bc73ca9df47e9874692725a60bb1c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:37.699499 27516 raft_consensus.cc:3058] T 259bc73ca9df47e9874692725a60bb1c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:37.699692 27692 raft_consensus.cc:357] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.699838 27692 raft_consensus.cc:383] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:37.699888 27692 raft_consensus.cc:738] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:37.700111 27692 consensus_queue.cc:260] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.700577 27692 ts_tablet_manager.cc:1428] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:37.700868 27692 tablet_bootstrap.cc:492] T 1bf570855b204b96acd070587c434463 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:37.702721 27693 tablet_bootstrap.cc:492] T 259bc73ca9df47e9874692725a60bb1c P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:37.703610 27516 raft_consensus.cc:2466] T 259bc73ca9df47e9874692725a60bb1c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:37.704228 27183 leader_election.cc:304] T 3f303233e6c347289650c6cf83f8127f P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:37.704361 27693 tablet_bootstrap.cc:654] T 259bc73ca9df47e9874692725a60bb1c P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:37.704636 27183 leader_election.cc:304] T 259bc73ca9df47e9874692725a60bb1c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:37.704870 29411 raft_consensus.cc:2802] T 3f303233e6c347289650c6cf83f8127f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:37.704947 29411 raft_consensus.cc:491] T 3f303233e6c347289650c6cf83f8127f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:37.705263 29411 raft_consensus.cc:3058] T 3f303233e6c347289650c6cf83f8127f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:37.707723 27211 tablet_service.cc:1468] Processing CreateTablet for tablet 1c75bfd1a2f1489393fed76d6e2eb31c (DEFAULT_TABLE table=default.Table_a20d16f1ce384a83998a23435eb76dff [id=a4cb747dbb9c495e85cc2497b4f0397f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.707937 29411 raft_consensus.cc:513] T 3f303233e6c347289650c6cf83f8127f P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.708271 27211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c75bfd1a2f1489393fed76d6e2eb31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.708559 29411 leader_election.cc:290] T 3f303233e6c347289650c6cf83f8127f P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:37.708729 29411 raft_consensus.cc:2802] T 259bc73ca9df47e9874692725a60bb1c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:37.708979 29411 raft_consensus.cc:695] T 259bc73ca9df47e9874692725a60bb1c P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:37.709630 29411 consensus_queue.cc:237] T 259bc73ca9df47e9874692725a60bb1c P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.710508 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1117) took 2898 ms. Trace:
I20250903 20:05:37.710613 27684 rpcz_store.cc:276] 0903 20:05:34.812564 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.812623 (+ 59us) service_pool.cc:225] Handling call
0903 20:05:37.710340 (+2897717us) ts_tablet_manager.cc:1145] Deleting tablet 7203b4ba9047430ab782d4c0498466cc
0903 20:05:37.710352 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:37.710484 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2896773}
I20250903 20:05:37.711519 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f303233e6c347289650c6cf83f8127f" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:37.711699 27516 raft_consensus.cc:3058] T 3f303233e6c347289650c6cf83f8127f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:37.712095 27692 tablet_bootstrap.cc:654] T 1bf570855b204b96acd070587c434463 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:37.713162 27212 tablet_service.cc:1468] Processing CreateTablet for tablet 55711d641a664e3996230d7c6a256ec1 (DEFAULT_TABLE table=default.Table_9f971775443a46128184a77aa0233d86 [id=3a23a6f1ed57437bbf971f3bc4393e0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.713616 27212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55711d641a664e3996230d7c6a256ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.714730 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1118) took 2875 ms. Trace:
I20250903 20:05:37.716091 27516 raft_consensus.cc:2466] T 3f303233e6c347289650c6cf83f8127f P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:37.716645 27183 leader_election.cc:304] T 3f303233e6c347289650c6cf83f8127f P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:37.716079 27684 rpcz_store.cc:276] 0903 20:05:34.838790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.842486 (+ 3696us) service_pool.cc:225] Handling call
0903 20:05:37.714575 (+2872089us) ts_tablet_manager.cc:1145] Deleting tablet baac5b97f7eb4d18a0f294ca1595a383
0903 20:05:37.714585 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:37.714707 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2871901}
I20250903 20:05:37.716948 29412 raft_consensus.cc:2802] T 3f303233e6c347289650c6cf83f8127f P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:37.717201 29412 raft_consensus.cc:695] T 3f303233e6c347289650c6cf83f8127f P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:37.717149 27217 tablet_service.cc:1468] Processing CreateTablet for tablet a0890b5be4cd4c8995da37b47e52b820 (DEFAULT_TABLE table=default.Table_017a71fa3a044b73bd2ced468094acac [id=b0e506ed02a44adbb79969b2accf9816]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.717514 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0890b5be4cd4c8995da37b47e52b820. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.717442 29412 consensus_queue.cc:237] T 3f303233e6c347289650c6cf83f8127f P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.719342 27221 tablet_service.cc:1468] Processing CreateTablet for tablet b78645d4e9574ce3b2a78eebb541fbfe (DEFAULT_TABLE table=default.Table_2ae7f1d891744aeca75e2ed6bf43666f [id=af2de94e42d44e389fac2030cde1f623]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.721599 27684 tablet_replica.cc:331] T 4e34d6d5d68e4395aed78821ea077c1b P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:37.720152 27215 tablet_service.cc:1468] Processing CreateTablet for tablet a307376a81904146ae8eeb39f11d9235 (DEFAULT_TABLE table=default.Table_9cac2ed28d064dfebdc8acfc692b00d9 [id=76cde1597b644ba28f701714955dfdae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.720825 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 7f909c505e774b86ad107dc03a592180 (DEFAULT_TABLE table=default.Table_f0fc60c750344a2d96326556708d661e [id=ee453f94b790498f84e3488ce8b0d809]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.724350 27684 raft_consensus.cc:2241] T 4e34d6d5d68e4395aed78821ea077c1b P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:37.724594 27684 raft_consensus.cc:2270] T 4e34d6d5d68e4395aed78821ea077c1b P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:37.700635 27220 tablet_service.cc:1468] Processing CreateTablet for tablet 1ed519e2e9c94fa3a134dcfc8d5364e6 (DEFAULT_TABLE table=default.Table_48a0a980c12f40128907043e0ad8a7a8 [id=6c8ffb4fd82644f6a75a57bb196c6d6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.721779 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 071f2a97aed640ddad8119ec4fc452e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.726398 27219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b0c7097d19a412bbe5720126621b79d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.726806 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d5e686dcd754132b9d7ba8ce808bbbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.727130 27223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fd11faff1f54553accb793aa910303d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.727447 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a5e2cc1df3d46b9b34feaa1fdfb7ed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.733996 27225 tablet_service.cc:1468] Processing CreateTablet for tablet 0c2ea06c87b243d9a1aecf4a81841dfd (DEFAULT_TABLE table=default.Table_36f5d67acb72437493d261988d5ccb28 [id=8c216327759c4647b9cabb512e8cba6b]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:37.734469 27051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.125:44059, user_credentials={real_user=slave}} blocked reactor thread for 88449.1us
I20250903 20:05:37.743745 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 09fa32312c9648fbbd6fd327e9738066 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:37.744755 27320 consensus_peers.cc:489] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 238f2f30377b42c2852b016018aa3245. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:37.748602 27684 ts_tablet_manager.cc:1905] T 4e34d6d5d68e4395aed78821ea077c1b P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:37.747638 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet a3103e1595ec4c38ba65dca26fe1167c (table default.Table_fdf0ad621407452eb74b6a9a56be2e85 [id=e4270da20ece47aaa8897f47a02d86d8]) successfully deleted
W20250903 20:05:37.756820 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 52ee6ba19a734329acde7608327f2a78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52ee6ba19a734329acde7608327f2a78 already in progress: creating tablet
I20250903 20:05:37.762467 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 24eadc70e10d4d02bcb643d45c9e0e36 (table default.Table_76a88a96cd544cda882c413357a7f8c3 [id=e9a291cff3f4497cad9393ab4c571bf7]) successfully deleted
W20250903 20:05:37.764490 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 7203b4ba9047430ab782d4c0498466cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7203b4ba9047430ab782d4c0498466cc already in progress: creating tablet
W20250903 20:05:37.765854 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet baac5b97f7eb4d18a0f294ca1595a383 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet baac5b97f7eb4d18a0f294ca1595a383 already in progress: creating tablet
I20250903 20:05:37.767387 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 503dba32175c4459892c62d3f80693fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:37 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:37.771150 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet e4ec4cc1ef914cffa2d304cdf701cf33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:37 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:37.772782 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 74eab6ad2fe64b18bd00c0d5524428c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:37 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:37.745456 27213 tablet_service.cc:1468] Processing CreateTablet for tablet 701a652bc9d74992aabd6603cc93d0be (DEFAULT_TABLE table=default.Table_351c22f0442944f889808350e50d2887 [id=7e80b43d1ad04c998e1e47257cc44a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.762423 27220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ed519e2e9c94fa3a134dcfc8d5364e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.777576 27687 ts_tablet_manager.cc:1918] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:37.778182 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet d7d37849132d4420bb5fcb57c11cfb6b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:37 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:37.781388 27687 log.cc:1199] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/ed096b452ba747f89e91edc15e6e3ac1
I20250903 20:05:37.782423 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f909c505e774b86ad107dc03a592180. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.786391 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a307376a81904146ae8eeb39f11d9235. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.787357 27687 ts_tablet_manager.cc:1939] T ed096b452ba747f89e91edc15e6e3ac1 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:37.790351 27214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98d89282fc684dceb735e58ded9ea000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.791551 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1129) took 3084 ms. Trace:
I20250903 20:05:37.792428 27221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b78645d4e9574ce3b2a78eebb541fbfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.797441 27213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 701a652bc9d74992aabd6603cc93d0be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.801673 27687 rpcz_store.cc:276] 0903 20:05:34.707292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.707351 (+ 59us) service_pool.cc:225] Handling call
0903 20:05:37.594589 (+2887238us) ts_tablet_manager.cc:1145] Deleting tablet ed096b452ba747f89e91edc15e6e3ac1
0903 20:05:37.594599 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:37.720210 (+125611us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.777417 (+ 57207us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.791521 (+ 14104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2887073}
I20250903 20:05:37.766506 27227 tablet_service.cc:1468] Processing CreateTablet for tablet f1843a04837d4b76a80a38660ea8cbc5 (DEFAULT_TABLE table=default.Table_fdf6ca8eecbb4d1cb0fd5fcd4e67b5a7 [id=19263ce3f63d41c49677bce476201881]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.802551 27227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1843a04837d4b76a80a38660ea8cbc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.803990 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet ed096b452ba747f89e91edc15e6e3ac1 (table default.Table_5bc4b150c12940db90c7800f5d07b9ed [id=27548347ece844e3bb55ba894da5c8e0]) successfully deleted
I20250903 20:05:37.767743 27219 tablet_service.cc:1468] Processing CreateTablet for tablet 6d614ed6d2fe4217bc2d289c12554efa (DEFAULT_TABLE table=default.Table_092cc68e6e774d91ba49dc1622f26b0b [id=b8645b0a55534fbb82818616e31b3b96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.806512 27219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d614ed6d2fe4217bc2d289c12554efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.806839 27687 tablet_replica.cc:331] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:37.807842 27687 raft_consensus.cc:2241] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:37.807916 27225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c2ea06c87b243d9a1aecf4a81841dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.759838 27223 tablet_service.cc:1468] Processing CreateTablet for tablet 658ee69734254d33af0dce3b4407a41f (DEFAULT_TABLE table=default.Table_37b6f939fca548ada1f45f908c602ca9 [id=e43eca3152194fd5a1ce3311b4e55b0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.810709 27223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 658ee69734254d33af0dce3b4407a41f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.810966 27687 raft_consensus.cc:2270] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:37.816473 27687 ts_tablet_manager.cc:1905] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:37.818696 27684 ts_tablet_manager.cc:1918] T 4e34d6d5d68e4395aed78821ea077c1b P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:37.821643 27684 log.cc:1199] T 4e34d6d5d68e4395aed78821ea077c1b P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/4e34d6d5d68e4395aed78821ea077c1b
I20250903 20:05:37.830289 27684 ts_tablet_manager.cc:1939] T 4e34d6d5d68e4395aed78821ea077c1b P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:37.832893 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1129) took 2696 ms. Trace:
I20250903 20:05:37.862273 27684 rpcz_store.cc:276] 0903 20:05:35.136859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.137518 (+ 659us) service_pool.cc:225] Handling call
0903 20:05:37.721521 (+2584003us) ts_tablet_manager.cc:1145] Deleting tablet 4e34d6d5d68e4395aed78821ea077c1b
0903 20:05:37.721539 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:37.778911 (+ 57372us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.818530 (+ 39619us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.832858 (+ 14328us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2583682}
I20250903 20:05:37.864125 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 4e34d6d5d68e4395aed78821ea077c1b (table default.Table_f2505e03caaa40c4b17d2e60b0196843 [id=08f0b93efef74aa59b1ec3c286b3b45b]) successfully deleted
I20250903 20:05:37.864988 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1128) took 2728 ms. Trace:
I20250903 20:05:37.866236 27684 rpcz_store.cc:276] 0903 20:05:35.136325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.140406 (+ 4081us) service_pool.cc:225] Handling call
0903 20:05:37.864782 (+2724376us) ts_tablet_manager.cc:1145] Deleting tablet 20d826f1b1d64746bd8e409a12b74eb1
0903 20:05:37.864803 (+ 21us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:37.864961 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2724066}
W20250903 20:05:37.867558 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 20d826f1b1d64746bd8e409a12b74eb1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20d826f1b1d64746bd8e409a12b74eb1 already in progress: creating tablet
I20250903 20:05:37.868284 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1131) took 2704 ms. Trace:
I20250903 20:05:37.869313 27684 rpcz_store.cc:276] 0903 20:05:35.163363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.163419 (+ 56us) service_pool.cc:225] Handling call
0903 20:05:37.868113 (+2704694us) ts_tablet_manager.cc:1145] Deleting tablet f28fd3e1e0e14cf48a244580b4f355d0
0903 20:05:37.868128 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:37.868261 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2704124}
W20250903 20:05:37.874953 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet f28fd3e1e0e14cf48a244580b4f355d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f28fd3e1e0e14cf48a244580b4f355d0 already in progress: creating tablet
I20250903 20:05:37.875309 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1132) took 2667 ms. Trace:
I20250903 20:05:37.875454 27684 rpcz_store.cc:276] 0903 20:05:35.207838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.207898 (+ 60us) service_pool.cc:225] Handling call
0903 20:05:37.875154 (+2667256us) ts_tablet_manager.cc:1145] Deleting tablet 69f8eef52bee431a86f0eef45fd11f9f
0903 20:05:37.875167 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:37.875283 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2667034}
W20250903 20:05:37.875988 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 69f8eef52bee431a86f0eef45fd11f9f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69f8eef52bee431a86f0eef45fd11f9f already in progress: creating tablet
I20250903 20:05:37.877852 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet e9b5ab312f8744af99d557c0733bf71d
I20250903 20:05:37.877933 27684 tablet_replica.cc:331] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:37.878270 27684 raft_consensus.cc:2241] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:37.879163 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet e9b5ab312f8744af99d557c0733bf71d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:37 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:37.879716 27057 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c3c4f008ae4a422d9f6d0272097e9686 consensus state has no leader
I20250903 20:05:37.881755 27684 raft_consensus.cc:2270] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:37.882658 27684 ts_tablet_manager.cc:1905] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:37.886878 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet e9b5ab312f8744af99d557c0733bf71d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:37 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:37.888514 28567 raft_consensus.cc:491] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:37.889106 29371 raft_consensus.cc:491] T 5b46bb34ac214b1ba23fba1a33f746ac P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:37.889310 29371 raft_consensus.cc:513] T 5b46bb34ac214b1ba23fba1a33f746ac P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:37.890201 29371 leader_election.cc:290] T 5b46bb34ac214b1ba23fba1a33f746ac P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:37.891700 28567 raft_consensus.cc:513] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.892367 28567 leader_election.cc:290] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:37.891500 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b46bb34ac214b1ba23fba1a33f746ac" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:37.893989 27462 leader_election.cc:343] T 5b46bb34ac214b1ba23fba1a33f746ac P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:37.894567 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39c2ebe4ada5426590ec3769c8662644" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:37.894773 27511 raft_consensus.cc:2466] T 39c2ebe4ada5426590ec3769c8662644 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:37.895946 27320 leader_election.cc:304] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:37.896345 29329 raft_consensus.cc:2802] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:37.896445 29329 raft_consensus.cc:491] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:37.896550 29329 raft_consensus.cc:3058] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:37.902204 29329 raft_consensus.cc:513] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.904039 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39c2ebe4ada5426590ec3769c8662644" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:37.904265 27511 raft_consensus.cc:3058] T 39c2ebe4ada5426590ec3769c8662644 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:37.905030 29329 leader_election.cc:290] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:37.909030 27687 ts_tablet_manager.cc:1918] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:37.911378 27693 tablet_bootstrap.cc:492] T 259bc73ca9df47e9874692725a60bb1c P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:37.920644 27687 log.cc:1199] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/950a6fcd35f64f2d889831fe9cdf503a
I20250903 20:05:37.920907 27693 ts_tablet_manager.cc:1397] T 259bc73ca9df47e9874692725a60bb1c P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.218s user 0.010s sys 0.011s
I20250903 20:05:37.921200 27687 ts_tablet_manager.cc:1939] T 950a6fcd35f64f2d889831fe9cdf503a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:37.922067 27693 raft_consensus.cc:357] T 259bc73ca9df47e9874692725a60bb1c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.922515 27693 raft_consensus.cc:383] T 259bc73ca9df47e9874692725a60bb1c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:37.922660 27693 raft_consensus.cc:738] T 259bc73ca9df47e9874692725a60bb1c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:37.925303 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7824a0bccdc9485abde124572e502791" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:37.925742 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7824a0bccdc9485abde124572e502791" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:37.847638 27229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37071b15494b4c5ea450e34f7b93aafa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.931607 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 831d2bfa62e2439396fa53576201413d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.935097 27228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b801bf27b5a4e66812d127d1c5a47e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.936455 27210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34eadb8f690d41c0877eeeb00d4784a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.891340 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b46bb34ac214b1ba23fba1a33f746ac" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:37.893204 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39c2ebe4ada5426590ec3769c8662644" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:37.939633 27462 leader_election.cc:343] T 5b46bb34ac214b1ba23fba1a33f746ac P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:37.939766 27462 leader_election.cc:304] T 5b46bb34ac214b1ba23fba1a33f746ac P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
W20250903 20:05:37.939869 27321 leader_election.cc:343] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:37.880971 27685 ts_tablet_manager.cc:1918] T 8fff36d5a80e45af8a256c3908555e08 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:37.940176 29371 raft_consensus.cc:2747] T 5b46bb34ac214b1ba23fba1a33f746ac P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:37.940271 27685 log.cc:1199] T 8fff36d5a80e45af8a256c3908555e08 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/8fff36d5a80e45af8a256c3908555e08
I20250903 20:05:37.940641 27685 ts_tablet_manager.cc:1939] T 8fff36d5a80e45af8a256c3908555e08 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:37.904968 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39c2ebe4ada5426590ec3769c8662644" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
W20250903 20:05:37.941301 27321 leader_election.cc:343] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:37.899243 27212 tablet_service.cc:1515] Processing DeleteTablet for tablet 30cc0d4ba8d444f1b98232f906ea0c7b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:37.847560 27226 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e9a6a6e05464b7b97ae624f51402cf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:37.943377 27695 tablet_bootstrap.cc:492] T de20d18d7f264811a0631aa1144aef2c P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:37.945924 27695 ts_tablet_manager.cc:1397] T de20d18d7f264811a0631aa1144aef2c P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.377s user 0.008s sys 0.004s
I20250903 20:05:37.948664 29450 consensus_queue.cc:1035] T e2441aa3c2c748bba5360259fb663c81 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:37.949148 27693 consensus_queue.cc:260] T 259bc73ca9df47e9874692725a60bb1c P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.952003 27693 ts_tablet_manager.cc:1428] T 259bc73ca9df47e9874692725a60bb1c P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.031s user 0.002s sys 0.000s
I20250903 20:05:37.953909 27693 tablet_bootstrap.cc:492] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:37.956869 27066 master_service.cc:948] ReplaceTablet: received request to replace tablet e9b5ab312f8744af99d557c0733bf71d from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:37.959554 27063 master_service.cc:948] ReplaceTablet: received request to replace tablet 06a79706f1184fa49d10182ba669b0ab from {username='slave'} at 127.0.0.1:45936
W20250903 20:05:37.962715 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8e37674dd6f94feab26741033409f23c consensus state has no leader
I20250903 20:05:37.967355 27693 tablet_bootstrap.cc:654] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:37.971473 27693 tablet_bootstrap.cc:492] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:37.973054 27693 ts_tablet_manager.cc:1397] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.002s
I20250903 20:05:37.922094 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1130) took 3205 ms. Trace:
I20250903 20:05:37.983840 27687 rpcz_store.cc:276] 0903 20:05:34.716195 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.716259 (+ 64us) service_pool.cc:225] Handling call
0903 20:05:37.806780 (+3090521us) ts_tablet_manager.cc:1145] Deleting tablet 950a6fcd35f64f2d889831fe9cdf503a
0903 20:05:37.806793 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:37.856054 (+ 49261us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.908851 (+ 52797us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.922073 (+ 13222us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2927490}
I20250903 20:05:37.985103 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 950a6fcd35f64f2d889831fe9cdf503a (table default.Table_7dae8ff5ca9744708d223312079a4315 [id=7032fe3b789041d380f5db114f241e29]) successfully deleted
I20250903 20:05:37.986280 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1133) took 3220 ms. Trace:
I20250903 20:05:37.986407 27693 raft_consensus.cc:357] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:37.989116 27693 raft_consensus.cc:383] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:37.989257 27693 raft_consensus.cc:738] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:37.989707 27693 consensus_queue.cc:260] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:37.993101 27320 consensus_peers.cc:489] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 09fa32312c9648fbbd6fd327e9738066. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:37.993793 27693 ts_tablet_manager.cc:1428] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20250903 20:05:37.995342 27693 tablet_bootstrap.cc:492] T 4e9a6a6e05464b7b97ae624f51402cf0 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:37.988446 27687 rpcz_store.cc:276] 0903 20:05:34.765650 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.765716 (+ 66us) service_pool.cc:225] Handling call
0903 20:05:37.986057 (+3220341us) ts_tablet_manager.cc:1145] Deleting tablet 52ee6ba19a734329acde7608327f2a78
0903 20:05:37.986070 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:37.986247 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3096608}
W20250903 20:05:37.999779 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 52ee6ba19a734329acde7608327f2a78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52ee6ba19a734329acde7608327f2a78 already in progress: creating tablet
I20250903 20:05:38.000509 27687 tablet_replica.cc:331] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:38.001657 27687 raft_consensus.cc:2241] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:38.003290 27687 raft_consensus.cc:2270] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:38.004334 27053 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 8e37674dd6f94feab26741033409f23c (table default.Table_511ad95d56b5416c89bced96468410a4 [id=41adb50030d24515b6896b0159fcc1b2], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8e37674dd6f94feab26741033409f23c consensus state has no leader
I20250903 20:05:38.018471 27065 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6f835fbb9a1f4aff87000ff5e65133f8" } modify_external_catalogs: true
I20250903 20:05:38.018740 27065 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_6f835fbb9a1f4aff87000ff5e65133f8" } modify_external_catalogs: true
I20250903 20:05:38.022987 27687 ts_tablet_manager.cc:1905] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:38.023409 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e02c690220fb4ab2a949d23f2230eeca" } new_table_name: "default.Table_a31e595e9b9a41c7bc27ee7280cffa4c" modify_external_catalogs: true
I20250903 20:05:38.023298 27693 tablet_bootstrap.cc:654] T 4e9a6a6e05464b7b97ae624f51402cf0 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:38.030891 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3ef308651a284cc08daa6de000cc2522" } modify_external_catalogs: true
I20250903 20:05:38.031291 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3ef308651a284cc08daa6de000cc2522" } modify_external_catalogs: true
I20250903 20:05:38.032938 27511 raft_consensus.cc:2466] T 39c2ebe4ada5426590ec3769c8662644 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:38.033867 27320 leader_election.cc:304] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:38.033243 27692 tablet_bootstrap.cc:492] T 1bf570855b204b96acd070587c434463 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:38.034376 27692 ts_tablet_manager.cc:1397] T 1bf570855b204b96acd070587c434463 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.334s user 0.047s sys 0.062s
I20250903 20:05:38.034461 29329 raft_consensus.cc:2802] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:38.034595 29329 raft_consensus.cc:695] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:38.034837 29329 consensus_queue.cc:237] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.035204 27692 raft_consensus.cc:357] T 1bf570855b204b96acd070587c434463 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.035354 27692 raft_consensus.cc:383] T 1bf570855b204b96acd070587c434463 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:38.035406 27692 raft_consensus.cc:738] T 1bf570855b204b96acd070587c434463 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:38.035591 27692 consensus_queue.cc:260] T 1bf570855b204b96acd070587c434463 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.036116 27692 ts_tablet_manager.cc:1428] T 1bf570855b204b96acd070587c434463 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:38.036433 27692 tablet_bootstrap.cc:492] T 2385d3006f90486890ebe43e16a53d7a P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:38.037739 27064 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_8c666d5ff1be4e47b55808f90d025198" } modify_external_catalogs: true
I20250903 20:05:38.038122 27692 tablet_bootstrap.cc:654] T 2385d3006f90486890ebe43e16a53d7a P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:38.038039 27064 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_8c666d5ff1be4e47b55808f90d025198" } modify_external_catalogs: true
I20250903 20:05:38.038982 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_94f3e8f67f00459caaeb1430fa5203d5" } new_table_name: "default.Table_5307cd5405d348b0a2feeb026c556689" modify_external_catalogs: true
I20250903 20:05:38.042739 29329 consensus_queue.cc:1035] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:38.043007 27057 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 793d6157927143d4b9d8dbafd7cf0fcb (table default.Table_e79f52615fe142b9823a2a99b8b9c406 [id=3432f6ac076b4e49b9fe63bf5fce8591]). Expected version 1 got 0
I20250903 20:05:38.058589 27687 ts_tablet_manager.cc:1918] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:38.060340 27687 log.cc:1199] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/24eadc70e10d4d02bcb643d45c9e0e36
I20250903 20:05:38.059633 27693 tablet_bootstrap.cc:492] T 4e9a6a6e05464b7b97ae624f51402cf0 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:38.061285 27693 ts_tablet_manager.cc:1397] T 4e9a6a6e05464b7b97ae624f51402cf0 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.066s user 0.007s sys 0.001s
I20250903 20:05:38.061713 27687 ts_tablet_manager.cc:1939] T 24eadc70e10d4d02bcb643d45c9e0e36 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:38.062562 27693 raft_consensus.cc:357] T 4e9a6a6e05464b7b97ae624f51402cf0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.065877 27062 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 793d6157927143d4b9d8dbafd7cf0fcb (table default.Table_e79f52615fe142b9823a2a99b8b9c406 [id=3432f6ac076b4e49b9fe63bf5fce8591]). Expected version 1 got 0
I20250903 20:05:38.066205 27062 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet b4b15d77d07c4fa7856d9c198cf15450 (table default.Table_3edadd8954444273b91ea4d39cf3a834 [id=45cd523acd3c45b994d1b9cc6eb569ea]). Expected version 4 got 0
I20250903 20:05:38.066426 27062 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 5b46bb34ac214b1ba23fba1a33f746ac (table default.Table_975e53d37e054938a5b63957252ff50d [id=f261e84232234244824d0232efbb4399]). Expected version 1 got 0
I20250903 20:05:38.062887 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1134) took 3294 ms. Trace:
I20250903 20:05:38.066833 27693 raft_consensus.cc:383] T 4e9a6a6e05464b7b97ae624f51402cf0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:38.067548 29463 tablet.cc:1722] T 7a9e55f0811345f7afa71645f38ebdaf P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:38.066881 27687 rpcz_store.cc:276] 0903 20:05:34.768430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.768507 (+ 77us) service_pool.cc:225] Handling call
0903 20:05:38.000423 (+3231916us) ts_tablet_manager.cc:1145] Deleting tablet 24eadc70e10d4d02bcb643d45c9e0e36
0903 20:05:38.000443 (+ 20us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.044138 (+ 43695us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:38.058425 (+ 14287us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:38.062856 (+ 4431us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3110874}
I20250903 20:05:38.068327 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1136) took 3250 ms. Trace:
I20250903 20:05:38.068369 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 24eadc70e10d4d02bcb643d45c9e0e36 (table default.Table_76a88a96cd544cda882c413357a7f8c3 [id=e9a291cff3f4497cad9393ab4c571bf7]) successfully deleted
I20250903 20:05:38.068475 27687 rpcz_store.cc:276] 0903 20:05:34.817893 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.817956 (+ 63us) service_pool.cc:225] Handling call
0903 20:05:38.068170 (+3250214us) ts_tablet_manager.cc:1145] Deleting tablet 7203b4ba9047430ab782d4c0498466cc
0903 20:05:38.068185 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.068311 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3169808}
I20250903 20:05:38.068866 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1138) took 3228 ms. Trace:
I20250903 20:05:38.068944 27687 rpcz_store.cc:276] 0903 20:05:34.840518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.840611 (+ 93us) service_pool.cc:225] Handling call
0903 20:05:38.068754 (+3228143us) ts_tablet_manager.cc:1145] Deleting tablet baac5b97f7eb4d18a0f294ca1595a383
0903 20:05:38.068764 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.068850 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3167935}
W20250903 20:05:38.069106 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 7203b4ba9047430ab782d4c0498466cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7203b4ba9047430ab782d4c0498466cc already in progress: creating tablet
I20250903 20:05:38.069209 27687 tablet_replica.cc:331] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:38.069429 27687 raft_consensus.cc:2241] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
W20250903 20:05:38.069578 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet baac5b97f7eb4d18a0f294ca1595a383 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet baac5b97f7eb4d18a0f294ca1595a383 already in progress: creating tablet
I20250903 20:05:38.069779 27687 raft_consensus.cc:2270] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:38.068552 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 259bc73ca9df47e9874692725a60bb1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:38.069191 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e9a6a6e05464b7b97ae624f51402cf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:38.069789 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e34d6d5d68e4395aed78821ea077c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:38.070487 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 3919f7f95a954f59b68f56d0051c0109 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:19 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:38.070525 27687 ts_tablet_manager.cc:1905] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:38.072402 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_8a742d84a1dd47c1a154c8e4bd43dc79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:38.067888 27693 raft_consensus.cc:738] T 4e9a6a6e05464b7b97ae624f51402cf0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:38.073450 27693 consensus_queue.cc:260] T 4e9a6a6e05464b7b97ae624f51402cf0 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:38.073689 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a742d84a1dd47c1a154c8e4bd43dc79 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:38.074282 27693 ts_tablet_manager.cc:1428] T 4e9a6a6e05464b7b97ae624f51402cf0 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250903 20:05:38.080153 27692 tablet_bootstrap.cc:492] T 2385d3006f90486890ebe43e16a53d7a P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:38.080623 27693 tablet_bootstrap.cc:492] T 7824a0bccdc9485abde124572e502791 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:38.080367 27692 ts_tablet_manager.cc:1397] T 2385d3006f90486890ebe43e16a53d7a P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.000s
I20250903 20:05:38.081669 27692 raft_consensus.cc:357] T 2385d3006f90486890ebe43e16a53d7a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.081867 27692 raft_consensus.cc:383] T 2385d3006f90486890ebe43e16a53d7a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:38.081923 27692 raft_consensus.cc:738] T 2385d3006f90486890ebe43e16a53d7a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:38.082299 27692 consensus_queue.cc:260] T 2385d3006f90486890ebe43e16a53d7a P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.083084 27692 ts_tablet_manager.cc:1428] T 2385d3006f90486890ebe43e16a53d7a P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:38.083392 27692 tablet_bootstrap.cc:492] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:38.085819 27692 tablet_bootstrap.cc:654] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:38.085820 27693 tablet_bootstrap.cc:654] T 7824a0bccdc9485abde124572e502791 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:38.088577 27692 tablet_bootstrap.cc:492] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:38.088745 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bf570855b204b96acd070587c434463 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:38.088784 27692 ts_tablet_manager.cc:1397] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250903 20:05:38.089730 27692 raft_consensus.cc:357] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:38.089913 27692 raft_consensus.cc:383] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:38.089967 27692 raft_consensus.cc:738] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:38.090162 27692 consensus_queue.cc:260] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:38.090698 27692 ts_tablet_manager.cc:1428] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:38.091017 27692 tablet_bootstrap.cc:492] T a686ea49a29a4dd7b6e5dce5e1f9935a P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:38.095997 27693 tablet_bootstrap.cc:492] T 7824a0bccdc9485abde124572e502791 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:38.096153 27693 ts_tablet_manager.cc:1397] T 7824a0bccdc9485abde124572e502791 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20250903 20:05:38.096947 27693 raft_consensus.cc:357] T 7824a0bccdc9485abde124572e502791 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.097118 27693 raft_consensus.cc:383] T 7824a0bccdc9485abde124572e502791 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:38.097177 27693 raft_consensus.cc:738] T 7824a0bccdc9485abde124572e502791 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:38.097381 27693 consensus_queue.cc:260] T 7824a0bccdc9485abde124572e502791 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.097885 27693 ts_tablet_manager.cc:1428] T 7824a0bccdc9485abde124572e502791 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:38.097771 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_75162e2d436048639aa708c34fc68ec1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:38.098562 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75162e2d436048639aa708c34fc68ec1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:38.098824 27692 tablet_bootstrap.cc:654] T a686ea49a29a4dd7b6e5dce5e1f9935a P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:38.103049 27687 ts_tablet_manager.cc:1918] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:38.103188 27687 log.cc:1199] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/238f2f30377b42c2852b016018aa3245
I20250903 20:05:38.103592 27687 ts_tablet_manager.cc:1939] T 238f2f30377b42c2852b016018aa3245 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:38.105500 27693 tablet_bootstrap.cc:492] T 3f303233e6c347289650c6cf83f8127f P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:38.107297 27693 tablet_bootstrap.cc:654] T 3f303233e6c347289650c6cf83f8127f P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:38.117887 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet a7c8657489734199b346033bd4afb02e
I20250903 20:05:38.118572 27692 tablet_bootstrap.cc:492] T a686ea49a29a4dd7b6e5dce5e1f9935a P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:38.118714 27692 ts_tablet_manager.cc:1397] T a686ea49a29a4dd7b6e5dce5e1f9935a P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.004s
I20250903 20:05:38.119447 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet a7c8657489734199b346033bd4afb02e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:38.119517 27692 raft_consensus.cc:357] T a686ea49a29a4dd7b6e5dce5e1f9935a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.119681 27692 raft_consensus.cc:383] T a686ea49a29a4dd7b6e5dce5e1f9935a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:38.119740 27692 raft_consensus.cc:738] T a686ea49a29a4dd7b6e5dce5e1f9935a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:38.119943 27692 consensus_queue.cc:260] T a686ea49a29a4dd7b6e5dce5e1f9935a P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.120497 27692 ts_tablet_manager.cc:1428] T a686ea49a29a4dd7b6e5dce5e1f9935a P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:38.120782 27692 tablet_bootstrap.cc:492] T 249362a0b8304719863bcb82c47273b3 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:38.122323 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1140) took 3193 ms. Trace:
I20250903 20:05:38.123274 27687 rpcz_store.cc:276] 0903 20:05:34.928696 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.931461 (+ 2765us) service_pool.cc:225] Handling call
0903 20:05:38.069163 (+3137702us) ts_tablet_manager.cc:1145] Deleting tablet 238f2f30377b42c2852b016018aa3245
0903 20:05:38.069172 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.077568 (+ 8396us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:38.102930 (+ 25362us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:38.121627 (+ 18697us) spinlock_profiling.cc:243] Waited 17.6 ms on lock 0x61800000d518. stack: 00007efcbf8c4dc6 00007efcbf8c4b80 00007efcbe8ac3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007efcbf8ad6cd 00007efcbf8ae73e 00007efcd2f768ed 00007efcd2ef81d3 00007efccf144723 00007efccf176344 00007efccf157da0 00007efccf157b2d 0000000000a774dc 00007efcbf90b14f
0903 20:05:38.122288 (+ 661us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":40429440,"tablet-delete.queue_time_us":3133053}
I20250903 20:05:38.123482 27693 tablet_bootstrap.cc:492] T 3f303233e6c347289650c6cf83f8127f P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:38.124012 27693 ts_tablet_manager.cc:1397] T 3f303233e6c347289650c6cf83f8127f P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.002s
I20250903 20:05:38.124680 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 238f2f30377b42c2852b016018aa3245 (table default.Table_a22a6e16b90a461ba0c890fb7c5791e9 [id=8fe9f7413b124e39bd3c5645312415e0]) successfully deleted
I20250903 20:05:38.124851 27693 raft_consensus.cc:357] T 3f303233e6c347289650c6cf83f8127f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.125123 27693 raft_consensus.cc:383] T 3f303233e6c347289650c6cf83f8127f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:38.125237 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1149) took 2998 ms. Trace:
I20250903 20:05:38.125303 27693 raft_consensus.cc:738] T 3f303233e6c347289650c6cf83f8127f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:38.125380 27687 rpcz_store.cc:276] 0903 20:05:35.127075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.130047 (+ 2972us) service_pool.cc:225] Handling call
0903 20:05:38.125071 (+2995024us) ts_tablet_manager.cc:1145] Deleting tablet 20d826f1b1d64746bd8e409a12b74eb1
0903 20:05:38.125081 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.125218 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2994794}
I20250903 20:05:38.125737 27693 consensus_queue.cc:260] T 3f303233e6c347289650c6cf83f8127f P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:38.125949 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 20d826f1b1d64746bd8e409a12b74eb1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20d826f1b1d64746bd8e409a12b74eb1 already in progress: creating tablet
I20250903 20:05:38.127738 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet a7c8657489734199b346033bd4afb02e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:38.126519 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1151) took 2966 ms. Trace:
I20250903 20:05:38.126924 27693 ts_tablet_manager.cc:1428] T 3f303233e6c347289650c6cf83f8127f P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:38.129416 27687 rpcz_store.cc:276] 0903 20:05:35.159810 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.161067 (+ 1257us) service_pool.cc:225] Handling call
0903 20:05:38.126378 (+2965311us) ts_tablet_manager.cc:1145] Deleting tablet f28fd3e1e0e14cf48a244580b4f355d0
0903 20:05:38.126393 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.126501 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2965093}
I20250903 20:05:38.129825 27693 tablet_bootstrap.cc:492] T 64c4dd2645034a7eaf243f5474603f10 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
W20250903 20:05:38.130193 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet f28fd3e1e0e14cf48a244580b4f355d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f28fd3e1e0e14cf48a244580b4f355d0 already in progress: creating tablet
I20250903 20:05:38.130792 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1152) took 2922 ms. Trace:
I20250903 20:05:38.132730 27692 tablet_bootstrap.cc:654] T 249362a0b8304719863bcb82c47273b3 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:38.132139 27693 tablet_bootstrap.cc:654] T 64c4dd2645034a7eaf243f5474603f10 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:38.135100 27687 rpcz_store.cc:276] 0903 20:05:35.208557 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.220653 (+ 12096us) service_pool.cc:225] Handling call
0903 20:05:38.130670 (+2910017us) ts_tablet_manager.cc:1145] Deleting tablet 69f8eef52bee431a86f0eef45fd11f9f
0903 20:05:38.130680 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.130772 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2909701}
W20250903 20:05:38.135730 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 69f8eef52bee431a86f0eef45fd11f9f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69f8eef52bee431a86f0eef45fd11f9f already in progress: creating tablet
I20250903 20:05:38.135910 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 06a79706f1184fa49d10182ba669b0ab
I20250903 20:05:38.136415 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1154) took 2816 ms. Trace:
I20250903 20:05:38.137828 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 06a79706f1184fa49d10182ba669b0ab with delete_type TABLET_DATA_DELETED (replaced by tablet 97bc32f5b49a4bcfb8f5702da4a8e722) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:38.137040 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet 06a79706f1184fa49d10182ba669b0ab with delete_type TABLET_DATA_DELETED (replaced by tablet 97bc32f5b49a4bcfb8f5702da4a8e722) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:38.138069 27063 catalog_manager.cc:6652] ReplaceTablet: tablet 06a79706f1184fa49d10182ba669b0ab deleted and replaced by tablet 97bc32f5b49a4bcfb8f5702da4a8e722
I20250903 20:05:38.138252 27687 rpcz_store.cc:276] 0903 20:05:35.319798 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.335333 (+ 15535us) service_pool.cc:225] Handling call
0903 20:05:38.136274 (+2800941us) ts_tablet_manager.cc:1145] Deleting tablet eb878263eda9447e870a782907fe6a8d
0903 20:05:38.136287 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.136395 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2800506}
W20250903 20:05:38.138952 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet eb878263eda9447e870a782907fe6a8d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb878263eda9447e870a782907fe6a8d already in progress: creating tablet
W20250903 20:05:38.140966 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2bf6c73765534237ba3f1ccb751192b2 consensus state has no leader
I20250903 20:05:38.142985 27692 tablet_bootstrap.cc:492] T 249362a0b8304719863bcb82c47273b3 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:38.143157 27692 ts_tablet_manager.cc:1397] T 249362a0b8304719863bcb82c47273b3 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20250903 20:05:38.143985 27692 raft_consensus.cc:357] T 249362a0b8304719863bcb82c47273b3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.144171 27692 raft_consensus.cc:383] T 249362a0b8304719863bcb82c47273b3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:38.144248 27692 raft_consensus.cc:738] T 249362a0b8304719863bcb82c47273b3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:38.144465 27692 consensus_queue.cc:260] T 249362a0b8304719863bcb82c47273b3 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.147095 27693 tablet_bootstrap.cc:492] T 64c4dd2645034a7eaf243f5474603f10 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:38.147298 27693 ts_tablet_manager.cc:1397] T 64c4dd2645034a7eaf243f5474603f10 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.002s
I20250903 20:05:38.148309 27693 raft_consensus.cc:357] T 64c4dd2645034a7eaf243f5474603f10 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.148507 27693 raft_consensus.cc:383] T 64c4dd2645034a7eaf243f5474603f10 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:38.148592 27693 raft_consensus.cc:738] T 64c4dd2645034a7eaf243f5474603f10 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:38.148804 27693 consensus_queue.cc:260] T 64c4dd2645034a7eaf243f5474603f10 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.150775 27692 ts_tablet_manager.cc:1428] T 249362a0b8304719863bcb82c47273b3 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.008s user 0.002s sys 0.001s
I20250903 20:05:38.151197 27692 tablet_bootstrap.cc:492] T 54f1ce5192f547f595dea23a9eabf8e6 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:38.151532 27687 tablet_replica.cc:331] T eaa11d17303142ff935705c16f0b9690 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:38.151722 27687 raft_consensus.cc:2241] T eaa11d17303142ff935705c16f0b9690 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:38.151882 27687 raft_consensus.cc:2270] T eaa11d17303142ff935705c16f0b9690 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:38.153064 27687 ts_tablet_manager.cc:1905] T eaa11d17303142ff935705c16f0b9690 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:38.154026 27693 ts_tablet_manager.cc:1428] T 64c4dd2645034a7eaf243f5474603f10 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250903 20:05:38.155194 27693 tablet_bootstrap.cc:492] T b4b15d77d07c4fa7856d9c198cf15450 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:38.157778 27693 tablet_bootstrap.cc:654] T b4b15d77d07c4fa7856d9c198cf15450 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:38.158433 29371 raft_consensus.cc:491] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:38.158725 27692 tablet_bootstrap.cc:654] T 54f1ce5192f547f595dea23a9eabf8e6 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:38.158610 29371 raft_consensus.cc:513] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.159559 29371 leader_election.cc:290] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:38.160638 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e9a6a6e05464b7b97ae624f51402cf0" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:38.162034 27380 raft_consensus.cc:2466] T 4e9a6a6e05464b7b97ae624f51402cf0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:38.162798 27462 leader_election.cc:304] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:38.163195 29371 raft_consensus.cc:2802] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:38.163270 29371 raft_consensus.cc:491] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:38.163331 29371 raft_consensus.cc:3058] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:38.166723 29371 raft_consensus.cc:513] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.167435 29371 leader_election.cc:290] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:38.167817 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e9a6a6e05464b7b97ae624f51402cf0" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:38.167907 27692 tablet_bootstrap.cc:492] T 54f1ce5192f547f595dea23a9eabf8e6 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:38.168015 27380 raft_consensus.cc:3058] T 4e9a6a6e05464b7b97ae624f51402cf0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:38.168038 27692 ts_tablet_manager.cc:1397] T 54f1ce5192f547f595dea23a9eabf8e6 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20250903 20:05:38.168841 27692 raft_consensus.cc:357] T 54f1ce5192f547f595dea23a9eabf8e6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.169003 27692 raft_consensus.cc:383] T 54f1ce5192f547f595dea23a9eabf8e6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:38.169051 27692 raft_consensus.cc:738] T 54f1ce5192f547f595dea23a9eabf8e6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:38.169255 27692 consensus_queue.cc:260] T 54f1ce5192f547f595dea23a9eabf8e6 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.169750 27692 ts_tablet_manager.cc:1428] T 54f1ce5192f547f595dea23a9eabf8e6 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:38.170056 27692 tablet_bootstrap.cc:492] T f9375fff5f5945078c4412aa21c06f0d P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:38.171871 27380 raft_consensus.cc:2466] T 4e9a6a6e05464b7b97ae624f51402cf0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:38.172463 27462 leader_election.cc:304] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:38.172816 29437 raft_consensus.cc:2802] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:38.172935 29437 raft_consensus.cc:695] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:38.173182 29437 consensus_queue.cc:237] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.178189 28567 consensus_queue.cc:1035] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250903 20:05:38.181627 29170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20250903 20:05:38.186625 27693 tablet_bootstrap.cc:492] T b4b15d77d07c4fa7856d9c198cf15450 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:38.186826 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet ed096b452ba747f89e91edc15e6e3ac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:38.186983 27693 ts_tablet_manager.cc:1397] T b4b15d77d07c4fa7856d9c198cf15450 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.003s
I20250903 20:05:38.188172 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 24eadc70e10d4d02bcb643d45c9e0e36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:38.188166 27693 raft_consensus.cc:357] T b4b15d77d07c4fa7856d9c198cf15450 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.188716 27693 raft_consensus.cc:383] T b4b15d77d07c4fa7856d9c198cf15450 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:38.188856 27693 raft_consensus.cc:738] T b4b15d77d07c4fa7856d9c198cf15450 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:38.189109 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 39c2ebe4ada5426590ec3769c8662644 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c772ab7c50c4994aacde5a05bb0f08e) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:38.189344 28567 raft_consensus.cc:491] T 259bc73ca9df47e9874692725a60bb1c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:38.189469 28567 raft_consensus.cc:513] T 259bc73ca9df47e9874692725a60bb1c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.190045 28567 leader_election.cc:290] T 259bc73ca9df47e9874692725a60bb1c P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:38.190455 29170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250903 20:05:38.190657 29170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250903 20:05:38.191045 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 30cc0d4ba8d444f1b98232f906ea0c7b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:38.190750 27692 tablet_bootstrap.cc:654] T f9375fff5f5945078c4412aa21c06f0d P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:38.191522 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 238f2f30377b42c2852b016018aa3245 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:38.192399 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 950a6fcd35f64f2d889831fe9cdf503a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:38.193012 27057 master_service.cc:948] ReplaceTablet: received request to replace tablet 3f08c7edbcf54c16bc222f59773f72b1 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:38.193895 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "259bc73ca9df47e9874692725a60bb1c" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:38.194234 27516 raft_consensus.cc:2391] T 259bc73ca9df47e9874692725a60bb1c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 51578fb7bceb4e119b3ab1e58c4f5528 in current term 1: Already voted for candidate 89f8f0ee68b940db9ce250c0604f934a in this term.
I20250903 20:05:38.201908 29329 consensus_queue.cc:1035] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:38.202680 27349 tablet_service.cc:1468] Processing CreateTablet for tablet 8a910f8157df4308b5720e66f2a78347 (DEFAULT_TABLE table=default.Table_bbe7d51c5b5140b68deb8ad68216a05e [id=1bf009a82e1a4c11b751b53dcf5cef91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:38.203125 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a910f8157df4308b5720e66f2a78347. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:38.221800 27575 tablet.cc:2378] T 1bf570855b204b96acd070587c434463 P e72ef1dbcc5143cf9f04a808f101c80f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250903 20:05:38.228988 29170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250903 20:05:38.232390 29170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250903 20:05:38.232643 29170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250903 20:05:38.232820 29170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250903 20:05:38.235057 29463 tablet.cc:1722] T 6238740ce1894e899be9a40321f0ab73 P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250903 20:05:38.244598 27692 tablet_bootstrap.cc:492] T f9375fff5f5945078c4412aa21c06f0d P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:38.244784 27692 ts_tablet_manager.cc:1397] T f9375fff5f5945078c4412aa21c06f0d P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.075s user 0.003s sys 0.003s
I20250903 20:05:38.245679 27692 raft_consensus.cc:357] T f9375fff5f5945078c4412aa21c06f0d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.245867 27692 raft_consensus.cc:383] T f9375fff5f5945078c4412aa21c06f0d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:38.245916 27692 raft_consensus.cc:738] T f9375fff5f5945078c4412aa21c06f0d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:38.249543 29490 tablet.cc:1722] T 6238740ce1894e899be9a40321f0ab73 P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250903 20:05:38.249985 27692 consensus_queue.cc:260] T f9375fff5f5945078c4412aa21c06f0d P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.251191 27692 ts_tablet_manager.cc:1428] T f9375fff5f5945078c4412aa21c06f0d P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250903 20:05:38.253444 27692 tablet_bootstrap.cc:492] T f4dec89fc6bb436ba6542c267eaf9c15 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:38.249939 27687 ts_tablet_manager.cc:1918] T eaa11d17303142ff935705c16f0b9690 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:38.256449 27687 log.cc:1199] T eaa11d17303142ff935705c16f0b9690 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/eaa11d17303142ff935705c16f0b9690
I20250903 20:05:38.257207 27687 ts_tablet_manager.cc:1939] T eaa11d17303142ff935705c16f0b9690 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:38.258116 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 8a910f8157df4308b5720e66f2a78347 (DEFAULT_TABLE table=default.Table_bbe7d51c5b5140b68deb8ad68216a05e [id=1bf009a82e1a4c11b751b53dcf5cef91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:38.258637 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a910f8157df4308b5720e66f2a78347. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:38.261554 27692 tablet_bootstrap.cc:654] T f4dec89fc6bb436ba6542c267eaf9c15 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:38.266511 29170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.65.84:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.26.65.126:39015,127.26.65.125:44059,127.26.65.124:40911
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.65.125:44059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.26.65.125
--webserver_port=41613
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 03 Sep 2025 19:52:33 UTC on 24a791456cd2
build id 7884
ASAN enabled
I20250903 20:05:37.922255 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet d4c703dafbca4e669dbb18946ffac699 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:37.925962 29412 raft_consensus.cc:491] T 30cc0d4ba8d444f1b98232f906ea0c7b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:38.267323 29412 raft_consensus.cc:513] T 30cc0d4ba8d444f1b98232f906ea0c7b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.267968 29412 leader_election.cc:290] T 30cc0d4ba8d444f1b98232f906ea0c7b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:37.922792 27216 tablet_service.cc:1468] Processing CreateTablet for tablet a655abd16c9c4f50b86fb9c3d12b3805 (DEFAULT_TABLE table=default.Table_ecfa891cdd9c4b52a981ca48c937bf64 [id=b991811dc93547cdab83ffaeda7c51b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:38.268416 27216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a655abd16c9c4f50b86fb9c3d12b3805. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.846184 27217 tablet_service.cc:1468] Processing CreateTablet for tablet ccb671d7e67444ac8e5b78055271d02a (DEFAULT_TABLE table=default.Table_d0aec832c86d4e868563056c6100c6b8 [id=8d69d1671a82411da646bc38ae0a5109]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:37.927654 29196 consensus_queue.cc:1035] T f326ec776ba446be8d5dda5c1dc5b6ec P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.035s
W20250903 20:05:37.927481 27184 leader_election.cc:343] T 4e9a6a6e05464b7b97ae624f51402cf0 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:38.270951 27184 leader_election.cc:304] T 4e9a6a6e05464b7b97ae624f51402cf0 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:38.271225 29411 raft_consensus.cc:2747] T 4e9a6a6e05464b7b97ae624f51402cf0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:38.271263 27184 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 343796us
I20250903 20:05:38.271265 27349 tablet_service.cc:1468] Processing CreateTablet for tablet 4b256c6340bf453180048ed621c3df8c (DEFAULT_TABLE table=default.Table_48ce0908f41148948ef0a0d8907a8b45 [id=9dec72d284574ec5ad258f480839d0e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:38.271513 27184 leader_election.cc:304] T ee40566e497740789575275d0cdca0db P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:38.271721 27365 tablet_service.cc:1468] Processing CreateTablet for tablet f5269c446aff43c5bcd4ad928c765585 (DEFAULT_TABLE table=default.Table_dec07ab3580e47e1a0be5bb8eb0dd841 [id=ec221a6c8d4448dca0c49887b58797f6]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:38.271906 27184 consensus_peers.cc:489] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1d22e76ea72441db51f7e9db26eb520. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:38.272028 29411 raft_consensus.cc:2747] T ee40566e497740789575275d0cdca0db P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20250903 20:05:38.272099 27184 leader_election.cc:343] T 64c4dd2645034a7eaf243f5474603f10 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:38.272382 27184 consensus_peers.cc:489] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 548b50819f694fa3ad508da9f442a99c. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:38.272320 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 4b256c6340bf453180048ed621c3df8c (DEFAULT_TABLE table=default.Table_48ce0908f41148948ef0a0d8907a8b45 [id=9dec72d284574ec5ad258f480839d0e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:38.272505 27504 tablet_service.cc:1468] Processing CreateTablet for tablet f5269c446aff43c5bcd4ad928c765585 (DEFAULT_TABLE table=default.Table_dec07ab3580e47e1a0be5bb8eb0dd841 [id=ec221a6c8d4448dca0c49887b58797f6]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:38.272714 27184 leader_election.cc:343] T 7824a0bccdc9485abde124572e502791 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:38.272881 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b256c6340bf453180048ed621c3df8c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:38.272993 27184 leader_election.cc:343] T 7824a0bccdc9485abde124572e502791 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:38.273126 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5269c446aff43c5bcd4ad928c765585. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:37.964166 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1001) took 5124 ms. Trace:
I20250903 20:05:38.274273 27685 rpcz_store.cc:276] 0903 20:05:32.839545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.840133 (+ 588us) service_pool.cc:225] Handling call
0903 20:05:37.636980 (+4796847us) ts_tablet_manager.cc:1145] Deleting tablet 8fff36d5a80e45af8a256c3908555e08
0903 20:05:37.636988 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:37.658205 (+ 21217us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.880763 (+222558us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:37.964143 (+ 83380us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4796675}
I20250903 20:05:38.274667 27685 tablet_replica.cc:331] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:38.276424 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f303233e6c347289650c6cf83f8127f" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:38.276713 27380 raft_consensus.cc:2466] T 3f303233e6c347289650c6cf83f8127f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:38.002643 29456 raft_consensus.cc:491] T e877d56bbbfa4c5a92d025d0311ce5c6 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51578fb7bceb4e119b3ab1e58c4f5528)
I20250903 20:05:38.276999 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet a0f79831866b40c49c5b27ce2014c3f6 (table default.Table_3d20da97fa86497e9376f8aa69f1c3fd [id=35b0c6c60b9b43b5ad6c376284f45da5]) successfully deleted
I20250903 20:05:38.277033 29456 raft_consensus.cc:513] T e877d56bbbfa4c5a92d025d0311ce5c6 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:38.277297 27052 catalog_manager.cc:4895] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 1bcb9d5f010f4297bab78c0183889238 because the tablet was not found. No further retry: Not found: Tablet not found: 1bcb9d5f010f4297bab78c0183889238
I20250903 20:05:38.277911 29456 leader_election.cc:290] T e877d56bbbfa4c5a92d025d0311ce5c6 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:38.278342 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "259bc73ca9df47e9874692725a60bb1c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:38.278613 27381 raft_consensus.cc:2466] T 259bc73ca9df47e9874692725a60bb1c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:38.278962 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "259bc73ca9df47e9874692725a60bb1c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:38.279145 27385 raft_consensus.cc:3058] T 259bc73ca9df47e9874692725a60bb1c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:38.282454 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f303233e6c347289650c6cf83f8127f" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:38.282716 27383 raft_consensus.cc:3058] T 3f303233e6c347289650c6cf83f8127f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:38.049942 29462 tablet.cc:1722] T 7a9e55f0811345f7afa71645f38ebdaf P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:38.190725 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "259bc73ca9df47e9874692725a60bb1c" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:38.234930 29457 raft_consensus.cc:491] T d4c703dafbca4e669dbb18946ffac699 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:38.286798 29457 raft_consensus.cc:513] T d4c703dafbca4e669dbb18946ffac699 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.287660 29457 leader_election.cc:290] T d4c703dafbca4e669dbb18946ffac699 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:38.236900 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e9a6a6e05464b7b97ae624f51402cf0" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:38.288089 29412 raft_consensus.cc:491] T bf459db3327640f88574fca7b5f2ca98 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51578fb7bceb4e119b3ab1e58c4f5528)
I20250903 20:05:38.288246 27243 raft_consensus.cc:2466] T 4e9a6a6e05464b7b97ae624f51402cf0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:38.288226 29412 raft_consensus.cc:513] T bf459db3327640f88574fca7b5f2ca98 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.236974 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e9a6a6e05464b7b97ae624f51402cf0" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:38.288515 27249 raft_consensus.cc:3058] T 4e9a6a6e05464b7b97ae624f51402cf0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:38.288734 29412 leader_election.cc:290] T bf459db3327640f88574fca7b5f2ca98 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:38.237301 29461 consensus_queue.cc:1035] T 64c4dd2645034a7eaf243f5474603f10 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250903 20:05:38.289819 27051 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.65.125:44059: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20250903 20:05:38.290426 27321 leader_election.cc:304] T 259bc73ca9df47e9874692725a60bb1c P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:38.290774 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet 259bc73ca9df47e9874692725a60bb1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:38.291221 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet 503dba32175c4459892c62d3f80693fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:37 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:38.291425 27051 consensus_peers.cc:489] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979 -> Peer 35dd69f3fa7a4fd8843b97f8a5d68296 (127.26.65.125:44059): Couldn't send request to peer 35dd69f3fa7a4fd8843b97f8a5d68296. Status: Network error: Client connection negotiation failed: client connection to 127.26.65.125:44059: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250903 20:05:38.292151 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet e4ec4cc1ef914cffa2d304cdf701cf33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:37 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:38.292392 27384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30cc0d4ba8d444f1b98232f906ea0c7b" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:38.292575 27384 raft_consensus.cc:2466] T 30cc0d4ba8d444f1b98232f906ea0c7b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:38.294608 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet 74eab6ad2fe64b18bd00c0d5524428c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:37 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:38.295089 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet d7d37849132d4420bb5fcb57c11cfb6b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:37 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:38.295508 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet e9b5ab312f8744af99d557c0733bf71d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:37 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:38.301640 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet a7c8657489734199b346033bd4afb02e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:38.302636 27249 raft_consensus.cc:2466] T 4e9a6a6e05464b7b97ae624f51402cf0 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
W20250903 20:05:38.304008 27183 leader_election.cc:343] T e877d56bbbfa4c5a92d025d0311ce5c6 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: e877d56bbbfa4c5a92d025d0311ce5c6
I20250903 20:05:38.308056 27510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4c703dafbca4e669dbb18946ffac699" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:38.308382 27510 raft_consensus.cc:2466] T d4c703dafbca4e669dbb18946ffac699 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:38.311153 27222 tablet_service.cc:1515] Processing DeleteTablet for tablet 06a79706f1184fa49d10182ba669b0ab with delete_type TABLET_DATA_DELETED (replaced by tablet 97bc32f5b49a4bcfb8f5702da4a8e722) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:38.314730 27222 tablet_service.cc:1468] Processing CreateTablet for tablet 8a910f8157df4308b5720e66f2a78347 (DEFAULT_TABLE table=default.Table_bbe7d51c5b5140b68deb8ad68216a05e [id=1bf009a82e1a4c11b751b53dcf5cef91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:38.316968 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet a868191630454a1eb3a4301664178a9a
I20250903 20:05:38.317503 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 8fff36d5a80e45af8a256c3908555e08 (table default.Table_644507bb34724729854f122d8191cf82 [id=5616ab4744594f75b25cc4b7db7c3797]) successfully deleted
I20250903 20:05:38.317826 29371 raft_consensus.cc:491] T a686ea49a29a4dd7b6e5dce5e1f9935a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:38.318033 29371 raft_consensus.cc:513] T a686ea49a29a4dd7b6e5dce5e1f9935a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.318334 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet a868191630454a1eb3a4301664178a9a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:38.318740 29371 leader_election.cc:290] T a686ea49a29a4dd7b6e5dce5e1f9935a P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:38.318936 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet a868191630454a1eb3a4301664178a9a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:38.319708 27385 raft_consensus.cc:2466] T 259bc73ca9df47e9874692725a60bb1c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:38.319917 29329 raft_consensus.cc:2747] T 259bc73ca9df47e9874692725a60bb1c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:38.321030 27383 raft_consensus.cc:2466] T 3f303233e6c347289650c6cf83f8127f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:38.322872 27224 tablet_service.cc:1468] Processing CreateTablet for tablet 4b256c6340bf453180048ed621c3df8c (DEFAULT_TABLE table=default.Table_48ce0908f41148948ef0a0d8907a8b45 [id=9dec72d284574ec5ad258f480839d0e9]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:38.323617 27183 leader_election.cc:343] T bf459db3327640f88574fca7b5f2ca98 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: bf459db3327640f88574fca7b5f2ca98
I20250903 20:05:38.323899 27183 leader_election.cc:304] T d4c703dafbca4e669dbb18946ffac699 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:38.324229 29452 raft_consensus.cc:2802] T d4c703dafbca4e669dbb18946ffac699 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:38.324338 29452 raft_consensus.cc:491] T d4c703dafbca4e669dbb18946ffac699 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:38.324402 29452 raft_consensus.cc:3058] T d4c703dafbca4e669dbb18946ffac699 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:38.327558 29452 raft_consensus.cc:513] T d4c703dafbca4e669dbb18946ffac699 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.328331 29452 leader_election.cc:290] T d4c703dafbca4e669dbb18946ffac699 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:38.330119 27509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4c703dafbca4e669dbb18946ffac699" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:38.330390 27509 raft_consensus.cc:3058] T d4c703dafbca4e669dbb18946ffac699 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:38.331447 29463 tablet.cc:1722] T fc634116a78f45dd833034ddda90e9bf P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:38.333586 27509 raft_consensus.cc:2466] T d4c703dafbca4e669dbb18946ffac699 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:37.947665 27695 raft_consensus.cc:357] T de20d18d7f264811a0631aa1144aef2c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.334244 27380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4c703dafbca4e669dbb18946ffac699" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:38.334791 27695 raft_consensus.cc:383] T de20d18d7f264811a0631aa1144aef2c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:38.335422 27383 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a686ea49a29a4dd7b6e5dce5e1f9935a" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:38.335800 27695 raft_consensus.cc:738] T de20d18d7f264811a0631aa1144aef2c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
W20250903 20:05:38.330562 27185 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.26.65.126:39015, user_credentials={real_user=slave}} blocked reactor thread for 93569us
W20250903 20:05:38.338306 27462 leader_election.cc:343] T a686ea49a29a4dd7b6e5dce5e1f9935a P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:38.343109 27695 consensus_queue.cc:260] T de20d18d7f264811a0631aa1144aef2c P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.344923 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e79f52615fe142b9823a2a99b8b9c406" } modify_external_catalogs: true
I20250903 20:05:38.345240 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e79f52615fe142b9823a2a99b8b9c406" } modify_external_catalogs: true
I20250903 20:05:38.350528 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_edc3f0ba095c4b65b0bb1f12d3f231c5" } new_table_name: "default.Table_aa093df048e642abb72d99af3918e182" modify_external_catalogs: true
I20250903 20:05:38.351068 29491 consensus_queue.cc:1035] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:38.352720 27064 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2e2dd17c15234bb3836b57cd2989c12f" } modify_external_catalogs: true
I20250903 20:05:38.352931 27064 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2e2dd17c15234bb3836b57cd2989c12f" } modify_external_catalogs: true
I20250903 20:05:38.353451 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30cc0d4ba8d444f1b98232f906ea0c7b" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:38.354609 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_c19f2a545c9e49848bfa6313fa9a9543"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:38.354782 27511 raft_consensus.cc:2466] T 30cc0d4ba8d444f1b98232f906ea0c7b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
W20250903 20:05:38.355381 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c19f2a545c9e49848bfa6313fa9a9543 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:38.357041 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_3f0e8bdd369f4e8c89ae8ee2ef5dd22f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:38.357698 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f0e8bdd369f4e8c89ae8ee2ef5dd22f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:38.360742 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f6d8f4f4883c410c82f937ebeae0141c" } new_table_name: "default.Table_a36f8bf4d36e4ac29b4e52dea10e8a32" modify_external_catalogs: true
I20250903 20:05:38.362397 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b7cf7b55414f48c7925459bc4b5b8f8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:38.363157 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7cf7b55414f48c7925459bc4b5b8f8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:38.189257 27693 consensus_queue.cc:260] T b4b15d77d07c4fa7856d9c198cf15450 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.390422 29487 raft_consensus.cc:491] T b4b15d77d07c4fa7856d9c198cf15450 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:38.390547 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet ed73c335b042476fa03f154d23a611ad
I20250903 20:05:38.390822 29487 raft_consensus.cc:513] T b4b15d77d07c4fa7856d9c198cf15450 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.391391 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet ed73c335b042476fa03f154d23a611ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:38.392608 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet ed73c335b042476fa03f154d23a611ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:38.392731 29487 leader_election.cc:290] T b4b15d77d07c4fa7856d9c198cf15450 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:38.395610 27510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b4b15d77d07c4fa7856d9c198cf15450" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:38.395970 27510 raft_consensus.cc:2466] T b4b15d77d07c4fa7856d9c198cf15450 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:38.396535 27320 leader_election.cc:304] T b4b15d77d07c4fa7856d9c198cf15450 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:38.396040 29170 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 20:05:38.400940 27693 ts_tablet_manager.cc:1428] T b4b15d77d07c4fa7856d9c198cf15450 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.214s user 0.002s sys 0.000s
W20250903 20:05:38.346038 27184 consensus_peers.cc:489] T e2441aa3c2c748bba5360259fb663c81 P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250903 20:05:38.403404 27184 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 57438.7us
I20250903 20:05:38.403930 27695 ts_tablet_manager.cc:1428] T de20d18d7f264811a0631aa1144aef2c P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.458s user 0.001s sys 0.001s
I20250903 20:05:38.405094 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_47fab307e9c2482a8bf34af44051f49c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:38.405731 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47fab307e9c2482a8bf34af44051f49c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:38.392536 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b4b15d77d07c4fa7856d9c198cf15450" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:38.397441 29503 tablet.cc:1722] T 6238740ce1894e899be9a40321f0ab73 P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250903 20:05:38.402750 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a686ea49a29a4dd7b6e5dce5e1f9935a" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:38.404691 27184 leader_election.cc:304] T 30cc0d4ba8d444f1b98232f906ea0c7b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:38.354462 27183 leader_election.cc:304] T d4c703dafbca4e669dbb18946ffac699 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:38.412232 27695 tablet_bootstrap.cc:492] T 2385d3006f90486890ebe43e16a53d7a P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:38.405032 29492 tablet.cc:1722] T fc634116a78f45dd833034ddda90e9bf P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250903 20:05:38.412393 27183 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 58223us
W20250903 20:05:38.412431 27462 leader_election.cc:343] T a686ea49a29a4dd7b6e5dce5e1f9935a P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:38.412561 27462 leader_election.cc:304] T a686ea49a29a4dd7b6e5dce5e1f9935a P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:38.412909 29456 raft_consensus.cc:2802] T 30cc0d4ba8d444f1b98232f906ea0c7b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:38.412986 29456 raft_consensus.cc:491] T 30cc0d4ba8d444f1b98232f906ea0c7b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:38.412958 29437 raft_consensus.cc:2747] T a686ea49a29a4dd7b6e5dce5e1f9935a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:38.413048 29456 raft_consensus.cc:3058] T 30cc0d4ba8d444f1b98232f906ea0c7b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:38.403522 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b256c6340bf453180048ed621c3df8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:38.413968 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5269c446aff43c5bcd4ad928c765585. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:38.414942 27066 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c2da49d4d4ce4dc782d0b5ce8f74efbc consensus state has no leader
I20250903 20:05:38.415807 29456 raft_consensus.cc:513] T 30cc0d4ba8d444f1b98232f906ea0c7b P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.418807 29457 raft_consensus.cc:2802] T d4c703dafbca4e669dbb18946ffac699 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:38.419018 29457 raft_consensus.cc:695] T d4c703dafbca4e669dbb18946ffac699 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:38.420020 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet ca722dfe38ac4e63b024508a3997cedb
I20250903 20:05:38.420939 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet ca722dfe38ac4e63b024508a3997cedb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:38.404170 29487 raft_consensus.cc:2802] T b4b15d77d07c4fa7856d9c198cf15450 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:38.422356 29487 raft_consensus.cc:491] T b4b15d77d07c4fa7856d9c198cf15450 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:38.422552 29487 raft_consensus.cc:3058] T b4b15d77d07c4fa7856d9c198cf15450 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:38.412253 27184 leader_election.cc:343] T e877d56bbbfa4c5a92d025d0311ce5c6 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: e877d56bbbfa4c5a92d025d0311ce5c6
I20250903 20:05:38.423051 27184 leader_election.cc:304] T e877d56bbbfa4c5a92d025d0311ce5c6 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:38.423094 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet ca722dfe38ac4e63b024508a3997cedb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:38.423967 29411 raft_consensus.cc:2747] T e877d56bbbfa4c5a92d025d0311ce5c6 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:38.423951 29457 consensus_queue.cc:237] T d4c703dafbca4e669dbb18946ffac699 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:38.424750 27184 leader_election.cc:343] T d4c703dafbca4e669dbb18946ffac699 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:38.425006 27184 leader_election.cc:343] T bf459db3327640f88574fca7b5f2ca98 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: bf459db3327640f88574fca7b5f2ca98
I20250903 20:05:38.426327 27353 tablet_service.cc:1468] Processing CreateTablet for tablet f5c42a292acf489a868363bced8d839b (DEFAULT_TABLE table=default.Table_4277890a8cb0439db05edde0b26bd8bb [id=edced480ee344c50adba42a98a424a55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:38.425099 27184 leader_election.cc:304] T bf459db3327640f88574fca7b5f2ca98 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:38.431116 29456 leader_election.cc:290] T 30cc0d4ba8d444f1b98232f906ea0c7b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:38.431264 29196 raft_consensus.cc:2747] T bf459db3327640f88574fca7b5f2ca98 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:38.432617 27376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4c703dafbca4e669dbb18946ffac699" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:38.406917 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1153) took 3127 ms. Trace:
W20250903 20:05:38.411753 27321 leader_election.cc:343] T b4b15d77d07c4fa7856d9c198cf15450 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:38.433707 27687 rpcz_store.cc:276] 0903 20:05:35.279904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.357232 (+ 77328us) service_pool.cc:225] Handling call
0903 20:05:38.139316 (+2782084us) ts_tablet_manager.cc:1145] Deleting tablet eaa11d17303142ff935705c16f0b9690
0903 20:05:38.146000 (+ 6684us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.158838 (+ 12838us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:38.249816 (+ 90978us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:38.406888 (+157072us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2781888}
I20250903 20:05:38.434305 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet eaa11d17303142ff935705c16f0b9690 (table default.Table_857612f25ca54f8b8d0ea31485d96a5d [id=ddebc54a9f9849478f0d1606dfe3bc96]) successfully deleted
I20250903 20:05:38.445961 27509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30cc0d4ba8d444f1b98232f906ea0c7b" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:38.446192 27509 raft_consensus.cc:3058] T 30cc0d4ba8d444f1b98232f906ea0c7b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:38.448302 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_1ee4b42dee0f45299e1beacc5af5b301"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:38.449036 27509 raft_consensus.cc:2466] T 30cc0d4ba8d444f1b98232f906ea0c7b P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
W20250903 20:05:38.449316 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ee4b42dee0f45299e1beacc5af5b301 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:38.449641 27183 leader_election.cc:304] T 30cc0d4ba8d444f1b98232f906ea0c7b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:38.452842 27384 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30cc0d4ba8d444f1b98232f906ea0c7b" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:38.453090 27384 raft_consensus.cc:3058] T 30cc0d4ba8d444f1b98232f906ea0c7b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:38.453238 27383 raft_consensus.cc:3058] T 7824a0bccdc9485abde124572e502791 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:38.453099 27380 raft_consensus.cc:3058] T 64c4dd2645034a7eaf243f5474603f10 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:38.453487 27228 tablet_service.cc:1515] Processing DeleteTablet for tablet a868191630454a1eb3a4301664178a9a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:38.453938 27228 tablet_service.cc:1515] Processing DeleteTablet for tablet ed73c335b042476fa03f154d23a611ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:38.454146 29490 tablet.cc:1722] T fc634116a78f45dd833034ddda90e9bf P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:38.454622 27228 tablet_service.cc:1468] Processing CreateTablet for tablet f5269c446aff43c5bcd4ad928c765585 (DEFAULT_TABLE table=default.Table_dec07ab3580e47e1a0be5bb8eb0dd841 [id=ec221a6c8d4448dca0c49887b58797f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:38.455272 27229 tablet_service.cc:1515] Processing DeleteTablet for tablet ca722dfe38ac4e63b024508a3997cedb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:56534
W20250903 20:05:38.456107 27184 leader_election.cc:343] T d4c703dafbca4e669dbb18946ffac699 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:38.458267 27504 tablet_service.cc:1468] Processing CreateTablet for tablet f5c42a292acf489a868363bced8d839b (DEFAULT_TABLE table=default.Table_4277890a8cb0439db05edde0b26bd8bb [id=edced480ee344c50adba42a98a424a55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:38.460661 29450 raft_consensus.cc:2802] T 30cc0d4ba8d444f1b98232f906ea0c7b P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:38.460814 29450 raft_consensus.cc:695] T 30cc0d4ba8d444f1b98232f906ea0c7b P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:38.461140 29450 consensus_queue.cc:237] T 30cc0d4ba8d444f1b98232f906ea0c7b P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.461585 29490 tablet.cc:1722] T 7a9e55f0811345f7afa71645f38ebdaf P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:38.465390 29487 raft_consensus.cc:513] T b4b15d77d07c4fa7856d9c198cf15450 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.466820 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b4b15d77d07c4fa7856d9c198cf15450" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
W20250903 20:05:38.467448 27321 leader_election.cc:343] T b4b15d77d07c4fa7856d9c198cf15450 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:38.467684 29196 consensus_queue.cc:1035] T 7824a0bccdc9485abde124572e502791 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:38.469440 27353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5c42a292acf489a868363bced8d839b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:38.477142 29463 tablet.cc:1722] T 3b3bfc63aa0b4e56bec189e834dc6a8d P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:38.479635 27384 raft_consensus.cc:2466] T 30cc0d4ba8d444f1b98232f906ea0c7b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:38.482014 27687 tablet_replica.cc:331] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:38.482357 27687 raft_consensus.cc:2241] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:38.482801 27687 raft_consensus.cc:2270] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:38.483651 27687 ts_tablet_manager.cc:1905] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:38.483912 29455 consensus_queue.cc:1035] T 7824a0bccdc9485abde124572e502791 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:38.489792 27510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b4b15d77d07c4fa7856d9c198cf15450" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:38.490449 27510 raft_consensus.cc:3058] T b4b15d77d07c4fa7856d9c198cf15450 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:38.491438 29487 leader_election.cc:290] T b4b15d77d07c4fa7856d9c198cf15450 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:38.492383 29196 consensus_queue.cc:1035] T 64c4dd2645034a7eaf243f5474603f10 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:38.492709 27692 tablet_bootstrap.cc:492] T f4dec89fc6bb436ba6542c267eaf9c15 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:38.492851 27692 ts_tablet_manager.cc:1397] T f4dec89fc6bb436ba6542c267eaf9c15 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.240s user 0.015s sys 0.010s
I20250903 20:05:38.494908 27693 tablet_bootstrap.cc:492] T e2441aa3c2c748bba5360259fb663c81 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:38.497831 27693 tablet_bootstrap.cc:654] T e2441aa3c2c748bba5360259fb663c81 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:38.504914 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5c42a292acf489a868363bced8d839b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:38.508754 29518 tablet.cc:1722] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:38.509738 27510 raft_consensus.cc:2466] T b4b15d77d07c4fa7856d9c198cf15450 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:38.493809 27692 raft_consensus.cc:357] T f4dec89fc6bb436ba6542c267eaf9c15 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.516693 27692 raft_consensus.cc:383] T f4dec89fc6bb436ba6542c267eaf9c15 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:38.517386 27692 raft_consensus.cc:738] T f4dec89fc6bb436ba6542c267eaf9c15 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:38.519335 27692 consensus_queue.cc:260] T f4dec89fc6bb436ba6542c267eaf9c15 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.528932 27066 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 6d4b8d4bf29e408cbdf9557c8b0d8a4b (table default.Table_ccdb4130549b4be5888f17097ae35106 [id=cbcfc7067aca4796b39e3eb195f38903]). Expected version 2 got 0
I20250903 20:05:38.531812 27692 ts_tablet_manager.cc:1428] T f4dec89fc6bb436ba6542c267eaf9c15 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.038s user 0.002s sys 0.002s
I20250903 20:05:38.538569 27692 tablet_bootstrap.cc:492] T ed4902c002ef4198802375ced39ef2ad P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:38.544590 27229 tablet_service.cc:1468] Processing CreateTablet for tablet f5c42a292acf489a868363bced8d839b (DEFAULT_TABLE table=default.Table_4277890a8cb0439db05edde0b26bd8bb [id=edced480ee344c50adba42a98a424a55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:38.546082 27066 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 8ba2636641654d8596fd2a5b0644d073 (table default.Table_e7148e16606e4cd3908c86b439f75b78 [id=4bffa29ea98049718ac3ce35de9813c5]). Expected version 1 got 0
I20250903 20:05:38.551177 27692 tablet_bootstrap.cc:654] T ed4902c002ef4198802375ced39ef2ad P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:38.562453 27183 consensus_peers.cc:489] T 4e34d6d5d68e4395aed78821ea077c1b P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e34d6d5d68e4395aed78821ea077c1b. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:38.565280 27228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5269c446aff43c5bcd4ad928c765585. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:38.569183 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a910f8157df4308b5720e66f2a78347. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:38.571130 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b256c6340bf453180048ed621c3df8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:38.576391 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccb671d7e67444ac8e5b78055271d02a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:38.576604 27692 tablet_bootstrap.cc:492] T ed4902c002ef4198802375ced39ef2ad P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:38.576889 27692 ts_tablet_manager.cc:1397] T ed4902c002ef4198802375ced39ef2ad P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.011s
I20250903 20:05:38.577970 27692 raft_consensus.cc:357] T ed4902c002ef4198802375ced39ef2ad P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.579522 27692 raft_consensus.cc:383] T ed4902c002ef4198802375ced39ef2ad P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:38.579599 27692 raft_consensus.cc:738] T ed4902c002ef4198802375ced39ef2ad P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:38.579869 27692 consensus_queue.cc:260] T ed4902c002ef4198802375ced39ef2ad P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.580387 27695 tablet_bootstrap.cc:654] T 2385d3006f90486890ebe43e16a53d7a P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:38.581313 27692 ts_tablet_manager.cc:1428] T ed4902c002ef4198802375ced39ef2ad P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20250903 20:05:38.581672 27692 tablet_bootstrap.cc:492] T 60c32370a5274347afafc6a7e1e94270 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:38.584633 27685 raft_consensus.cc:2241] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:38.585098 27685 raft_consensus.cc:2270] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:38.585312 27229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5c42a292acf489a868363bced8d839b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:38.585697 27685 ts_tablet_manager.cc:1905] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:38.587404 27692 tablet_bootstrap.cc:654] T 60c32370a5274347afafc6a7e1e94270 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:38.592684 27685 ts_tablet_manager.cc:1918] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:38.592893 27685 log.cc:1199] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/a253e3399622409e92299069b19d3698
I20250903 20:05:38.593354 27685 ts_tablet_manager.cc:1939] T a253e3399622409e92299069b19d3698 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:38.593796 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 999) took 5754 ms. Trace:
I20250903 20:05:38.593952 27685 rpcz_store.cc:276] 0903 20:05:32.839096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.843250 (+ 4154us) service_pool.cc:225] Handling call
0903 20:05:38.274610 (+5431360us) ts_tablet_manager.cc:1145] Deleting tablet a253e3399622409e92299069b19d3698
0903 20:05:38.274622 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.588799 (+314177us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:38.592568 (+ 3769us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:38.593778 (+ 1210us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":309690,"tablet-delete.queue_time_us":5431135}
I20250903 20:05:38.594906 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1009) took 5616 ms. Trace:
I20250903 20:05:38.595041 27685 rpcz_store.cc:276] 0903 20:05:32.978504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.978562 (+ 58us) service_pool.cc:225] Handling call
0903 20:05:38.594777 (+5616215us) ts_tablet_manager.cc:1145] Deleting tablet 3c47c5fca0ee4349b2667494f45af5d0
0903 20:05:38.594786 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.594890 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5615968}
I20250903 20:05:38.595634 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1010) took 5617 ms. Trace:
I20250903 20:05:38.595845 27685 rpcz_store.cc:276] 0903 20:05:32.978626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:32.979036 (+ 410us) service_pool.cc:225] Handling call
0903 20:05:38.595474 (+5616438us) ts_tablet_manager.cc:1145] Deleting tablet b32c82c3d8c4420cb6e3a95129550076
0903 20:05:38.595485 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.595606 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5616224}
I20250903 20:05:38.596297 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1011) took 5526 ms. Trace:
I20250903 20:05:38.596432 27685 rpcz_store.cc:276] 0903 20:05:33.069425 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:33.070284 (+ 859us) service_pool.cc:225] Handling call
0903 20:05:38.596202 (+5525918us) ts_tablet_manager.cc:1145] Deleting tablet 16549a6d9aa9432eb5cee19f69ce6957
0903 20:05:38.596209 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.596281 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5525641}
I20250903 20:05:38.596899 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1027) took 5020 ms. Trace:
I20250903 20:05:38.597033 27685 rpcz_store.cc:276] 0903 20:05:33.576492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:33.576543 (+ 51us) service_pool.cc:225] Handling call
0903 20:05:38.596821 (+5020278us) ts_tablet_manager.cc:1145] Deleting tablet 59a1dd48d76241d7ba817070fbffd853
0903 20:05:38.596829 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.596885 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5020070}
I20250903 20:05:38.597406 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1026) took 5021 ms. Trace:
I20250903 20:05:38.597518 27685 rpcz_store.cc:276] 0903 20:05:33.576375 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:33.576793 (+ 418us) service_pool.cc:225] Handling call
0903 20:05:38.597334 (+5020541us) ts_tablet_manager.cc:1145] Deleting tablet 8a223f75045345ae833f52bcc8f239f9
0903 20:05:38.597342 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.597392 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5020397}
I20250903 20:05:38.597870 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1028) took 4899 ms. Trace:
I20250903 20:05:38.597995 27685 rpcz_store.cc:276] 0903 20:05:33.698572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:33.698678 (+ 106us) service_pool.cc:225] Handling call
0903 20:05:38.597775 (+4899097us) ts_tablet_manager.cc:1145] Deleting tablet 3590506b3fd44a6c84aa469c14ed3e9f
0903 20:05:38.597783 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.597855 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4898914}
I20250903 20:05:38.598459 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1029) took 4891 ms. Trace:
I20250903 20:05:38.598587 27685 rpcz_store.cc:276] 0903 20:05:33.707091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:33.707268 (+ 177us) service_pool.cc:225] Handling call
0903 20:05:38.598365 (+4891097us) ts_tablet_manager.cc:1145] Deleting tablet d1a254d560904b9aa31b8ee7de65ffa5
0903 20:05:38.598373 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.598443 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4890900}
I20250903 20:05:38.598976 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1030) took 4891 ms. Trace:
I20250903 20:05:38.599731 29170 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250903 20:05:38.607517 27692 tablet_bootstrap.cc:492] T 60c32370a5274347afafc6a7e1e94270 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:38.609515 27692 ts_tablet_manager.cc:1397] T 60c32370a5274347afafc6a7e1e94270 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.014s
I20250903 20:05:38.615875 29437 consensus_queue.cc:1035] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:38.617442 27692 raft_consensus.cc:357] T 60c32370a5274347afafc6a7e1e94270 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.617641 27692 raft_consensus.cc:383] T 60c32370a5274347afafc6a7e1e94270 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:38.617702 27692 raft_consensus.cc:738] T 60c32370a5274347afafc6a7e1e94270 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:38.617906 27692 consensus_queue.cc:260] T 60c32370a5274347afafc6a7e1e94270 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.618852 27692 ts_tablet_manager.cc:1428] T 60c32370a5274347afafc6a7e1e94270 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250903 20:05:38.619176 27692 tablet_bootstrap.cc:492] T 4739f10128d64e2c986257b1d9deb90f P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:38.510555 27320 leader_election.cc:304] T b4b15d77d07c4fa7856d9c198cf15450 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:38.619822 29487 raft_consensus.cc:2802] T b4b15d77d07c4fa7856d9c198cf15450 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
W20250903 20:05:38.619916 27320 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 109403us
I20250903 20:05:38.544932 29490 tablet.cc:1722] T 3b3bfc63aa0b4e56bec189e834dc6a8d P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:38.556574 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 3f303233e6c347289650c6cf83f8127f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:38.555809 27349 tablet_service.cc:1515] Processing DeleteTablet for tablet 0fef9aa7153a4902abce522e3ff0df3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:38.558138 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e9a6a6e05464b7b97ae624f51402cf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:38.621776 27692 tablet_bootstrap.cc:654] T 4739f10128d64e2c986257b1d9deb90f P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:38.557258 27353 tablet_service.cc:1515] Processing DeleteTablet for tablet 64c4dd2645034a7eaf243f5474603f10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:38.558980 27347 tablet_service.cc:1515] Processing DeleteTablet for tablet 259bc73ca9df47e9874692725a60bb1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:38.619966 29487 raft_consensus.cc:695] T b4b15d77d07c4fa7856d9c198cf15450 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:38.624223 29487 consensus_queue.cc:237] T b4b15d77d07c4fa7856d9c198cf15450 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.609536 29455 consensus_queue.cc:1035] T 3f303233e6c347289650c6cf83f8127f P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:38.647465 27685 rpcz_store.cc:276] 0903 20:05:33.707209 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:33.707303 (+ 94us) service_pool.cc:225] Handling call
0903 20:05:38.598886 (+4891583us) ts_tablet_manager.cc:1145] Deleting tablet dd27901224d6442389e8fc09c6b498ab
0903 20:05:38.598895 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.598962 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4891317}
I20250903 20:05:38.647902 29411 consensus_queue.cc:1035] T 3f303233e6c347289650c6cf83f8127f P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:38.648021 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1031) took 4911 ms. Trace:
I20250903 20:05:38.648161 27685 rpcz_store.cc:276] 0903 20:05:33.736266 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:33.736323 (+ 57us) service_pool.cc:225] Handling call
0903 20:05:38.647871 (+4911548us) ts_tablet_manager.cc:1145] Deleting tablet df5f439f5aca4231af229159811b7a75
0903 20:05:38.647886 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.648001 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4910832}
I20250903 20:05:38.648514 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1032) took 4841 ms. Trace:
I20250903 20:05:38.648582 27685 rpcz_store.cc:276] 0903 20:05:33.807427 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:33.807486 (+ 59us) service_pool.cc:225] Handling call
0903 20:05:38.648408 (+4840922us) ts_tablet_manager.cc:1145] Deleting tablet 65d07ebe1ec94baabc6ce011e761b4d5
0903 20:05:38.648423 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.648500 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4840736}
I20250903 20:05:38.648890 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1033) took 4836 ms. Trace:
I20250903 20:05:38.649116 29443 raft_consensus.cc:491] T f4dec89fc6bb436ba6542c267eaf9c15 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:38.649130 27685 rpcz_store.cc:276] 0903 20:05:33.812528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:33.812601 (+ 73us) service_pool.cc:225] Handling call
0903 20:05:38.648797 (+4836196us) ts_tablet_manager.cc:1145] Deleting tablet e35ae50252734a099ba5ce2d04e49e1f
0903 20:05:38.648806 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.648877 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4835999}
I20250903 20:05:38.649324 29443 raft_consensus.cc:513] T f4dec89fc6bb436ba6542c267eaf9c15 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.649536 27685 tablet_replica.cc:331] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:38.649760 27685 raft_consensus.cc:2241] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:38.650094 27685 raft_consensus.cc:2270] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:38.650329 29443 leader_election.cc:290] T f4dec89fc6bb436ba6542c267eaf9c15 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:38.650954 27685 ts_tablet_manager.cc:1905] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:38.651903 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4dec89fc6bb436ba6542c267eaf9c15" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:38.653954 27462 leader_election.cc:343] T f4dec89fc6bb436ba6542c267eaf9c15 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:38.654721 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4dec89fc6bb436ba6542c267eaf9c15" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:38.655586 27462 leader_election.cc:343] T f4dec89fc6bb436ba6542c267eaf9c15 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:38.655714 27462 leader_election.cc:304] T f4dec89fc6bb436ba6542c267eaf9c15 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:38.656419 29371 raft_consensus.cc:2747] T f4dec89fc6bb436ba6542c267eaf9c15 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:38.658265 29443 raft_consensus.cc:491] T 249362a0b8304719863bcb82c47273b3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:38.658406 29443 raft_consensus.cc:513] T 249362a0b8304719863bcb82c47273b3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.659008 29443 leader_election.cc:290] T 249362a0b8304719863bcb82c47273b3 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:38.659791 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "249362a0b8304719863bcb82c47273b3" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:38.660363 27462 leader_election.cc:343] T 249362a0b8304719863bcb82c47273b3 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:38.554411 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_61fe5d6eecb744fdabbe92ed7320115a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:38.595320 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet a253e3399622409e92299069b19d3698 (table default.Table_8bbb5561118c4a68af214f8d319d2caf [id=2218f11e07d243adaa3a4d24e521b1b7]) successfully deleted
W20250903 20:05:38.660847 27052 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 65539.6us
W20250903 20:05:38.661199 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 3c47c5fca0ee4349b2667494f45af5d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c47c5fca0ee4349b2667494f45af5d0 already in progress: creating tablet
W20250903 20:05:38.661237 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61fe5d6eecb744fdabbe92ed7320115a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:38.661419 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet b32c82c3d8c4420cb6e3a95129550076 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b32c82c3d8c4420cb6e3a95129550076 already in progress: creating tablet
W20250903 20:05:38.661600 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 16549a6d9aa9432eb5cee19f69ce6957 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16549a6d9aa9432eb5cee19f69ce6957 already in progress: creating tablet
W20250903 20:05:38.661823 27052 catalog_manager.cc:4895] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 59a1dd48d76241d7ba817070fbffd853 because the tablet was not found. No further retry: Not found: Tablet not found: 59a1dd48d76241d7ba817070fbffd853
W20250903 20:05:38.661998 27052 catalog_manager.cc:4895] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 8a223f75045345ae833f52bcc8f239f9 because the tablet was not found. No further retry: Not found: Tablet not found: 8a223f75045345ae833f52bcc8f239f9
W20250903 20:05:38.662170 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 3590506b3fd44a6c84aa469c14ed3e9f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3590506b3fd44a6c84aa469c14ed3e9f already in progress: creating tablet
W20250903 20:05:38.662420 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet d1a254d560904b9aa31b8ee7de65ffa5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1a254d560904b9aa31b8ee7de65ffa5 already in progress: creating tablet
W20250903 20:05:38.662768 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet dd27901224d6442389e8fc09c6b498ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dd27901224d6442389e8fc09c6b498ab already in progress: creating tablet
W20250903 20:05:38.662932 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet df5f439f5aca4231af229159811b7a75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df5f439f5aca4231af229159811b7a75 already in progress: creating tablet
W20250903 20:05:38.663111 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 65d07ebe1ec94baabc6ce011e761b4d5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65d07ebe1ec94baabc6ce011e761b4d5 already in progress: creating tablet
I20250903 20:05:38.663254 27066 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 249362a0b8304719863bcb82c47273b3 (table default.Table_1dd1b4a75c914e6a88058266d2a7e98a [id=565dc96c38de4efeb4328c01e030eb8c]). Expected version 1 got 0
W20250903 20:05:38.663260 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet e35ae50252734a099ba5ce2d04e49e1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e35ae50252734a099ba5ce2d04e49e1f already in progress: creating tablet
I20250903 20:05:38.663549 27066 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 7345305820be4d32a3ceb227f045f009 (table default.Table_5f0199a660844c15a254a062c1540498 [id=52f969e50b9d466cbd14de6f252b7706]). Expected version 1 got 0
I20250903 20:05:38.665371 27217 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56534 (request call id 1211) took 1065 ms. Trace:
I20250903 20:05:38.665475 27217 rpcz_store.cc:276] 0903 20:05:37.600265 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:37.845859 (+245594us) service_pool.cc:225] Handling call
0903 20:05:38.270368 (+424509us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0903 20:05:38.270401 (+ 33us) ts_tablet_manager.cc:676] Creating new metadata...
0903 20:05:38.580309 (+309908us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:38.665192 (+ 84883us) tablet_replica.cc:206] Creating consensus instance
0903 20:05:38.665352 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250903 20:05:38.666570 27692 tablet_bootstrap.cc:492] T 4739f10128d64e2c986257b1d9deb90f P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:38.666718 27692 ts_tablet_manager.cc:1397] T 4739f10128d64e2c986257b1d9deb90f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.048s user 0.012s sys 0.016s
I20250903 20:05:38.667560 27692 raft_consensus.cc:357] T 4739f10128d64e2c986257b1d9deb90f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.667739 27692 raft_consensus.cc:383] T 4739f10128d64e2c986257b1d9deb90f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:38.667794 27692 raft_consensus.cc:738] T 4739f10128d64e2c986257b1d9deb90f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:38.667976 27692 consensus_queue.cc:260] T 4739f10128d64e2c986257b1d9deb90f P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.670074 27216 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56534 (request call id 1188) took 1076 ms. Trace:
I20250903 20:05:38.670192 27216 rpcz_store.cc:276] 0903 20:05:37.593521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:37.922312 (+328791us) service_pool.cc:225] Handling call
0903 20:05:38.268169 (+345857us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0903 20:05:38.268206 (+ 37us) ts_tablet_manager.cc:676] Creating new metadata...
0903 20:05:38.643495 (+375289us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:38.646551 (+ 3056us) tablet_replica.cc:206] Creating consensus instance
0903 20:05:38.670053 (+ 23502us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250903 20:05:38.674718 27462 consensus_peers.cc:489] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:38.677994 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 793d6157927143d4b9d8dbafd7cf0fcb
I20250903 20:05:38.681653 27692 ts_tablet_manager.cc:1428] T 4739f10128d64e2c986257b1d9deb90f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20250903 20:05:38.682121 27692 tablet_bootstrap.cc:492] T 5fb2426aa01e47598f268d0a90475271 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:38.683861 27692 tablet_bootstrap.cc:654] T 5fb2426aa01e47598f268d0a90475271 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:38.687157 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_6eb8ab17496f4e13906809835d0e8d72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:38.687717 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6eb8ab17496f4e13906809835d0e8d72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:38.691236 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "249362a0b8304719863bcb82c47273b3" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:38.692797 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 793d6157927143d4b9d8dbafd7cf0fcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:38.693030 27462 leader_election.cc:343] T 249362a0b8304719863bcb82c47273b3 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:38.693146 27462 leader_election.cc:304] T 249362a0b8304719863bcb82c47273b3 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:38.694164 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_9cac2ed28d064dfebdc8acfc692b00d9" } new_table_name: "default.Table_49567c20a4e64e08842cc320aafa9a01" modify_external_catalogs: true
I20250903 20:05:38.697082 29371 raft_consensus.cc:2747] T 249362a0b8304719863bcb82c47273b3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:38.700346 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet de20d18d7f264811a0631aa1144aef2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:38.702397 27224 tablet_service.cc:1515] Processing DeleteTablet for tablet 793d6157927143d4b9d8dbafd7cf0fcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:38.705257 27695 tablet_bootstrap.cc:492] T 2385d3006f90486890ebe43e16a53d7a P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:38.705487 27695 ts_tablet_manager.cc:1397] T 2385d3006f90486890ebe43e16a53d7a P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.293s user 0.000s sys 0.004s
I20250903 20:05:38.706670 27347 tablet_service.cc:1515] Processing DeleteTablet for tablet 793d6157927143d4b9d8dbafd7cf0fcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:38.706651 27695 raft_consensus.cc:357] T 2385d3006f90486890ebe43e16a53d7a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.706962 27695 raft_consensus.cc:383] T 2385d3006f90486890ebe43e16a53d7a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:38.707067 27695 raft_consensus.cc:738] T 2385d3006f90486890ebe43e16a53d7a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:38.707383 27695 consensus_queue.cc:260] T 2385d3006f90486890ebe43e16a53d7a P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.707945 27216 tablet_service.cc:1515] Processing DeleteTablet for tablet a253e3399622409e92299069b19d3698 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:38.708205 27695 ts_tablet_manager.cc:1428] T 2385d3006f90486890ebe43e16a53d7a P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:38.714717 27695 tablet_bootstrap.cc:492] T 0fef9aa7153a4902abce522e3ff0df3f P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:38.718016 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_3110722931cd47ad8e82109243b34fc1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:38.718600 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3110722931cd47ad8e82109243b34fc1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:38.720273 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6ce46d642ad64aa8a5edb5dd465fb9cc consensus state has no leader
I20250903 20:05:38.720580 27065 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_fd33384721334ad0b70806e8e70a60f2" } modify_external_catalogs: true
I20250903 20:05:38.720718 27065 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_fd33384721334ad0b70806e8e70a60f2" } modify_external_catalogs: true
I20250903 20:05:38.728379 27687 ts_tablet_manager.cc:1918] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:38.730574 27687 log.cc:1199] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/09fa32312c9648fbbd6fd327e9738066
I20250903 20:05:38.731477 27695 tablet_bootstrap.cc:654] T 0fef9aa7153a4902abce522e3ff0df3f P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:38.736059 27685 ts_tablet_manager.cc:1918] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 20:05:38.752796 27687 ts_tablet_manager.cc:1939] T 09fa32312c9648fbbd6fd327e9738066 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:38.754091 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1155) took 3381 ms. Trace:
I20250903 20:05:38.754340 27687 rpcz_store.cc:276] 0903 20:05:35.372744 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.372801 (+ 57us) service_pool.cc:225] Handling call
0903 20:05:38.434635 (+3061834us) ts_tablet_manager.cc:1145] Deleting tablet 09fa32312c9648fbbd6fd327e9738066
0903 20:05:38.481918 (+ 47283us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.675211 (+193293us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:38.728250 (+ 53039us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:38.754065 (+ 25815us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3047186}
I20250903 20:05:38.754966 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 09fa32312c9648fbbd6fd327e9738066 (table default.Table_61ee151cc2184644b05a60d8feb9e3c4 [id=f459194547904acd99011de4c21c4e33]) successfully deleted
I20250903 20:05:38.755577 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1156) took 3303 ms. Trace:
I20250903 20:05:38.755738 27687 rpcz_store.cc:276] 0903 20:05:35.451858 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.454983 (+ 3125us) service_pool.cc:225] Handling call
0903 20:05:38.755406 (+3300423us) ts_tablet_manager.cc:1145] Deleting tablet a7f25f00dff7426195ac51214a14022b
0903 20:05:38.755418 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.755548 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3300107}
W20250903 20:05:38.756248 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet a7f25f00dff7426195ac51214a14022b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7f25f00dff7426195ac51214a14022b already in progress: creating tablet
I20250903 20:05:38.756634 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1157) took 3266 ms. Trace:
I20250903 20:05:38.756763 27687 rpcz_store.cc:276] 0903 20:05:35.489770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.492091 (+ 2321us) service_pool.cc:225] Handling call
0903 20:05:38.756526 (+3264435us) ts_tablet_manager.cc:1145] Deleting tablet c9d09ecd8fe347d79283fc9a3056e844
0903 20:05:38.756535 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.756618 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3264228}
I20250903 20:05:38.757158 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 3f08c7edbcf54c16bc222f59773f72b1
I20250903 20:05:38.757361 27687 tablet_replica.cc:331] T f326ec776ba446be8d5dda5c1dc5b6ec P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:38.757645 27057 catalog_manager.cc:6652] ReplaceTablet: tablet 3f08c7edbcf54c16bc222f59773f72b1 deleted and replaced by tablet 760a0bd9eb924ff7a33b336b057b850a
I20250903 20:05:38.758747 27063 catalog_manager.cc:5582] T b4b15d77d07c4fa7856d9c198cf15450 P 51578fb7bceb4e119b3ab1e58c4f5528 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66). New cstate: current_term: 1 leader_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:38.758997 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 3f08c7edbcf54c16bc222f59773f72b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 760a0bd9eb924ff7a33b336b057b850a) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:38.757171 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet c9d09ecd8fe347d79283fc9a3056e844 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9d09ecd8fe347d79283fc9a3056e844 already in progress: creating tablet
I20250903 20:05:38.759469 27685 log.cc:1199] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/2d9c1642fec841d497ff3496dd4564db
I20250903 20:05:38.759841 27347 tablet_service.cc:1515] Processing DeleteTablet for tablet 3f08c7edbcf54c16bc222f59773f72b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 760a0bd9eb924ff7a33b336b057b850a) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:38.760039 27216 tablet_service.cc:1515] Processing DeleteTablet for tablet 3f08c7edbcf54c16bc222f59773f72b1 with delete_type TABLET_DATA_DELETED (replaced by tablet 760a0bd9eb924ff7a33b336b057b850a) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:38.762445 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_74cf241e5c694a219f63258fb4b582c6" } new_table_name: "default.Table_6cd80dfca865441f985374382f5be9ec" modify_external_catalogs: true
I20250903 20:05:38.762123 27057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_79300b77de8c404a9e85783ce977861c" } new_table_name: "default.Table_0a4db1d67753479197d68a85fb3d6c5c" modify_external_catalogs: true
I20250903 20:05:38.765102 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0276baa7ef3b41c994e5d131569d5955" } modify_external_catalogs: true
I20250903 20:05:38.765259 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0276baa7ef3b41c994e5d131569d5955" } modify_external_catalogs: true
I20250903 20:05:38.765559 27064 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_99280cacf5a44f0a8818e3e0dedc93f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:38.766283 27064 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99280cacf5a44f0a8818e3e0dedc93f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:38.777194 27685 ts_tablet_manager.cc:1939] T 2d9c1642fec841d497ff3496dd4564db P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:38.777488 29455 consensus_queue.cc:1035] T 259bc73ca9df47e9874692725a60bb1c P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:38.782121 29371 raft_consensus.cc:491] T 54f1ce5192f547f595dea23a9eabf8e6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:38.782338 29371 raft_consensus.cc:513] T 54f1ce5192f547f595dea23a9eabf8e6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.782872 29371 leader_election.cc:290] T 54f1ce5192f547f595dea23a9eabf8e6 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:38.783514 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f1ce5192f547f595dea23a9eabf8e6" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:38.784241 27462 leader_election.cc:343] T 54f1ce5192f547f595dea23a9eabf8e6 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:38.786944 27692 tablet_bootstrap.cc:492] T 5fb2426aa01e47598f268d0a90475271 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:38.787122 27692 ts_tablet_manager.cc:1397] T 5fb2426aa01e47598f268d0a90475271 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.105s user 0.005s sys 0.016s
I20250903 20:05:38.787856 27692 raft_consensus.cc:357] T 5fb2426aa01e47598f268d0a90475271 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.788053 27692 raft_consensus.cc:383] T 5fb2426aa01e47598f268d0a90475271 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:38.788137 27692 raft_consensus.cc:738] T 5fb2426aa01e47598f268d0a90475271 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:38.788371 27692 consensus_queue.cc:260] T 5fb2426aa01e47598f268d0a90475271 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.788698 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1043) took 4716 ms. Trace:
I20250903 20:05:38.788874 27692 ts_tablet_manager.cc:1428] T 5fb2426aa01e47598f268d0a90475271 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:38.788839 27685 rpcz_store.cc:276] 0903 20:05:34.071861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.071909 (+ 48us) service_pool.cc:225] Handling call
0903 20:05:38.649355 (+4577446us) ts_tablet_manager.cc:1145] Deleting tablet 2d9c1642fec841d497ff3496dd4564db
0903 20:05:38.649363 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.671626 (+ 22263us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:38.735928 (+ 64302us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:38.788678 (+ 52750us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4566232}
I20250903 20:05:38.789184 27692 tablet_bootstrap.cc:492] T 659a127405d341c7b074d3445854256f P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:38.789587 27685 tablet_replica.cc:331] T e877d56bbbfa4c5a92d025d0311ce5c6 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:38.789554 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 2d9c1642fec841d497ff3496dd4564db (table default.Table_63d8bb9ad5a14a7d81fb6f15d20e6ff0 [id=f5cb786b03db4395a0f40b2956f5d331]) successfully deleted
I20250903 20:05:38.789974 27685 raft_consensus.cc:2241] T e877d56bbbfa4c5a92d025d0311ce5c6 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:38.790200 27685 raft_consensus.cc:2270] T e877d56bbbfa4c5a92d025d0311ce5c6 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:38.790683 27692 tablet_bootstrap.cc:654] T 659a127405d341c7b074d3445854256f P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:38.797379 29463 tablet.cc:1722] T 9e98cd1d9bd246e182e2969d147e68bf P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:38.798010 29518 tablet.cc:1722] T 9e98cd1d9bd246e182e2969d147e68bf P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:38.799530 27685 ts_tablet_manager.cc:1905] T e877d56bbbfa4c5a92d025d0311ce5c6 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:38.759192 27063 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet b4b15d77d07c4fa7856d9c198cf15450 (table default.Table_3edadd8954444273b91ea4d39cf3a834 [id=45cd523acd3c45b994d1b9cc6eb569ea]). Expected version 4 got 0
I20250903 20:05:38.803119 27692 tablet_bootstrap.cc:492] T 659a127405d341c7b074d3445854256f P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:38.803311 27692 ts_tablet_manager.cc:1397] T 659a127405d341c7b074d3445854256f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20250903 20:05:38.804018 27692 raft_consensus.cc:357] T 659a127405d341c7b074d3445854256f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.804203 27692 raft_consensus.cc:383] T 659a127405d341c7b074d3445854256f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:38.804277 27692 raft_consensus.cc:738] T 659a127405d341c7b074d3445854256f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:38.804476 27692 consensus_queue.cc:260] T 659a127405d341c7b074d3445854256f P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.804934 27692 ts_tablet_manager.cc:1428] T 659a127405d341c7b074d3445854256f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:38.805219 27692 tablet_bootstrap.cc:492] T 7d5a3fbe93e44c228222ab1b1e78c187 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:38.811007 29461 consensus_queue.cc:1035] T 259bc73ca9df47e9874692725a60bb1c P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250903 20:05:38.825762 29550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 20:05:38.827319 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f1ce5192f547f595dea23a9eabf8e6" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:38.827735 27321 consensus_peers.cc:489] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Couldn't send request to peer 89f8f0ee68b940db9ce250c0604f934a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250903 20:05:38.827426 29545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 20:05:38.828282 27462 leader_election.cc:343] T 54f1ce5192f547f595dea23a9eabf8e6 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:38.828428 27462 leader_election.cc:304] T 54f1ce5192f547f595dea23a9eabf8e6 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:38.828859 29443 raft_consensus.cc:2747] T 54f1ce5192f547f595dea23a9eabf8e6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:38.829488 27692 tablet_bootstrap.cc:654] T 7d5a3fbe93e44c228222ab1b1e78c187 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:38.831202 27693 tablet_bootstrap.cc:492] T e2441aa3c2c748bba5360259fb663c81 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:38.831769 27066 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_5307cd5405d348b0a2feeb026c556689 [id=53074cd97dec40c7b0e1444183d6707a] alter complete (version 4)
I20250903 20:05:38.834002 29170 server_base.cc:1047] running on GCE node
W20250903 20:05:38.834614 27303 tablet.cc:2378] T 2385d3006f90486890ebe43e16a53d7a P 89f8f0ee68b940db9ce250c0604f934a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 20:05:38.838517 28567 raft_consensus.cc:491] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:38.842618 28567 raft_consensus.cc:513] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.843326 28567 leader_election.cc:290] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:38.844673 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fef9aa7153a4902abce522e3ff0df3f" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:38.844673 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fef9aa7153a4902abce522e3ff0df3f" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:38.844895 27516 raft_consensus.cc:2466] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
W20250903 20:05:38.845357 27321 leader_election.cc:343] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:38.845475 27321 leader_election.cc:304] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:38.845948 28567 raft_consensus.cc:2802] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:38.846027 28567 raft_consensus.cc:491] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:38.846087 28567 raft_consensus.cc:3058] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:38.848852 28567 raft_consensus.cc:513] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.850064 28567 leader_election.cc:290] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:38.850582 27695 tablet_bootstrap.cc:492] T 0fef9aa7153a4902abce522e3ff0df3f P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:38.850793 27695 ts_tablet_manager.cc:1397] T 0fef9aa7153a4902abce522e3ff0df3f P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.136s user 0.004s sys 0.002s
I20250903 20:05:38.850796 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fef9aa7153a4902abce522e3ff0df3f" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:38.851006 27516 raft_consensus.cc:3058] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:38.851387 27692 tablet_bootstrap.cc:492] T 7d5a3fbe93e44c228222ab1b1e78c187 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:38.851679 27692 ts_tablet_manager.cc:1397] T 7d5a3fbe93e44c228222ab1b1e78c187 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.000s
I20250903 20:05:38.852083 27695 raft_consensus.cc:357] T 0fef9aa7153a4902abce522e3ff0df3f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.852663 27692 raft_consensus.cc:357] T 7d5a3fbe93e44c228222ab1b1e78c187 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.852900 27692 raft_consensus.cc:383] T 7d5a3fbe93e44c228222ab1b1e78c187 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:38.853000 27692 raft_consensus.cc:738] T 7d5a3fbe93e44c228222ab1b1e78c187 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:38.853212 27695 raft_consensus.cc:383] T 0fef9aa7153a4902abce522e3ff0df3f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:38.853423 27695 raft_consensus.cc:738] T 0fef9aa7153a4902abce522e3ff0df3f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:38.853554 27692 consensus_queue.cc:260] T 7d5a3fbe93e44c228222ab1b1e78c187 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.853909 27695 consensus_queue.cc:260] T 0fef9aa7153a4902abce522e3ff0df3f P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.854235 27692 ts_tablet_manager.cc:1428] T 7d5a3fbe93e44c228222ab1b1e78c187 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250903 20:05:38.854350 27516 raft_consensus.cc:2466] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:38.854604 27692 tablet_bootstrap.cc:492] T 69f8eef52bee431a86f0eef45fd11f9f P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:38.855125 27320 leader_election.cc:304] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:38.855705 28567 raft_consensus.cc:2802] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:38.855935 28567 raft_consensus.cc:695] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:38.856189 28567 consensus_queue.cc:237] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.856922 27065 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 421dc8af40c94a89bc372e8779872a4e
W20250903 20:05:38.857054 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a307376a81904146ae8eeb39f11d9235 consensus state has no leader
I20250903 20:05:38.858418 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 421dc8af40c94a89bc372e8779872a4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:38.858712 27692 tablet_bootstrap.cc:654] T 69f8eef52bee431a86f0eef45fd11f9f P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:38.858798 27347 tablet_service.cc:1515] Processing DeleteTablet for tablet 421dc8af40c94a89bc372e8779872a4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:38.859666 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_9b36ebc04fa442909cce88c8e4695c10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:38.860386 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b36ebc04fa442909cce88c8e4695c10 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:38.864722 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_42f95b6c82f949a082559298f94e747c" } modify_external_catalogs: true
I20250903 20:05:38.864712 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_656f015bd5f84e7c92ecf269159af5de" } new_table_name: "default.Table_8822d0b35d264ac08b273006c92d23d7" modify_external_catalogs: true
I20250903 20:05:38.865154 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_42f95b6c82f949a082559298f94e747c" } modify_external_catalogs: true
I20250903 20:05:38.866652 27693 ts_tablet_manager.cc:1397] T e2441aa3c2c748bba5360259fb663c81 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.372s user 0.006s sys 0.006s
I20250903 20:05:38.855523 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fef9aa7153a4902abce522e3ff0df3f" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:38.858698 27216 tablet_service.cc:1515] Processing DeleteTablet for tablet 421dc8af40c94a89bc372e8779872a4e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:38.867444 29443 raft_consensus.cc:491] T 4739f10128d64e2c986257b1d9deb90f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:38.867641 29443 raft_consensus.cc:513] T 4739f10128d64e2c986257b1d9deb90f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.867753 27693 raft_consensus.cc:357] T e2441aa3c2c748bba5360259fb663c81 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.867918 27693 raft_consensus.cc:383] T e2441aa3c2c748bba5360259fb663c81 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:38.867991 27693 raft_consensus.cc:738] T e2441aa3c2c748bba5360259fb663c81 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:38.868269 27693 consensus_queue.cc:260] T e2441aa3c2c748bba5360259fb663c81 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.868481 29443 leader_election.cc:290] T 4739f10128d64e2c986257b1d9deb90f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:38.869560 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4739f10128d64e2c986257b1d9deb90f" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:38.869719 27243 raft_consensus.cc:3058] T 0fef9aa7153a4902abce522e3ff0df3f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:38.870246 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4739f10128d64e2c986257b1d9deb90f" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:38.874387 27462 leader_election.cc:343] T 4739f10128d64e2c986257b1d9deb90f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:38.875063 27462 leader_election.cc:343] T 4739f10128d64e2c986257b1d9deb90f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:38.875195 27462 leader_election.cc:304] T 4739f10128d64e2c986257b1d9deb90f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:38.875948 29443 raft_consensus.cc:2747] T 4739f10128d64e2c986257b1d9deb90f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:38.873358 27243 raft_consensus.cc:2466] T 0fef9aa7153a4902abce522e3ff0df3f P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:38.880247 27695 ts_tablet_manager.cc:1428] T 0fef9aa7153a4902abce522e3ff0df3f P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20250903 20:05:38.890761 27692 tablet_bootstrap.cc:492] T 69f8eef52bee431a86f0eef45fd11f9f P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:38.890954 27692 ts_tablet_manager.cc:1397] T 69f8eef52bee431a86f0eef45fd11f9f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.036s user 0.000s sys 0.008s
I20250903 20:05:38.891320 27695 tablet_bootstrap.cc:492] T 4739f10128d64e2c986257b1d9deb90f P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:38.891867 27692 raft_consensus.cc:357] T 69f8eef52bee431a86f0eef45fd11f9f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.892134 27692 raft_consensus.cc:383] T 69f8eef52bee431a86f0eef45fd11f9f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:38.892238 27692 raft_consensus.cc:738] T 69f8eef52bee431a86f0eef45fd11f9f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:38.892505 27692 consensus_queue.cc:260] T 69f8eef52bee431a86f0eef45fd11f9f P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.893119 27692 ts_tablet_manager.cc:1428] T 69f8eef52bee431a86f0eef45fd11f9f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:38.893494 27692 tablet_bootstrap.cc:492] T 16549a6d9aa9432eb5cee19f69ce6957 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:38.893808 29490 tablet.cc:1722] T 9e98cd1d9bd246e182e2969d147e68bf P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:38.895787 27692 tablet_bootstrap.cc:654] T 16549a6d9aa9432eb5cee19f69ce6957 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:38.902760 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet d41de82304bf453fab8b15d96cf32a74
W20250903 20:05:38.902846 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1355ab18761f4ab894d10d95abe5fec5 consensus state has no leader
I20250903 20:05:38.903957 27347 tablet_service.cc:1515] Processing DeleteTablet for tablet d41de82304bf453fab8b15d96cf32a74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:38.904816 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet d41de82304bf453fab8b15d96cf32a74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:38.905829 27216 tablet_service.cc:1515] Processing DeleteTablet for tablet d41de82304bf453fab8b15d96cf32a74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:38.912851 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e1253db17bbc45d0a1f22c8ba8ea96c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:38.913473 27692 tablet_bootstrap.cc:492] T 16549a6d9aa9432eb5cee19f69ce6957 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:38.913610 27692 ts_tablet_manager.cc:1397] T 16549a6d9aa9432eb5cee19f69ce6957 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.003s
W20250903 20:05:38.913584 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1253db17bbc45d0a1f22c8ba8ea96c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:38.914387 27692 raft_consensus.cc:357] T 16549a6d9aa9432eb5cee19f69ce6957 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.914563 27692 raft_consensus.cc:383] T 16549a6d9aa9432eb5cee19f69ce6957 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:38.914610 27692 raft_consensus.cc:738] T 16549a6d9aa9432eb5cee19f69ce6957 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:38.914803 27692 consensus_queue.cc:260] T 16549a6d9aa9432eb5cee19f69ce6957 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.915352 27692 ts_tablet_manager.cc:1428] T 16549a6d9aa9432eb5cee19f69ce6957 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:38.915653 27692 tablet_bootstrap.cc:492] T 615f42e95bb24c8480f73d371fd63b9c P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:38.915728 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_76c9c4d442e7410990e7b588771507b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:38.916321 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_76c9c4d442e7410990e7b588771507b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:38.917448 27692 tablet_bootstrap.cc:654] T 615f42e95bb24c8480f73d371fd63b9c P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:38.917750 29371 raft_consensus.cc:491] T 1bf570855b204b96acd070587c434463 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:38.917887 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7ced6983993143f48ffb33fe2b19e03c" } modify_external_catalogs: true
I20250903 20:05:38.917894 29371 raft_consensus.cc:513] T 1bf570855b204b96acd070587c434463 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.918059 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7ced6983993143f48ffb33fe2b19e03c" } modify_external_catalogs: true
I20250903 20:05:38.918563 29371 leader_election.cc:290] T 1bf570855b204b96acd070587c434463 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:38.921685 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bf570855b204b96acd070587c434463" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:38.922027 27064 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c7086111ba3d4f73b8cf50389b3ac155" } modify_external_catalogs: true
I20250903 20:05:38.922580 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bf570855b204b96acd070587c434463" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:38.922822 27462 leader_election.cc:343] T 1bf570855b204b96acd070587c434463 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:38.922190 27064 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c7086111ba3d4f73b8cf50389b3ac155" } modify_external_catalogs: true
W20250903 20:05:38.923275 27462 leader_election.cc:343] T 1bf570855b204b96acd070587c434463 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:38.923372 27462 leader_election.cc:304] T 1bf570855b204b96acd070587c434463 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:38.923753 29371 raft_consensus.cc:2747] T 1bf570855b204b96acd070587c434463 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:38.925365 29371 raft_consensus.cc:491] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:38.925540 29371 raft_consensus.cc:513] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:38.926158 29371 leader_election.cc:290] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:38.927299 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d4b8d4bf29e408cbdf9557c8b0d8a4b" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:38.927670 29455 raft_consensus.cc:491] T 2385d3006f90486890ebe43e16a53d7a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:38.927860 29455 raft_consensus.cc:513] T 2385d3006f90486890ebe43e16a53d7a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:38.928100 27462 leader_election.cc:343] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:38.928352 27692 tablet_bootstrap.cc:492] T 615f42e95bb24c8480f73d371fd63b9c P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:38.928575 27692 ts_tablet_manager.cc:1397] T 615f42e95bb24c8480f73d371fd63b9c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20250903 20:05:38.928582 29455 leader_election.cc:290] T 2385d3006f90486890ebe43e16a53d7a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:38.929867 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2385d3006f90486890ebe43e16a53d7a" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:38.930087 27385 raft_consensus.cc:2466] T 2385d3006f90486890ebe43e16a53d7a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:38.929982 27692 raft_consensus.cc:357] T 615f42e95bb24c8480f73d371fd63b9c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.930320 27692 raft_consensus.cc:383] T 615f42e95bb24c8480f73d371fd63b9c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:38.930469 27692 raft_consensus.cc:738] T 615f42e95bb24c8480f73d371fd63b9c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:38.930657 27184 leader_election.cc:304] T 2385d3006f90486890ebe43e16a53d7a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:38.930820 27692 consensus_queue.cc:260] T 615f42e95bb24c8480f73d371fd63b9c P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.930977 29455 raft_consensus.cc:2802] T 2385d3006f90486890ebe43e16a53d7a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:38.931066 29455 raft_consensus.cc:491] T 2385d3006f90486890ebe43e16a53d7a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:38.931126 29455 raft_consensus.cc:3058] T 2385d3006f90486890ebe43e16a53d7a P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:38.931785 27692 ts_tablet_manager.cc:1428] T 615f42e95bb24c8480f73d371fd63b9c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250903 20:05:38.932258 27692 tablet_bootstrap.cc:492] T 733f8024e1fa401ca9614a97db91a19b P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:38.934446 29455 raft_consensus.cc:513] T 2385d3006f90486890ebe43e16a53d7a P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.935518 29455 leader_election.cc:290] T 2385d3006f90486890ebe43e16a53d7a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:38.935688 27692 tablet_bootstrap.cc:654] T 733f8024e1fa401ca9614a97db91a19b P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:38.935745 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2385d3006f90486890ebe43e16a53d7a" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:38.935911 27385 raft_consensus.cc:3058] T 2385d3006f90486890ebe43e16a53d7a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:38.939152 27385 raft_consensus.cc:2466] T 2385d3006f90486890ebe43e16a53d7a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:38.939883 27693 ts_tablet_manager.cc:1428] T e2441aa3c2c748bba5360259fb663c81 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.073s user 0.001s sys 0.001s
I20250903 20:05:38.940214 27184 leader_election.cc:304] T 2385d3006f90486890ebe43e16a53d7a P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:38.943895 29461 raft_consensus.cc:2802] T 2385d3006f90486890ebe43e16a53d7a P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:38.944042 29461 raft_consensus.cc:695] T 2385d3006f90486890ebe43e16a53d7a P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:38.944341 29461 consensus_queue.cc:237] T 2385d3006f90486890ebe43e16a53d7a P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.943501 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d4b8d4bf29e408cbdf9557c8b0d8a4b" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:38.950155 27462 leader_election.cc:343] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:38.950453 27462 leader_election.cc:304] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:38.950839 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2385d3006f90486890ebe43e16a53d7a" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:38.951412 27516 raft_consensus.cc:2466] T 2385d3006f90486890ebe43e16a53d7a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:38.951457 27510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2385d3006f90486890ebe43e16a53d7a" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:38.951687 27510 raft_consensus.cc:3058] T 2385d3006f90486890ebe43e16a53d7a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:38.951921 29371 raft_consensus.cc:2747] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:38.953228 29540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 20:05:38.955222 27510 raft_consensus.cc:2466] T 2385d3006f90486890ebe43e16a53d7a P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:38.957537 27695 tablet_bootstrap.cc:654] T 4739f10128d64e2c986257b1d9deb90f P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:38.962396 27685 ts_tablet_manager.cc:1918] T e877d56bbbfa4c5a92d025d0311ce5c6 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:38.962626 27685 log.cc:1199] T e877d56bbbfa4c5a92d025d0311ce5c6 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/e877d56bbbfa4c5a92d025d0311ce5c6
I20250903 20:05:38.963114 27685 ts_tablet_manager.cc:1939] T e877d56bbbfa4c5a92d025d0311ce5c6 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:38.963558 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1038) took 4892 ms. Trace:
I20250903 20:05:38.963728 27685 rpcz_store.cc:276] 0903 20:05:34.071392 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.072376 (+ 984us) service_pool.cc:225] Handling call
0903 20:05:38.789535 (+4717159us) ts_tablet_manager.cc:1145] Deleting tablet e877d56bbbfa4c5a92d025d0311ce5c6
0903 20:05:38.789545 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.958719 (+169174us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:38.961703 (+ 2984us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:38.963541 (+ 1838us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4703110}
I20250903 20:05:38.964295 27685 tablet_replica.cc:331] T 200c06f296b94eab94f35f312c5a443b P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:38.964833 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet e877d56bbbfa4c5a92d025d0311ce5c6 (table default.Table_d9105c35c28247f1a71866f2096d93f2 [id=33b22e921daa4ccf9406155bac438d08]) successfully deleted
I20250903 20:05:38.965399 27685 raft_consensus.cc:2241] T 200c06f296b94eab94f35f312c5a443b P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:38.965565 27685 raft_consensus.cc:2270] T 200c06f296b94eab94f35f312c5a443b P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:38.966090 27685 ts_tablet_manager.cc:1905] T 200c06f296b94eab94f35f312c5a443b P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:38.967288 29405 consensus_queue.cc:1035] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:38.972361 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_46fddb6b15134d34a8a8702787f38259"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:38.973048 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46fddb6b15134d34a8a8702787f38259 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:38.975500 29371 raft_consensus.cc:491] T f9375fff5f5945078c4412aa21c06f0d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:38.975723 29371 raft_consensus.cc:513] T f9375fff5f5945078c4412aa21c06f0d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:38.976629 29371 leader_election.cc:290] T f9375fff5f5945078c4412aa21c06f0d P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:38.977316 27685 ts_tablet_manager.cc:1918] T 200c06f296b94eab94f35f312c5a443b P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:38.977643 27685 log.cc:1199] T 200c06f296b94eab94f35f312c5a443b P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/200c06f296b94eab94f35f312c5a443b
I20250903 20:05:38.977811 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9375fff5f5945078c4412aa21c06f0d" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:38.978147 27685 ts_tablet_manager.cc:1939] T 200c06f296b94eab94f35f312c5a443b P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:38.978674 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1039) took 4907 ms. Trace:
I20250903 20:05:38.978821 27685 rpcz_store.cc:276] 0903 20:05:34.071487 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.072439 (+ 952us) service_pool.cc:225] Handling call
0903 20:05:38.964246 (+4891807us) ts_tablet_manager.cc:1145] Deleting tablet 200c06f296b94eab94f35f312c5a443b
0903 20:05:38.964256 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.969199 (+ 4943us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:38.977140 (+ 7941us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:38.978657 (+ 1517us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4877717}
I20250903 20:05:38.978843 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9375fff5f5945078c4412aa21c06f0d" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:38.979470 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1040) took 4907 ms. Trace:
I20250903 20:05:38.979413 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 200c06f296b94eab94f35f312c5a443b (table default.Table_c5abd2620f3f45628ce2531cbcb81d9d [id=37960c5e67734caa951c9a151b0ece51]) successfully deleted
I20250903 20:05:38.979620 27685 rpcz_store.cc:276] 0903 20:05:34.071573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.072489 (+ 916us) service_pool.cc:225] Handling call
0903 20:05:38.979367 (+4906878us) ts_tablet_manager.cc:1145] Deleting tablet 7e2f98f760cf4df4bc2a10530e2157c6
0903 20:05:38.979378 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.979449 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4892759}
I20250903 20:05:38.980108 27685 tablet_replica.cc:331] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
W20250903 20:05:38.980141 27052 catalog_manager.cc:4895] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 7e2f98f760cf4df4bc2a10530e2157c6 because the tablet was not found. No further retry: Not found: Tablet not found: 7e2f98f760cf4df4bc2a10530e2157c6
I20250903 20:05:38.980475 27685 raft_consensus.cc:2241] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
W20250903 20:05:38.980624 27462 leader_election.cc:343] T f9375fff5f5945078c4412aa21c06f0d P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:38.980885 27462 leader_election.cc:343] T f9375fff5f5945078c4412aa21c06f0d P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:38.981026 27462 leader_election.cc:304] T f9375fff5f5945078c4412aa21c06f0d P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:38.981434 27685 raft_consensus.cc:2270] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:38.981752 29170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250903 20:05:38.982462 27685 ts_tablet_manager.cc:1905] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:38.984068 29437 raft_consensus.cc:2747] T f9375fff5f5945078c4412aa21c06f0d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:38.991248 27685 ts_tablet_manager.cc:1918] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:38.996475 27685 log.cc:1199] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/4c18a98d5c0f4a43ae91c9130047b681
I20250903 20:05:38.997078 27685 ts_tablet_manager.cc:1939] T 4c18a98d5c0f4a43ae91c9130047b681 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:38.997599 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1041) took 4925 ms. Trace:
I20250903 20:05:38.997792 27685 rpcz_store.cc:276] 0903 20:05:34.071682 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.072540 (+ 858us) service_pool.cc:225] Handling call
0903 20:05:38.980058 (+4907518us) ts_tablet_manager.cc:1145] Deleting tablet 4c18a98d5c0f4a43ae91c9130047b681
0903 20:05:38.980068 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.987702 (+ 7634us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:38.991086 (+ 3384us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:38.997575 (+ 6489us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4893373}
I20250903 20:05:38.998539 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 4c18a98d5c0f4a43ae91c9130047b681 (table default.Table_16b62656e76142339c0a42ca5a7f5e6b [id=189c3642e92c43759dee94f23ef1aec3]) successfully deleted
I20250903 20:05:38.998670 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1042) took 4926 ms. Trace:
I20250903 20:05:38.998867 27685 rpcz_store.cc:276] 0903 20:05:34.071772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.072593 (+ 821us) service_pool.cc:225] Handling call
0903 20:05:38.998553 (+4925960us) ts_tablet_manager.cc:1145] Deleting tablet 53881ad30bc843839008ace4012a3f70
0903 20:05:38.998569 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:38.998657 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4911753}
I20250903 20:05:38.999356 27685 tablet_replica.cc:331] T bf459db3327640f88574fca7b5f2ca98 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
W20250903 20:05:39.000468 27052 catalog_manager.cc:4895] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 53881ad30bc843839008ace4012a3f70 because the tablet was not found. No further retry: Not found: Tablet not found: 53881ad30bc843839008ace4012a3f70
I20250903 20:05:39.000806 27685 raft_consensus.cc:2241] T bf459db3327640f88574fca7b5f2ca98 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:39.001112 27685 raft_consensus.cc:2270] T bf459db3327640f88574fca7b5f2ca98 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:39.002007 27685 ts_tablet_manager.cc:1905] T bf459db3327640f88574fca7b5f2ca98 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:39.005878 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8808c2668707437f9db006c7f9a91e70 consensus state has no leader
I20250903 20:05:39.007704 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_584007f8f96047bfa933decba8b5d048" } new_table_name: "default.Table_88e81f58d0ad46289e7b1f7053a4cc47" modify_external_catalogs: true
W20250903 20:05:39.009945 27053 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 421dc8af40c94a89bc372e8779872a4e (table default.Table_fd33384721334ad0b70806e8e70a60f2 [id=f78623e5b0444dac9c633558aba62f5e], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 421dc8af40c94a89bc372e8779872a4e consensus state has no leader
I20250903 20:05:39.008299 27695 tablet_bootstrap.cc:492] T 4739f10128d64e2c986257b1d9deb90f P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:39.013192 27684 ts_tablet_manager.cc:1918] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:39.013294 27695 ts_tablet_manager.cc:1397] T 4739f10128d64e2c986257b1d9deb90f P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.122s user 0.001s sys 0.005s
I20250903 20:05:39.014101 27695 raft_consensus.cc:357] T 4739f10128d64e2c986257b1d9deb90f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.016330 27695 raft_consensus.cc:383] T 4739f10128d64e2c986257b1d9deb90f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.016611 27695 raft_consensus.cc:738] T 4739f10128d64e2c986257b1d9deb90f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.017040 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 8e12f4c420874ab382d433982bddbf05
I20250903 20:05:39.017124 27695 consensus_queue.cc:260] T 4739f10128d64e2c986257b1d9deb90f P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.017697 27695 ts_tablet_manager.cc:1428] T 4739f10128d64e2c986257b1d9deb90f P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250903 20:05:39.020390 29170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250903 20:05:39.022490 29170 hybrid_clock.cc:648] HybridClock initialized: now 1756929939022395 us; error 99 us; skew 500 ppm
I20250903 20:05:39.022997 29170 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 20:05:39.023064 27216 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e12f4c420874ab382d433982bddbf05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:39.023449 27685 ts_tablet_manager.cc:1918] T bf459db3327640f88574fca7b5f2ca98 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:39.025169 27347 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e12f4c420874ab382d433982bddbf05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:39.026132 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_1dc6317976b84af1b853bbb3825bcf82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:39.026834 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1dc6317976b84af1b853bbb3825bcf82 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:39.030754 26890 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 117239us
I20250903 20:05:39.041735 27684 log.cc:1199] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/eaa11d17303142ff935705c16f0b9690
I20250903 20:05:39.042433 27684 ts_tablet_manager.cc:1939] T eaa11d17303142ff935705c16f0b9690 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:39.047001 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_5b8fcb7885294903a6849c86ad91ccca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:39.047642 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b8fcb7885294903a6849c86ad91ccca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:39.022084 29371 raft_consensus.cc:491] T 5fb2426aa01e47598f268d0a90475271 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.019124 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e12f4c420874ab382d433982bddbf05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:39.050417 29371 raft_consensus.cc:513] T 5fb2426aa01e47598f268d0a90475271 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.051203 29371 leader_election.cc:290] T 5fb2426aa01e47598f268d0a90475271 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:39.051920 27376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5fb2426aa01e47598f268d0a90475271" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:39.057890 29371 consensus_queue.cc:1035] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:39.065728 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1133) took 3784 ms. Trace:
I20250903 20:05:39.068492 27684 rpcz_store.cc:276] 0903 20:05:35.281105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.285453 (+ 4348us) service_pool.cc:225] Handling call
0903 20:05:37.877844 (+2592391us) ts_tablet_manager.cc:1145] Deleting tablet eaa11d17303142ff935705c16f0b9690
0903 20:05:37.877863 (+ 19us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:37.919423 (+ 41560us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.013066 (+1093643us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.065698 (+ 52632us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11363,"tablet-delete.queue_time_us":2590056}
I20250903 20:05:39.072587 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet eaa11d17303142ff935705c16f0b9690 (table default.Table_857612f25ca54f8b8d0ea31485d96a5d [id=ddebc54a9f9849478f0d1606dfe3bc96]) successfully deleted
W20250903 20:05:39.067945 27462 leader_election.cc:343] T 5fb2426aa01e47598f268d0a90475271 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.074949 29463 tablet.cc:1722] T c0343d2226e345669d42375de2545986 P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:39.075058 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_0b669e82cd084404b53110469f908cd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:39.075750 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b669e82cd084404b53110469f908cd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:39.080098 29443 raft_consensus.cc:491] T 16549a6d9aa9432eb5cee19f69ce6957 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.080297 29443 raft_consensus.cc:513] T 16549a6d9aa9432eb5cee19f69ce6957 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.080972 29443 leader_election.cc:290] T 16549a6d9aa9432eb5cee19f69ce6957 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:39.081570 27692 tablet_bootstrap.cc:492] T 733f8024e1fa401ca9614a97db91a19b P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:39.081699 27692 ts_tablet_manager.cc:1397] T 733f8024e1fa401ca9614a97db91a19b P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.150s user 0.019s sys 0.042s
I20250903 20:05:39.082531 27692 raft_consensus.cc:357] T 733f8024e1fa401ca9614a97db91a19b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.082688 27692 raft_consensus.cc:383] T 733f8024e1fa401ca9614a97db91a19b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.082741 27692 raft_consensus.cc:738] T 733f8024e1fa401ca9614a97db91a19b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.082911 27692 consensus_queue.cc:260] T 733f8024e1fa401ca9614a97db91a19b P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.083400 27692 ts_tablet_manager.cc:1428] T 733f8024e1fa401ca9614a97db91a19b P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:39.089095 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16549a6d9aa9432eb5cee19f69ce6957" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:39.089751 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet d1e89ad94adb4de8b3738e30a84a8f79
I20250903 20:05:39.090402 27064 master_service.cc:948] ReplaceTablet: received request to replace tablet 1fce05bd4e6c425e952001ffdc25c78a from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:39.091600 27347 tablet_service.cc:1515] Processing DeleteTablet for tablet d1e89ad94adb4de8b3738e30a84a8f79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:39.092188 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9ef10e3c467a4966ad936a384e26acb9 consensus state has no leader
I20250903 20:05:39.093065 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet d1e89ad94adb4de8b3738e30a84a8f79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:39.094722 29170 webserver.cc:480] Webserver started at http://127.26.65.125:41613/ using document root <none> and password file <none>
I20250903 20:05:39.096038 29170 fs_manager.cc:362] Metadata directory not provided
I20250903 20:05:39.096179 29170 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 20:05:39.097031 29490 tablet.cc:1722] T c0343d2226e345669d42375de2545986 P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 20:05:39.102535 27693 tablet_bootstrap.cc:492] T 249362a0b8304719863bcb82c47273b3 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:39.104089 27693 tablet_bootstrap.cc:654] T 249362a0b8304719863bcb82c47273b3 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:39.106359 27462 leader_election.cc:343] T 16549a6d9aa9432eb5cee19f69ce6957 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.110648 29170 fs_manager.cc:714] Time spent opening directory manager: real 0.012s user 0.003s sys 0.005s
I20250903 20:05:39.120266 29371 raft_consensus.cc:491] T 7d5a3fbe93e44c228222ab1b1e78c187 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.120647 29371 raft_consensus.cc:513] T 7d5a3fbe93e44c228222ab1b1e78c187 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.121556 29371 leader_election.cc:290] T 7d5a3fbe93e44c228222ab1b1e78c187 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:39.122874 29443 consensus_queue.cc:1035] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:39.124336 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d5a3fbe93e44c228222ab1b1e78c187" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:39.128445 27687 raft_consensus.cc:2241] T f326ec776ba446be8d5dda5c1dc5b6ec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:39.129093 27687 raft_consensus.cc:2270] T f326ec776ba446be8d5dda5c1dc5b6ec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:39.130075 27693 tablet_bootstrap.cc:492] T 249362a0b8304719863bcb82c47273b3 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:39.130487 27693 ts_tablet_manager.cc:1397] T 249362a0b8304719863bcb82c47273b3 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.005s
I20250903 20:05:39.131403 27693 raft_consensus.cc:357] T 249362a0b8304719863bcb82c47273b3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.131672 27693 raft_consensus.cc:383] T 249362a0b8304719863bcb82c47273b3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.131778 27693 raft_consensus.cc:738] T 249362a0b8304719863bcb82c47273b3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.132081 27693 consensus_queue.cc:260] T 249362a0b8304719863bcb82c47273b3 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.133003 27693 ts_tablet_manager.cc:1428] T 249362a0b8304719863bcb82c47273b3 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:39.134053 27693 tablet_bootstrap.cc:492] T 3919f7f95a954f59b68f56d0051c0109 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
W20250903 20:05:39.137058 27462 leader_election.cc:343] T 7d5a3fbe93e44c228222ab1b1e78c187 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.137140 27693 tablet_bootstrap.cc:654] T 3919f7f95a954f59b68f56d0051c0109 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.045324 29452 raft_consensus.cc:491] T a38f65e992884635aae9942681c0dc4f P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e72ef1dbcc5143cf9f04a808f101c80f)
I20250903 20:05:39.137814 29452 raft_consensus.cc:513] T a38f65e992884635aae9942681c0dc4f P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:39.051970 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5fb2426aa01e47598f268d0a90475271" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:39.139210 29452 leader_election.cc:290] T a38f65e992884635aae9942681c0dc4f P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:39.057672 27695 tablet_bootstrap.cc:492] T 1bf570855b204b96acd070587c434463 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:39.091879 27216 tablet_service.cc:1515] Processing DeleteTablet for tablet d1e89ad94adb4de8b3738e30a84a8f79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:39.095026 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16549a6d9aa9432eb5cee19f69ce6957" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:39.105108 29518 tablet.cc:1722] T c0343d2226e345669d42375de2545986 P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250903 20:05:39.140076 27184 leader_election.cc:343] T a38f65e992884635aae9942681c0dc4f P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: a38f65e992884635aae9942681c0dc4f
I20250903 20:05:39.142405 27695 tablet_bootstrap.cc:654] T 1bf570855b204b96acd070587c434463 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.125553 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d5a3fbe93e44c228222ab1b1e78c187" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:39.144918 27685 log.cc:1199] T bf459db3327640f88574fca7b5f2ca98 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/bf459db3327640f88574fca7b5f2ca98
W20250903 20:05:39.145200 27462 leader_election.cc:343] T 5fb2426aa01e47598f268d0a90475271 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.145606 27695 tablet_bootstrap.cc:492] T 1bf570855b204b96acd070587c434463 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
W20250903 20:05:39.145627 27183 leader_election.cc:343] T a38f65e992884635aae9942681c0dc4f P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: a38f65e992884635aae9942681c0dc4f
I20250903 20:05:39.145816 27685 ts_tablet_manager.cc:1939] T bf459db3327640f88574fca7b5f2ca98 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:39.145866 27183 leader_election.cc:304] T a38f65e992884635aae9942681c0dc4f P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:39.146129 29452 raft_consensus.cc:2747] T a38f65e992884635aae9942681c0dc4f P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:39.146459 27462 leader_election.cc:304] T 5fb2426aa01e47598f268d0a90475271 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:39.147022 27695 ts_tablet_manager.cc:1397] T 1bf570855b204b96acd070587c434463 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.089s user 0.001s sys 0.003s
I20250903 20:05:39.146835 27692 tablet_bootstrap.cc:492] T ef02f91c0964446daf62ac1dde0330ff P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:39.147823 27695 raft_consensus.cc:357] T 1bf570855b204b96acd070587c434463 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.148041 27695 raft_consensus.cc:383] T 1bf570855b204b96acd070587c434463 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.148126 27695 raft_consensus.cc:738] T 1bf570855b204b96acd070587c434463 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.148363 27695 consensus_queue.cc:260] T 1bf570855b204b96acd070587c434463 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.148854 27692 tablet_bootstrap.cc:654] T ef02f91c0964446daf62ac1dde0330ff P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.148864 27695 ts_tablet_manager.cc:1428] T 1bf570855b204b96acd070587c434463 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:39.149899 27695 tablet_bootstrap.cc:492] T 615f42e95bb24c8480f73d371fd63b9c P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:39.146950 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1134) took 3825 ms. Trace:
I20250903 20:05:39.150578 27684 rpcz_store.cc:276] 0903 20:05:35.322016 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.348550 (+ 26534us) service_pool.cc:225] Handling call
0903 20:05:39.088556 (+3740006us) ts_tablet_manager.cc:1145] Deleting tablet eb878263eda9447e870a782907fe6a8d
0903 20:05:39.146818 (+ 58262us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.146931 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3739737}
W20250903 20:05:39.151407 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet eb878263eda9447e870a782907fe6a8d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb878263eda9447e870a782907fe6a8d already in progress: creating tablet
I20250903 20:05:39.151899 27684 tablet_replica.cc:331] T aa9f5e0964b348b7af184f60cd145666 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:39.152361 27684 raft_consensus.cc:2241] T aa9f5e0964b348b7af184f60cd145666 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:39.152741 27684 raft_consensus.cc:2270] T aa9f5e0964b348b7af184f60cd145666 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:39.153424 27692 tablet_bootstrap.cc:492] T ef02f91c0964446daf62ac1dde0330ff P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:39.153573 27692 ts_tablet_manager.cc:1397] T ef02f91c0964446daf62ac1dde0330ff P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.002s
I20250903 20:05:39.153842 27684 ts_tablet_manager.cc:1905] T aa9f5e0964b348b7af184f60cd145666 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:39.154394 27692 raft_consensus.cc:357] T ef02f91c0964446daf62ac1dde0330ff P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.154498 29452 raft_consensus.cc:491] T 1223743918814d88b401782ca4b521dd P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51578fb7bceb4e119b3ab1e58c4f5528)
I20250903 20:05:39.154587 27692 raft_consensus.cc:383] T ef02f91c0964446daf62ac1dde0330ff P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.154660 27692 raft_consensus.cc:738] T ef02f91c0964446daf62ac1dde0330ff P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.154660 29452 raft_consensus.cc:513] T 1223743918814d88b401782ca4b521dd P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.154868 27692 consensus_queue.cc:260] T ef02f91c0964446daf62ac1dde0330ff P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.155254 29452 leader_election.cc:290] T 1223743918814d88b401782ca4b521dd P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:39.155357 27692 ts_tablet_manager.cc:1428] T ef02f91c0964446daf62ac1dde0330ff P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:39.155637 27692 tablet_bootstrap.cc:492] T 5c42d84a6a5649208b0539a10da21aec P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
W20250903 20:05:39.156708 27184 leader_election.cc:343] T 1223743918814d88b401782ca4b521dd P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 1223743918814d88b401782ca4b521dd
W20250903 20:05:39.157065 27183 leader_election.cc:343] T 1223743918814d88b401782ca4b521dd P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 1223743918814d88b401782ca4b521dd
I20250903 20:05:39.157305 27692 tablet_bootstrap.cc:654] T 5c42d84a6a5649208b0539a10da21aec P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.157406 27183 leader_election.cc:304] T 1223743918814d88b401782ca4b521dd P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:39.157845 29443 raft_consensus.cc:2747] T 5fb2426aa01e47598f268d0a90475271 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:39.157883 29452 raft_consensus.cc:2747] T 1223743918814d88b401782ca4b521dd P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:39.143086 27687 ts_tablet_manager.cc:1905] T f326ec776ba446be8d5dda5c1dc5b6ec P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:39.159842 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1053) took 4780 ms. Trace:
W20250903 20:05:39.160065 27462 leader_election.cc:343] T 16549a6d9aa9432eb5cee19f69ce6957 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.160423 27462 leader_election.cc:304] T 16549a6d9aa9432eb5cee19f69ce6957 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:39.160706 27685 rpcz_store.cc:276] 0903 20:05:34.379423 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.379475 (+ 52us) service_pool.cc:225] Handling call
0903 20:05:38.999292 (+4619817us) ts_tablet_manager.cc:1145] Deleting tablet bf459db3327640f88574fca7b5f2ca98
0903 20:05:38.999302 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.018946 (+ 19644us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.023292 (+ 4346us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.159797 (+136505us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4619469}
I20250903 20:05:39.161545 29443 raft_consensus.cc:2747] T 16549a6d9aa9432eb5cee19f69ce6957 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:39.162851 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet bf459db3327640f88574fca7b5f2ca98 (table default.Table_f6ddd620f29a4333a8f71f99316e1a8d [id=b6932927b05f46be967419e342532b3f]) successfully deleted
I20250903 20:05:39.163666 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1052) took 4785 ms. Trace:
W20250903 20:05:39.162142 27462 leader_election.cc:343] T 7d5a3fbe93e44c228222ab1b1e78c187 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.166028 27462 leader_election.cc:304] T 7d5a3fbe93e44c228222ab1b1e78c187 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:39.166318 27685 rpcz_store.cc:276] 0903 20:05:34.378705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.379125 (+ 420us) service_pool.cc:225] Handling call
0903 20:05:39.163513 (+4784388us) ts_tablet_manager.cc:1145] Deleting tablet f2629f46704e4a0fa7922533928ca2ed
0903 20:05:39.163529 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.163623 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4782963}
W20250903 20:05:39.167791 27052 catalog_manager.cc:4895] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet f2629f46704e4a0fa7922533928ca2ed because the tablet was not found. No further retry: Not found: Tablet not found: f2629f46704e4a0fa7922533928ca2ed
I20250903 20:05:39.167958 27692 tablet_bootstrap.cc:492] T 5c42d84a6a5649208b0539a10da21aec P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:39.166862 27695 tablet_bootstrap.cc:654] T 615f42e95bb24c8480f73d371fd63b9c P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.168097 27692 ts_tablet_manager.cc:1397] T 5c42d84a6a5649208b0539a10da21aec P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.001s
I20250903 20:05:39.170805 29452 raft_consensus.cc:491] T 1bf570855b204b96acd070587c434463 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.171015 29452 raft_consensus.cc:513] T 1bf570855b204b96acd070587c434463 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.171898 29452 leader_election.cc:290] T 1bf570855b204b96acd070587c434463 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:39.172858 29443 raft_consensus.cc:2747] T 7d5a3fbe93e44c228222ab1b1e78c187 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:39.173141 27692 raft_consensus.cc:357] T 5c42d84a6a5649208b0539a10da21aec P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.173233 27510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bf570855b204b96acd070587c434463" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:39.173470 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bf570855b204b96acd070587c434463" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:39.173661 27510 raft_consensus.cc:2466] T 1bf570855b204b96acd070587c434463 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:39.173337 27692 raft_consensus.cc:383] T 5c42d84a6a5649208b0539a10da21aec P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.173957 27692 raft_consensus.cc:738] T 5c42d84a6a5649208b0539a10da21aec P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
W20250903 20:05:39.174463 27184 leader_election.cc:343] T 1bf570855b204b96acd070587c434463 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.174707 27183 leader_election.cc:304] T 1bf570855b204b96acd070587c434463 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:39.175187 29452 raft_consensus.cc:2802] T 1bf570855b204b96acd070587c434463 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:39.175380 29452 raft_consensus.cc:491] T 1bf570855b204b96acd070587c434463 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:39.175469 29452 raft_consensus.cc:3058] T 1bf570855b204b96acd070587c434463 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:39.175688 27692 consensus_queue.cc:260] T 5c42d84a6a5649208b0539a10da21aec P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.176539 27692 ts_tablet_manager.cc:1428] T 5c42d84a6a5649208b0539a10da21aec P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250903 20:05:39.176837 27692 tablet_bootstrap.cc:492] T 9c77bc0e8bb34712bf8109cadbc0fc06 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:39.178861 27692 tablet_bootstrap.cc:654] T 9c77bc0e8bb34712bf8109cadbc0fc06 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.180658 27685 tablet_replica.cc:331] T a38f65e992884635aae9942681c0dc4f P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
W20250903 20:05:39.181349 27184 consensus_peers.cc:489] T f326ec776ba446be8d5dda5c1dc5b6ec P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:39.181511 29461 raft_consensus.cc:491] T 4739f10128d64e2c986257b1d9deb90f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.183967 29461 raft_consensus.cc:513] T 4739f10128d64e2c986257b1d9deb90f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.184758 29461 leader_election.cc:290] T 4739f10128d64e2c986257b1d9deb90f P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:39.186203 27685 raft_consensus.cc:2241] T a38f65e992884635aae9942681c0dc4f P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:39.187371 27685 raft_consensus.cc:2270] T a38f65e992884635aae9942681c0dc4f P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:39.188712 27685 ts_tablet_manager.cc:1905] T a38f65e992884635aae9942681c0dc4f P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:39.191675 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4739f10128d64e2c986257b1d9deb90f" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:39.192426 27184 leader_election.cc:343] T 4739f10128d64e2c986257b1d9deb90f P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.194329 27687 ts_tablet_manager.cc:1918] T f326ec776ba446be8d5dda5c1dc5b6ec P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:39.196732 27692 tablet_bootstrap.cc:492] T 9c77bc0e8bb34712bf8109cadbc0fc06 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:39.196938 27692 ts_tablet_manager.cc:1397] T 9c77bc0e8bb34712bf8109cadbc0fc06 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20250903 20:05:39.197600 27687 log.cc:1199] T f326ec776ba446be8d5dda5c1dc5b6ec P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/f326ec776ba446be8d5dda5c1dc5b6ec
I20250903 20:05:39.197829 27692 raft_consensus.cc:357] T 9c77bc0e8bb34712bf8109cadbc0fc06 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.198025 27692 raft_consensus.cc:383] T 9c77bc0e8bb34712bf8109cadbc0fc06 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.198120 27692 raft_consensus.cc:738] T 9c77bc0e8bb34712bf8109cadbc0fc06 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.199659 27692 consensus_queue.cc:260] T 9c77bc0e8bb34712bf8109cadbc0fc06 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.200232 27692 ts_tablet_manager.cc:1428] T 9c77bc0e8bb34712bf8109cadbc0fc06 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250903 20:05:39.200562 27692 tablet_bootstrap.cc:492] T 36e1061915574407b6e7695be8b1e55a P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:39.202666 27684 ts_tablet_manager.cc:1918] T aa9f5e0964b348b7af184f60cd145666 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:39.204289 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4739f10128d64e2c986257b1d9deb90f" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:39.205484 27684 log.cc:1199] T aa9f5e0964b348b7af184f60cd145666 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/aa9f5e0964b348b7af184f60cd145666
I20250903 20:05:39.205549 27516 raft_consensus.cc:2466] T 4739f10128d64e2c986257b1d9deb90f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:39.205857 27684 ts_tablet_manager.cc:1939] T aa9f5e0964b348b7af184f60cd145666 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:39.208740 29452 raft_consensus.cc:513] T 1bf570855b204b96acd070587c434463 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.209370 27692 tablet_bootstrap.cc:654] T 36e1061915574407b6e7695be8b1e55a P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.210708 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bf570855b204b96acd070587c434463" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
W20250903 20:05:39.212059 27184 leader_election.cc:343] T 1bf570855b204b96acd070587c434463 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.213037 27183 leader_election.cc:304] T 4739f10128d64e2c986257b1d9deb90f P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:39.213613 29411 raft_consensus.cc:2802] T 4739f10128d64e2c986257b1d9deb90f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:39.213714 29411 raft_consensus.cc:491] T 4739f10128d64e2c986257b1d9deb90f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:39.213804 29411 raft_consensus.cc:3058] T 4739f10128d64e2c986257b1d9deb90f P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:39.215126 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1135) took 3858 ms. Trace:
I20250903 20:05:39.215238 27684 rpcz_store.cc:276] 0903 20:05:35.356822 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.357269 (+ 447us) service_pool.cc:225] Handling call
0903 20:05:39.151846 (+3794577us) ts_tablet_manager.cc:1145] Deleting tablet aa9f5e0964b348b7af184f60cd145666
0903 20:05:39.151858 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.159328 (+ 7470us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.200811 (+ 41483us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.215085 (+ 14274us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3794419}
I20250903 20:05:39.215665 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet aa9f5e0964b348b7af184f60cd145666 (table default.Table_14e02c1be9c74bd99f4184faa6753f52 [id=a6dc7550a6d643008c92b3a59ba022ce]) successfully deleted
I20250903 20:05:39.216976 29411 raft_consensus.cc:513] T 4739f10128d64e2c986257b1d9deb90f P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.217653 29411 leader_election.cc:290] T 4739f10128d64e2c986257b1d9deb90f P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:39.217658 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1136) took 3765 ms. Trace:
I20250903 20:05:39.218411 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4739f10128d64e2c986257b1d9deb90f" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:39.217752 27684 rpcz_store.cc:276] 0903 20:05:35.452542 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.452611 (+ 69us) service_pool.cc:225] Handling call
0903 20:05:39.217493 (+3764882us) ts_tablet_manager.cc:1145] Deleting tablet a7f25f00dff7426195ac51214a14022b
0903 20:05:39.217506 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.217621 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3743212}
W20250903 20:05:39.219072 27184 leader_election.cc:343] T 4739f10128d64e2c986257b1d9deb90f P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:39.220229 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet a7f25f00dff7426195ac51214a14022b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7f25f00dff7426195ac51214a14022b already in progress: creating tablet
I20250903 20:05:39.220391 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4739f10128d64e2c986257b1d9deb90f" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:39.220633 27516 raft_consensus.cc:3058] T 4739f10128d64e2c986257b1d9deb90f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:39.220924 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1137) took 3730 ms. Trace:
I20250903 20:05:39.221048 27684 rpcz_store.cc:276] 0903 20:05:35.490888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.490956 (+ 68us) service_pool.cc:225] Handling call
0903 20:05:39.220738 (+3729782us) ts_tablet_manager.cc:1145] Deleting tablet c9d09ecd8fe347d79283fc9a3056e844
0903 20:05:39.220753 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.220868 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3723907}
W20250903 20:05:39.221668 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet c9d09ecd8fe347d79283fc9a3056e844 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9d09ecd8fe347d79283fc9a3056e844 already in progress: creating tablet
I20250903 20:05:39.224063 27516 raft_consensus.cc:2466] T 4739f10128d64e2c986257b1d9deb90f P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:39.225140 27183 leader_election.cc:304] T 4739f10128d64e2c986257b1d9deb90f P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:39.225519 29411 raft_consensus.cc:2802] T 4739f10128d64e2c986257b1d9deb90f P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:39.226501 27684 tablet_replica.cc:331] T 47c1287bff1145a5b2ab6d14dbd608c7 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:39.225807 29411 raft_consensus.cc:695] T 4739f10128d64e2c986257b1d9deb90f P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:39.227183 29411 consensus_queue.cc:237] T 4739f10128d64e2c986257b1d9deb90f P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.228225 27687 ts_tablet_manager.cc:1939] T f326ec776ba446be8d5dda5c1dc5b6ec P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:39.229094 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1158) took 3684 ms. Trace:
I20250903 20:05:39.229200 27687 rpcz_store.cc:276] 0903 20:05:35.545134 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.545237 (+ 103us) service_pool.cc:225] Handling call
0903 20:05:38.757318 (+3212081us) ts_tablet_manager.cc:1145] Deleting tablet f326ec776ba446be8d5dda5c1dc5b6ec
0903 20:05:38.757326 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.167709 (+410383us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.194185 (+ 26476us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.229069 (+ 34884us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3211798}
I20250903 20:05:39.229615 27687 tablet_replica.cc:331] T cb2f50683e904dd39ee6eb128717d38c P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:39.229703 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet f326ec776ba446be8d5dda5c1dc5b6ec (table default.Table_b2a82a2c310140caaeac5d0dcb4bb61a [id=e2548ad469e54a41af7c0181357d2467]) successfully deleted
I20250903 20:05:39.229804 27687 raft_consensus.cc:2241] T cb2f50683e904dd39ee6eb128717d38c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:39.229954 27687 raft_consensus.cc:2270] T cb2f50683e904dd39ee6eb128717d38c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:39.230512 29452 leader_election.cc:290] T 1bf570855b204b96acd070587c434463 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:39.231102 27684 raft_consensus.cc:2241] T 47c1287bff1145a5b2ab6d14dbd608c7 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:39.231509 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bf570855b204b96acd070587c434463" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:39.231714 27692 tablet_bootstrap.cc:492] T 36e1061915574407b6e7695be8b1e55a P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:39.231755 27684 raft_consensus.cc:2270] T 47c1287bff1145a5b2ab6d14dbd608c7 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:39.231864 27516 raft_consensus.cc:3058] T 1bf570855b204b96acd070587c434463 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:39.233222 27684 ts_tablet_manager.cc:1905] T 47c1287bff1145a5b2ab6d14dbd608c7 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:39.236199 27516 raft_consensus.cc:2466] T 1bf570855b204b96acd070587c434463 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:39.236821 27183 leader_election.cc:304] T 1bf570855b204b96acd070587c434463 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:39.237440 29452 raft_consensus.cc:2802] T 1bf570855b204b96acd070587c434463 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:39.237727 29452 raft_consensus.cc:695] T 1bf570855b204b96acd070587c434463 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:39.238066 29452 consensus_queue.cc:237] T 1bf570855b204b96acd070587c434463 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.231890 27692 ts_tablet_manager.cc:1397] T 36e1061915574407b6e7695be8b1e55a P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.008s
I20250903 20:05:39.239645 27692 raft_consensus.cc:357] T 36e1061915574407b6e7695be8b1e55a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.239848 27692 raft_consensus.cc:383] T 36e1061915574407b6e7695be8b1e55a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.239928 27692 raft_consensus.cc:738] T 36e1061915574407b6e7695be8b1e55a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.240154 27692 consensus_queue.cc:260] T 36e1061915574407b6e7695be8b1e55a P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.240669 27692 ts_tablet_manager.cc:1428] T 36e1061915574407b6e7695be8b1e55a P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:39.240960 27692 tablet_bootstrap.cc:492] T 6d13884fff8c43b690836c6a341a2d92 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
W20250903 20:05:39.241387 27321 consensus_peers.cc:489] T b4b15d77d07c4fa7856d9c198cf15450 P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Couldn't send request to peer 89f8f0ee68b940db9ce250c0604f934a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:39.242262 27516 raft_consensus.cc:1273] T b4b15d77d07c4fa7856d9c198cf15450 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Refusing update from remote peer 51578fb7bceb4e119b3ab1e58c4f5528: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 20:05:39.243424 29405 consensus_queue.cc:1035] T b4b15d77d07c4fa7856d9c198cf15450 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:39.250519 29490 tablet.cc:1722] T b4b15d77d07c4fa7856d9c198cf15450 P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250903 20:05:39.251267 29463 tablet.cc:1722] T b4b15d77d07c4fa7856d9c198cf15450 P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250903 20:05:39.251821 29443 raft_consensus.cc:491] T 733f8024e1fa401ca9614a97db91a19b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.251982 29443 raft_consensus.cc:513] T 733f8024e1fa401ca9614a97db91a19b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.252580 29443 leader_election.cc:290] T 733f8024e1fa401ca9614a97db91a19b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:39.254745 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "733f8024e1fa401ca9614a97db91a19b" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:39.255486 27462 leader_election.cc:343] T 733f8024e1fa401ca9614a97db91a19b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.256474 27692 tablet_bootstrap.cc:654] T 6d13884fff8c43b690836c6a341a2d92 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.261901 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_cedba29d5f42445cba418e98e554bb61" } new_table_name: "default.Table_5fbfb89f12b1422688807d10f3aaaabf" modify_external_catalogs: true
I20250903 20:05:39.264945 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_cb5df7f62a6e49ccaf82d85fde99b264"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:39.265232 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_db9c4046409e4dfda92270621470ecd4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:39.265815 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db9c4046409e4dfda92270621470ecd4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:39.271800 27693 tablet_bootstrap.cc:492] T 3919f7f95a954f59b68f56d0051c0109 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:39.272194 27693 ts_tablet_manager.cc:1397] T 3919f7f95a954f59b68f56d0051c0109 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.138s user 0.008s sys 0.016s
I20250903 20:05:39.273025 27693 raft_consensus.cc:357] T 3919f7f95a954f59b68f56d0051c0109 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.273288 27693 raft_consensus.cc:383] T 3919f7f95a954f59b68f56d0051c0109 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.273384 27693 raft_consensus.cc:738] T 3919f7f95a954f59b68f56d0051c0109 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.273654 27693 consensus_queue.cc:260] T 3919f7f95a954f59b68f56d0051c0109 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:39.265820 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb5df7f62a6e49ccaf82d85fde99b264 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:39.274559 27693 ts_tablet_manager.cc:1428] T 3919f7f95a954f59b68f56d0051c0109 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:39.275388 27693 tablet_bootstrap.cc:492] T de20d18d7f264811a0631aa1144aef2c P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:39.277354 27693 tablet_bootstrap.cc:654] T de20d18d7f264811a0631aa1144aef2c P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.282799 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e914faf335d44d019f91bda71758c473" } modify_external_catalogs: true
I20250903 20:05:39.283001 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_e914faf335d44d019f91bda71758c473" } modify_external_catalogs: true
I20250903 20:05:39.285866 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c7ba7021073b42a8b57cb33054f9a141" } new_table_name: "default.Table_26ddae22d4cb461a845574af80122fc1" modify_external_catalogs: true
I20250903 20:05:39.288877 27692 tablet_bootstrap.cc:492] T 6d13884fff8c43b690836c6a341a2d92 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:39.289041 27692 ts_tablet_manager.cc:1397] T 6d13884fff8c43b690836c6a341a2d92 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.004s
I20250903 20:05:39.289821 27692 raft_consensus.cc:357] T 6d13884fff8c43b690836c6a341a2d92 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.290022 27692 raft_consensus.cc:383] T 6d13884fff8c43b690836c6a341a2d92 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.290095 27692 raft_consensus.cc:738] T 6d13884fff8c43b690836c6a341a2d92 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.290377 27692 consensus_queue.cc:260] T 6d13884fff8c43b690836c6a341a2d92 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.290912 27692 ts_tablet_manager.cc:1428] T 6d13884fff8c43b690836c6a341a2d92 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:39.291216 27692 tablet_bootstrap.cc:492] T 351208baafc94f5eab531101d52e3b3f P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:39.291962 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2f3c5f77e6bd4a0ebf0ac41e63589e79" } modify_external_catalogs: true
I20250903 20:05:39.292147 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2f3c5f77e6bd4a0ebf0ac41e63589e79" } modify_external_catalogs: true
I20250903 20:05:39.292961 27692 tablet_bootstrap.cc:654] T 351208baafc94f5eab531101d52e3b3f P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:39.293892 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cf341e1600a14cb1893d36c435e8667f consensus state has no leader
I20250903 20:05:39.295147 27065 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_32aedf31a0514e6f90ec9dc9f0831e13" } new_table_name: "default.Table_9e8c762fd92f456c9851e418c43f74df" modify_external_catalogs: true
I20250903 20:05:39.298378 27684 ts_tablet_manager.cc:1918] T 47c1287bff1145a5b2ab6d14dbd608c7 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:39.298518 27684 log.cc:1199] T 47c1287bff1145a5b2ab6d14dbd608c7 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/47c1287bff1145a5b2ab6d14dbd608c7
I20250903 20:05:39.298893 27684 ts_tablet_manager.cc:1939] T 47c1287bff1145a5b2ab6d14dbd608c7 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:39.299645 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1140) took 3668 ms. Trace:
I20250903 20:05:39.299743 27684 rpcz_store.cc:276] 0903 20:05:35.631057 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.631308 (+ 251us) service_pool.cc:225] Handling call
0903 20:05:39.226401 (+3595093us) ts_tablet_manager.cc:1145] Deleting tablet 47c1287bff1145a5b2ab6d14dbd608c7
0903 20:05:39.226418 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.286551 (+ 60133us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.298203 (+ 11652us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.299597 (+ 1394us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3594836}
I20250903 20:05:39.300611 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 47c1287bff1145a5b2ab6d14dbd608c7 (table default.Table_0cd65e00bd5a40bd9dc9db12fd0bda4d [id=15369700921549be8faa450d2ca98949]) successfully deleted
I20250903 20:05:39.301916 27063 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_c1a71b1082b54337a2d52b446d1d4258 [id=6d6f5e8cadf9445d8695c075cfabb2ea] alter complete (version 2)
I20250903 20:05:39.302137 27066 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_adf05e59a7be478b89666c3162ac83f1 [id=e69321e60a3a4e71a463a0011c0c6d65] alter complete (version 2)
W20250903 20:05:39.304112 27066 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.26.65.67:58355 (request call id 177) took 904 ms (client timeout 1000 ms). Trace:
W20250903 20:05:39.304220 27066 rpcz_store.cc:269] 0903 20:05:38.400030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.415179 (+ 15149us) service_pool.cc:225] Handling call
0903 20:05:38.450941 (+ 35762us) spinlock_profiling.cc:243] Waited 34.6 ms on lock 0x616000040f54. stack: 00007f4b9d6c2dc6 00007f4b9d6c2b80 00007f4b9c6aa3ad 0000000000972d94 0000000000972d5d 00007f4bac49014d 00007f4bac490243 00007f4ba914f941 00007f4bb0cdaa7d 00007f4bb0e5e6f8 00007f4ba7c98c86 00007f4ba7c989b5 00007f4ba44730be 00007f4ba4471f9e 00007f4ba44781d5 00007f4ba4478aff
0903 20:05:39.304083 (+853142us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:38.723456 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:38.723626 (+ 170us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:38.723635 (+ 9us) write_op.cc:435] Acquired schema lock
0903 20:05:38.723640 (+ 5us) tablet.cc:660] Decoding operations
0903 20:05:38.723779 (+ 139us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:38.723795 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:38.723802 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0903 20:05:38.723891 (+ 89us) tablet.cc:699] Row locks acquired
0903 20:05:38.723896 (+ 5us) write_op.cc:260] PREPARE: finished
0903 20:05:38.723944 (+ 48us) write_op.cc:270] Start()
0903 20:05:38.723989 (+ 45us) write_op.cc:276] Timestamp: P: 1756929938723890 usec, L: 0
0903 20:05:38.723995 (+ 6us) op_driver.cc:348] REPLICATION: starting
0903 20:05:38.724238 (+ 243us) log.cc:844] Serialized 356 byte log entry
0903 20:05:38.827425 (+103187us) op_driver.cc:464] REPLICATION: finished
0903 20:05:38.827719 (+ 294us) write_op.cc:301] APPLY: starting
0903 20:05:38.827740 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:38.827790 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:38.827850 (+ 60us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:38.827917 (+ 67us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:38.827955 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0903 20:05:38.827973 (+ 18us) write_op.cc:312] APPLY: finished
0903 20:05:38.828069 (+ 96us) log.cc:844] Serialized 23 byte log entry
0903 20:05:38.828151 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:38.828172 (+ 21us) write_op.cc:454] Released schema lock
0903 20:05:38.828205 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0903 20:05:38.828294 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:38.828299 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0903 20:05:38.924493 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:38.924760 (+ 267us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:38.924766 (+ 6us) write_op.cc:435] Acquired schema lock
0903 20:05:38.924768 (+ 2us) tablet.cc:660] Decoding operations
0903 20:05:38.924892 (+ 124us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:38.924908 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:38.924912 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0903 20:05:38.924985 (+ 73us) tablet.cc:699] Row locks acquired
0903 20:05:38.924988 (+ 3us) write_op.cc:260] PREPARE: finished
0903 20:05:38.925035 (+ 47us) write_op.cc:270] Start()
0903 20:05:38.925072 (+ 37us) write_op.cc:276] Timestamp: P: 1756929938925011 usec, L: 0
0903 20:05:38.925075 (+ 3us) op_driver.cc:348] REPLICATION: starting
0903 20:05:38.925262 (+ 187us) log.cc:844] Serialized 356 byte log entry
0903 20:05:38.940672 (+ 15410us) op_driver.cc:464] REPLICATION: finished
0903 20:05:38.947751 (+ 7079us) write_op.cc:301] APPLY: starting
0903 20:05:38.948775 (+ 1024us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:38.948831 (+ 56us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:38.948880 (+ 49us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:38.948954 (+ 74us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:38.949012 (+ 58us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0903 20:05:38.949019 (+ 7us) write_op.cc:312] APPLY: finished
0903 20:05:38.949163 (+ 144us) log.cc:844] Serialized 23 byte log entry
0903 20:05:38.949254 (+ 91us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:38.949280 (+ 26us) write_op.cc:454] Released schema lock
0903 20:05:38.949288 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0903 20:05:38.959413 (+ 10125us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:38.959422 (+ 9us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":19,"spinlock_wait_cycles":79594368,"child_traces":[["op",{"apply.queue_time_us":230,"apply.run_cpu_time_us":563,"apply.run_wall_time_us":563,"num_ops":1,"prepare.queue_time_us":31687,"prepare.run_cpu_time_us":933,"prepare.run_wall_time_us":1086,"raft.queue_time_us":76,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":7,"replication_time_us":103395}],["op",{"apply.queue_time_us":6957,"apply.run_cpu_time_us":689,"apply.run_wall_time_us":2414,"num_ops":1,"prepare.queue_time_us":88663,"prepare.run_cpu_time_us":859,"prepare.run_wall_time_us":34901,"raft.queue_time_us":87,"raft.run_cpu_time_us":349,"raft.run_wall_time_us":21863,"replication_time_us":15555,"spinlock_wait_cycles":122624}]]}
I20250903 20:05:39.305465 27693 tablet_bootstrap.cc:492] T de20d18d7f264811a0631aa1144aef2c P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:39.305601 27693 ts_tablet_manager.cc:1397] T de20d18d7f264811a0631aa1144aef2c P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.001s
I20250903 20:05:39.306563 27693 raft_consensus.cc:357] T de20d18d7f264811a0631aa1144aef2c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.306756 27693 raft_consensus.cc:383] T de20d18d7f264811a0631aa1144aef2c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.306816 27693 raft_consensus.cc:738] T de20d18d7f264811a0631aa1144aef2c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.305445 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_883db815aa994ff4bde1c00592681e1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:39.307080 27693 consensus_queue.cc:260] T de20d18d7f264811a0631aa1144aef2c P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:39.307627 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_883db815aa994ff4bde1c00592681e1b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:39.309263 27684 tablet_replica.cc:331] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:39.309505 27684 raft_consensus.cc:2241] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:39.309873 27684 raft_consensus.cc:2270] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:39.310477 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet f9375fff5f5945078c4412aa21c06f0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:39.310672 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet a686ea49a29a4dd7b6e5dce5e1f9935a with delete_type TABLET_DATA_DELETED (replaced by tablet 9657ef024273400398d5e76f15359c40) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:39.310828 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet 54f1ce5192f547f595dea23a9eabf8e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:39.311798 27684 ts_tablet_manager.cc:1905] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:39.314754 27693 ts_tablet_manager.cc:1428] T de20d18d7f264811a0631aa1144aef2c P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250903 20:05:39.317559 29455 raft_consensus.cc:491] T de20d18d7f264811a0631aa1144aef2c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.319142 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "733f8024e1fa401ca9614a97db91a19b" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:39.320273 27462 leader_election.cc:343] T 733f8024e1fa401ca9614a97db91a19b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.320427 27462 leader_election.cc:304] T 733f8024e1fa401ca9614a97db91a19b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:39.320667 29443 raft_consensus.cc:2747] T 733f8024e1fa401ca9614a97db91a19b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:39.322888 27693 tablet_bootstrap.cc:492] T 4739f10128d64e2c986257b1d9deb90f P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:39.325080 27693 tablet_bootstrap.cc:654] T 4739f10128d64e2c986257b1d9deb90f P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.326088 29572 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 20:05:39.327519 27692 tablet_bootstrap.cc:492] T 351208baafc94f5eab531101d52e3b3f P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:39.327680 27692 ts_tablet_manager.cc:1397] T 351208baafc94f5eab531101d52e3b3f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.001s
I20250903 20:05:39.328527 27692 raft_consensus.cc:357] T 351208baafc94f5eab531101d52e3b3f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.315222 27687 ts_tablet_manager.cc:1905] T cb2f50683e904dd39ee6eb128717d38c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:39.328779 27692 raft_consensus.cc:383] T 351208baafc94f5eab531101d52e3b3f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.328109 27347 tablet_service.cc:1468] Processing CreateTablet for tablet d9e522ecaf6042448b49e126a147cd9d (DEFAULT_TABLE table=default.Table_c19f2a545c9e49848bfa6313fa9a9543 [id=4a9daa9c5645434ba4e9b5002df9b6f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.328872 27692 raft_consensus.cc:738] T 351208baafc94f5eab531101d52e3b3f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.328985 27497 tablet_service.cc:1468] Processing CreateTablet for tablet d9e522ecaf6042448b49e126a147cd9d (DEFAULT_TABLE table=default.Table_c19f2a545c9e49848bfa6313fa9a9543 [id=4a9daa9c5645434ba4e9b5002df9b6f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.329167 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9e522ecaf6042448b49e126a147cd9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:39.329133 27692 consensus_queue.cc:260] T 351208baafc94f5eab531101d52e3b3f P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.329350 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9e522ecaf6042448b49e126a147cd9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:39.329808 27692 ts_tablet_manager.cc:1428] T 351208baafc94f5eab531101d52e3b3f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:39.331038 27684 ts_tablet_manager.cc:1918] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:39.331152 27684 log.cc:1199] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/cb2f50683e904dd39ee6eb128717d38c
I20250903 20:05:39.331525 27684 ts_tablet_manager.cc:1939] T cb2f50683e904dd39ee6eb128717d38c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:39.332017 29170 fs_manager.cc:730] Time spent opening block manager: real 0.219s user 0.032s sys 0.023s
I20250903 20:05:39.332223 29170 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/data,/tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/wal
uuid: "35dd69f3fa7a4fd8843b97f8a5d68296"
format_stamp: "Formatted at 2025-09-03 20:05:05 on dist-test-slave-fs2w"
I20250903 20:05:39.333456 29170 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 20:05:39.334597 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1139) took 3703 ms. Trace:
I20250903 20:05:39.334692 29443 raft_consensus.cc:491] T 9c77bc0e8bb34712bf8109cadbc0fc06 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.335453 29443 raft_consensus.cc:513] T 9c77bc0e8bb34712bf8109cadbc0fc06 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.336211 29443 leader_election.cc:290] T 9c77bc0e8bb34712bf8109cadbc0fc06 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:39.337423 27376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c77bc0e8bb34712bf8109cadbc0fc06" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:39.337451 29455 raft_consensus.cc:513] T de20d18d7f264811a0631aa1144aef2c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.337924 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c77bc0e8bb34712bf8109cadbc0fc06" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:39.338296 29455 leader_election.cc:290] T de20d18d7f264811a0631aa1144aef2c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:39.338562 29455 raft_consensus.cc:491] T 886b8783aa9d4618aad498d4a1115843 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e72ef1dbcc5143cf9f04a808f101c80f)
I20250903 20:05:39.338737 29455 raft_consensus.cc:513] T 886b8783aa9d4618aad498d4a1115843 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:39.339025 27462 leader_election.cc:343] T 9c77bc0e8bb34712bf8109cadbc0fc06 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.339394 29455 leader_election.cc:290] T 886b8783aa9d4618aad498d4a1115843 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:39.339587 27685 ts_tablet_manager.cc:1918] T a38f65e992884635aae9942681c0dc4f P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:39.339303 27684 rpcz_store.cc:276] 0903 20:05:35.630927 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.633465 (+ 2538us) service_pool.cc:225] Handling call
0903 20:05:39.309179 (+3675714us) ts_tablet_manager.cc:1145] Deleting tablet cb2f50683e904dd39ee6eb128717d38c
0903 20:05:39.309191 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.323390 (+ 14199us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.330919 (+ 7529us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.334495 (+ 3576us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3675533}
I20250903 20:05:39.339737 27685 log.cc:1199] T a38f65e992884635aae9942681c0dc4f P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/a38f65e992884635aae9942681c0dc4f
I20250903 20:05:39.340039 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet cb2f50683e904dd39ee6eb128717d38c (table default.Table_e914faf335d44d019f91bda71758c473 [id=374b9abdf6a54b38876a96d8ec4a0368]) successfully deleted
I20250903 20:05:39.340137 27685 ts_tablet_manager.cc:1939] T a38f65e992884635aae9942681c0dc4f P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:39.340484 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1054) took 4959 ms. Trace:
I20250903 20:05:39.340569 27685 rpcz_store.cc:276] 0903 20:05:34.380518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.380566 (+ 48us) service_pool.cc:225] Handling call
0903 20:05:39.180579 (+4800013us) ts_tablet_manager.cc:1145] Deleting tablet a38f65e992884635aae9942681c0dc4f
0903 20:05:39.180591 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.211260 (+ 30669us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.320577 (+109317us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.340454 (+ 19877us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4799763}
I20250903 20:05:39.340775 27684 tablet_replica.cc:331] T 64c4dd2645034a7eaf243f5474603f10 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:39.340866 27685 tablet_replica.cc:331] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:39.340996 27684 raft_consensus.cc:2241] T 64c4dd2645034a7eaf243f5474603f10 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:39.341085 27685 raft_consensus.cc:2241] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:39.341171 27684 raft_consensus.cc:2270] T 64c4dd2645034a7eaf243f5474603f10 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:39.341444 27685 raft_consensus.cc:2270] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:39.342090 27685 ts_tablet_manager.cc:1905] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:39.343161 27376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de20d18d7f264811a0631aa1144aef2c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:39.343367 27376 raft_consensus.cc:2466] T de20d18d7f264811a0631aa1144aef2c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:39.343793 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet a38f65e992884635aae9942681c0dc4f (table default.Table_bb8c68be79034a5c8469862c43953131 [id=f0171ee670294ce58ab84483fd71ab7e]) successfully deleted
I20250903 20:05:39.344785 27684 ts_tablet_manager.cc:1905] T 64c4dd2645034a7eaf243f5474603f10 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:39.344969 27462 leader_election.cc:343] T 9c77bc0e8bb34712bf8109cadbc0fc06 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.345113 27462 leader_election.cc:304] T 9c77bc0e8bb34712bf8109cadbc0fc06 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:39.345364 29443 raft_consensus.cc:2747] T 9c77bc0e8bb34712bf8109cadbc0fc06 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:39.330168 27692 tablet_bootstrap.cc:492] T ea3f82f4e4244219ab0919f3e341aead P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:39.346436 27510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de20d18d7f264811a0631aa1144aef2c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:39.346625 27510 raft_consensus.cc:2466] T de20d18d7f264811a0631aa1144aef2c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:39.347625 29443 raft_consensus.cc:491] T ed4902c002ef4198802375ced39ef2ad P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.347785 29443 raft_consensus.cc:513] T ed4902c002ef4198802375ced39ef2ad P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.348452 29443 leader_election.cc:290] T ed4902c002ef4198802375ced39ef2ad P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:39.348901 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed4902c002ef4198802375ced39ef2ad" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:39.349475 27462 leader_election.cc:343] T ed4902c002ef4198802375ced39ef2ad P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.348453 27692 tablet_bootstrap.cc:654] T ea3f82f4e4244219ab0919f3e341aead P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.353439 27687 ts_tablet_manager.cc:1918] T cb2f50683e904dd39ee6eb128717d38c P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:39.359505 27684 ts_tablet_manager.cc:1918] T 64c4dd2645034a7eaf243f5474603f10 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:39.362289 27684 log.cc:1199] T 64c4dd2645034a7eaf243f5474603f10 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/64c4dd2645034a7eaf243f5474603f10
W20250903 20:05:39.353627 27434 tablet.cc:2378] T 249362a0b8304719863bcb82c47273b3 P 51578fb7bceb4e119b3ab1e58c4f5528: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 20:05:39.362542 27687 log.cc:1199] T cb2f50683e904dd39ee6eb128717d38c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/cb2f50683e904dd39ee6eb128717d38c
I20250903 20:05:39.359138 27693 tablet_bootstrap.cc:492] T 4739f10128d64e2c986257b1d9deb90f P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:39.363085 27693 ts_tablet_manager.cc:1397] T 4739f10128d64e2c986257b1d9deb90f P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.002s
I20250903 20:05:39.363180 27684 ts_tablet_manager.cc:1939] T 64c4dd2645034a7eaf243f5474603f10 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:39.363953 27693 raft_consensus.cc:357] T 4739f10128d64e2c986257b1d9deb90f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.364205 27693 raft_consensus.cc:383] T 4739f10128d64e2c986257b1d9deb90f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.364305 27693 raft_consensus.cc:738] T 4739f10128d64e2c986257b1d9deb90f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.364485 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1141) took 3733 ms. Trace:
I20250903 20:05:39.364590 27693 consensus_queue.cc:260] T 4739f10128d64e2c986257b1d9deb90f P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.364730 29443 raft_consensus.cc:491] T 60c32370a5274347afafc6a7e1e94270 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.365711 29443 raft_consensus.cc:513] T 60c32370a5274347afafc6a7e1e94270 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.366412 29443 leader_election.cc:290] T 60c32370a5274347afafc6a7e1e94270 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:39.367461 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60c32370a5274347afafc6a7e1e94270" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:39.368469 27184 leader_election.cc:304] T de20d18d7f264811a0631aa1144aef2c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
W20250903 20:05:39.368826 27462 leader_election.cc:343] T 60c32370a5274347afafc6a7e1e94270 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.369269 27684 rpcz_store.cc:276] 0903 20:05:35.631205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.633673 (+ 2468us) service_pool.cc:225] Handling call
0903 20:05:39.340459 (+3706786us) ts_tablet_manager.cc:1145] Deleting tablet 64c4dd2645034a7eaf243f5474603f10
0903 20:05:39.340471 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.354693 (+ 14222us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.359382 (+ 4689us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.364442 (+ 5060us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3706668}
I20250903 20:05:39.369774 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed4902c002ef4198802375ced39ef2ad" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:39.370752 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1146) took 3554 ms. Trace:
I20250903 20:05:39.370738 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 64c4dd2645034a7eaf243f5474603f10 (table default.Table_d9878f04c3404885815ae53a9054de14 [id=a98d20608a054ad29a951bb47cf61711]) successfully deleted
I20250903 20:05:39.371519 27684 rpcz_store.cc:276] 0903 20:05:35.816714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.816759 (+ 45us) service_pool.cc:225] Handling call
0903 20:05:39.370566 (+3553807us) ts_tablet_manager.cc:1145] Deleting tablet 1598824a733a4a93bed043e81b86abf3
0903 20:05:39.370583 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.370713 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3553636}
W20250903 20:05:39.372367 27462 leader_election.cc:343] T ed4902c002ef4198802375ced39ef2ad P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.372535 27462 leader_election.cc:304] T ed4902c002ef4198802375ced39ef2ad P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
W20250903 20:05:39.372591 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 1598824a733a4a93bed043e81b86abf3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1598824a733a4a93bed043e81b86abf3 already in progress: creating tablet
I20250903 20:05:39.373117 29443 raft_consensus.cc:2747] T ed4902c002ef4198802375ced39ef2ad P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:39.373526 27693 ts_tablet_manager.cc:1428] T 4739f10128d64e2c986257b1d9deb90f P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250903 20:05:39.373550 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60c32370a5274347afafc6a7e1e94270" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:39.372361 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1147) took 3554 ms. Trace:
W20250903 20:05:39.373888 27184 leader_election.cc:343] T 886b8783aa9d4618aad498d4a1115843 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 886b8783aa9d4618aad498d4a1115843
I20250903 20:05:39.374060 27684 rpcz_store.cc:276] 0903 20:05:35.817649 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.817699 (+ 50us) service_pool.cc:225] Handling call
0903 20:05:39.372172 (+3554473us) ts_tablet_manager.cc:1145] Deleting tablet 4440a6c2afb247e0a58a83806cf5b334
0903 20:05:39.372182 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.372306 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3554283}
I20250903 20:05:39.374174 29461 raft_consensus.cc:2802] T de20d18d7f264811a0631aa1144aef2c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:39.374315 29461 raft_consensus.cc:491] T de20d18d7f264811a0631aa1144aef2c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:39.374405 29461 raft_consensus.cc:3058] T de20d18d7f264811a0631aa1144aef2c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:39.375013 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 4440a6c2afb247e0a58a83806cf5b334 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4440a6c2afb247e0a58a83806cf5b334 already in progress: creating tablet
I20250903 20:05:39.375665 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1148) took 3556 ms. Trace:
I20250903 20:05:39.376101 27684 rpcz_store.cc:276] 0903 20:05:35.819571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.819623 (+ 52us) service_pool.cc:225] Handling call
0903 20:05:39.375531 (+3555908us) ts_tablet_manager.cc:1145] Deleting tablet 7baeb72c8bd440c39c07f34a6cb5e3ea
0903 20:05:39.375541 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.375632 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3539905}
W20250903 20:05:39.376744 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 7baeb72c8bd440c39c07f34a6cb5e3ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7baeb72c8bd440c39c07f34a6cb5e3ea already in progress: creating tablet
I20250903 20:05:39.377059 29461 raft_consensus.cc:513] T de20d18d7f264811a0631aa1144aef2c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.377414 27693 tablet_bootstrap.cc:492] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:39.377878 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1151) took 3430 ms. Trace:
I20250903 20:05:39.378185 27684 rpcz_store.cc:276] 0903 20:05:35.947076 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.947900 (+ 824us) service_pool.cc:225] Handling call
0903 20:05:39.377060 (+3429160us) ts_tablet_manager.cc:1145] Deleting tablet c066691b278c4278937b4d8ef98b395f
0903 20:05:39.377070 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.377151 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3428918}
I20250903 20:05:39.378732 27687 ts_tablet_manager.cc:1939] T cb2f50683e904dd39ee6eb128717d38c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
W20250903 20:05:39.379022 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet c066691b278c4278937b4d8ef98b395f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c066691b278c4278937b4d8ef98b395f already in progress: creating tablet
I20250903 20:05:39.379474 27692 tablet_bootstrap.cc:492] T ea3f82f4e4244219ab0919f3e341aead P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:39.379640 27692 ts_tablet_manager.cc:1397] T ea3f82f4e4244219ab0919f3e341aead P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.050s user 0.005s sys 0.001s
W20250903 20:05:39.379760 27462 leader_election.cc:343] T 60c32370a5274347afafc6a7e1e94270 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.379910 27462 leader_election.cc:304] T 60c32370a5274347afafc6a7e1e94270 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:39.380295 29443 raft_consensus.cc:2747] T 60c32370a5274347afafc6a7e1e94270 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:39.380506 27692 raft_consensus.cc:357] T ea3f82f4e4244219ab0919f3e341aead P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.380733 27692 raft_consensus.cc:383] T ea3f82f4e4244219ab0919f3e341aead P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.380843 27692 raft_consensus.cc:738] T ea3f82f4e4244219ab0919f3e341aead P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.380864 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1152) took 3430 ms. Trace:
I20250903 20:05:39.378835 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de20d18d7f264811a0631aa1144aef2c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:39.381098 27692 consensus_queue.cc:260] T ea3f82f4e4244219ab0919f3e341aead P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.381229 27385 raft_consensus.cc:3058] T de20d18d7f264811a0631aa1144aef2c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:39.381654 27692 ts_tablet_manager.cc:1428] T ea3f82f4e4244219ab0919f3e341aead P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:39.382329 27692 tablet_bootstrap.cc:492] T 43fb27c857d543b7b78ea6108c406085 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:39.384053 27385 raft_consensus.cc:2466] T de20d18d7f264811a0631aa1144aef2c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:39.384101 27692 tablet_bootstrap.cc:654] T 43fb27c857d543b7b78ea6108c406085 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.384877 28567 raft_consensus.cc:491] T e2441aa3c2c748bba5360259fb663c81 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.385058 28567 raft_consensus.cc:513] T e2441aa3c2c748bba5360259fb663c81 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.386121 28567 leader_election.cc:290] T e2441aa3c2c748bba5360259fb663c81 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:39.387090 27184 leader_election.cc:304] T de20d18d7f264811a0631aa1144aef2c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:39.387233 27510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2441aa3c2c748bba5360259fb663c81" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:39.387563 29455 raft_consensus.cc:2802] T de20d18d7f264811a0631aa1144aef2c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:39.387708 29455 raft_consensus.cc:695] T de20d18d7f264811a0631aa1144aef2c P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:39.381165 27684 rpcz_store.cc:276] 0903 20:05:35.949105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.949162 (+ 57us) service_pool.cc:225] Handling call
0903 20:05:39.379405 (+3430243us) ts_tablet_manager.cc:1145] Deleting tablet b437eca9eb0b4c60bf797121a5a8bad8
0903 20:05:39.379414 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.379511 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3430045}
I20250903 20:05:39.388028 29455 consensus_queue.cc:237] T de20d18d7f264811a0631aa1144aef2c P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:39.388144 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet b437eca9eb0b4c60bf797121a5a8bad8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b437eca9eb0b4c60bf797121a5a8bad8 already in progress: creating tablet
I20250903 20:05:39.388763 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1153) took 3438 ms. Trace:
I20250903 20:05:39.389379 27684 rpcz_store.cc:276] 0903 20:05:35.950612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.950685 (+ 73us) service_pool.cc:225] Handling call
0903 20:05:39.388626 (+3437941us) ts_tablet_manager.cc:1145] Deleting tablet 435c002dffd24aeaa6221ab00ee5b24b
0903 20:05:39.388636 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.388731 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3437706}
W20250903 20:05:39.390559 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 435c002dffd24aeaa6221ab00ee5b24b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 435c002dffd24aeaa6221ab00ee5b24b already in progress: creating tablet
I20250903 20:05:39.391031 29461 leader_election.cc:290] T de20d18d7f264811a0631aa1144aef2c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:39.392876 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1155) took 3370 ms. Trace:
I20250903 20:05:39.393357 27692 tablet_bootstrap.cc:492] T 43fb27c857d543b7b78ea6108c406085 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:39.393649 27692 ts_tablet_manager.cc:1397] T 43fb27c857d543b7b78ea6108c406085 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20250903 20:05:39.393596 27684 rpcz_store.cc:276] 0903 20:05:36.022882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.022934 (+ 52us) service_pool.cc:225] Handling call
0903 20:05:39.392729 (+3369795us) ts_tablet_manager.cc:1145] Deleting tablet 7bb0da31c68440788dc296e67b38ec7c
0903 20:05:39.392739 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.392833 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3369591}
W20250903 20:05:39.394515 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 7bb0da31c68440788dc296e67b38ec7c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7bb0da31c68440788dc296e67b38ec7c already in progress: creating tablet
I20250903 20:05:39.394807 27692 raft_consensus.cc:357] T 43fb27c857d543b7b78ea6108c406085 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.395042 27684 tablet_replica.cc:331] T 3f303233e6c347289650c6cf83f8127f P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:39.395824 27684 raft_consensus.cc:2241] T 3f303233e6c347289650c6cf83f8127f P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:39.396055 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1160) took 3811 ms. Trace:
I20250903 20:05:39.396494 27684 raft_consensus.cc:2270] T 3f303233e6c347289650c6cf83f8127f P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:39.397051 27687 rpcz_store.cc:276] 0903 20:05:35.584814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.584876 (+ 62us) service_pool.cc:225] Handling call
0903 20:05:39.229563 (+3644687us) ts_tablet_manager.cc:1145] Deleting tablet cb2f50683e904dd39ee6eb128717d38c
0903 20:05:39.229574 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.349362 (+119788us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.353291 (+ 3929us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.396020 (+ 42729us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3644421}
I20250903 20:05:39.380592 27693 tablet_bootstrap.cc:654] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.397660 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet cb2f50683e904dd39ee6eb128717d38c (table default.Table_e914faf335d44d019f91bda71758c473 [id=374b9abdf6a54b38876a96d8ec4a0368]) successfully deleted
I20250903 20:05:39.398618 27684 ts_tablet_manager.cc:1905] T 3f303233e6c347289650c6cf83f8127f P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:39.399080 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1169) took 3625 ms. Trace:
I20250903 20:05:39.399273 27687 rpcz_store.cc:276] 0903 20:05:35.773639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.774617 (+ 978us) service_pool.cc:225] Handling call
0903 20:05:39.398817 (+3624200us) ts_tablet_manager.cc:1145] Deleting tablet 4440a6c2afb247e0a58a83806cf5b334
0903 20:05:39.398838 (+ 21us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.399050 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3623972}
W20250903 20:05:39.399865 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 4440a6c2afb247e0a58a83806cf5b334 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4440a6c2afb247e0a58a83806cf5b334 already in progress: creating tablet
I20250903 20:05:39.395131 27692 raft_consensus.cc:383] T 43fb27c857d543b7b78ea6108c406085 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.400094 27692 raft_consensus.cc:738] T 43fb27c857d543b7b78ea6108c406085 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.402057 27692 consensus_queue.cc:260] T 43fb27c857d543b7b78ea6108c406085 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.402809 27692 ts_tablet_manager.cc:1428] T 43fb27c857d543b7b78ea6108c406085 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20250903 20:05:39.403967 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1168) took 3665 ms. Trace:
I20250903 20:05:39.404405 27687 rpcz_store.cc:276] 0903 20:05:35.738889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.774853 (+ 35964us) service_pool.cc:225] Handling call
0903 20:05:39.403285 (+3628432us) ts_tablet_manager.cc:1145] Deleting tablet 1598824a733a4a93bed043e81b86abf3
0903 20:05:39.403296 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.403947 (+ 651us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3628101}
W20250903 20:05:39.405395 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 1598824a733a4a93bed043e81b86abf3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1598824a733a4a93bed043e81b86abf3 already in progress: creating tablet
I20250903 20:05:39.405606 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1170) took 3593 ms. Trace:
I20250903 20:05:39.407249 27692 tablet_bootstrap.cc:492] T 8400e23873fd4be18de2998afc64a689 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:39.404840 27693 tablet_bootstrap.cc:492] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:39.407534 27693 ts_tablet_manager.cc:1397] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20250903 20:05:39.408336 27685 ts_tablet_manager.cc:1918] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:39.406502 27687 rpcz_store.cc:276] 0903 20:05:35.812005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.812081 (+ 76us) service_pool.cc:225] Handling call
0903 20:05:39.405446 (+3593365us) ts_tablet_manager.cc:1145] Deleting tablet 7baeb72c8bd440c39c07f34a6cb5e3ea
0903 20:05:39.405457 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.405586 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3593088}
W20250903 20:05:39.409138 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 7baeb72c8bd440c39c07f34a6cb5e3ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7baeb72c8bd440c39c07f34a6cb5e3ea already in progress: creating tablet
I20250903 20:05:39.409271 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1172) took 3534 ms. Trace:
I20250903 20:05:39.409348 27687 rpcz_store.cc:276] 0903 20:05:35.874485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.874701 (+ 216us) service_pool.cc:225] Handling call
0903 20:05:39.409106 (+3534405us) ts_tablet_manager.cc:1145] Deleting tablet c066691b278c4278937b4d8ef98b395f
0903 20:05:39.409120 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.409251 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3534184}
W20250903 20:05:39.411087 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet c066691b278c4278937b4d8ef98b395f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c066691b278c4278937b4d8ef98b395f already in progress: creating tablet
I20250903 20:05:39.411285 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1174) took 3495 ms. Trace:
I20250903 20:05:39.412788 27692 tablet_bootstrap.cc:654] T 8400e23873fd4be18de2998afc64a689 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.413415 27687 rpcz_store.cc:276] 0903 20:05:35.915451 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.915682 (+ 231us) service_pool.cc:225] Handling call
0903 20:05:39.411132 (+3495450us) ts_tablet_manager.cc:1145] Deleting tablet b437eca9eb0b4c60bf797121a5a8bad8
0903 20:05:39.411145 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.411266 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3495237}
W20250903 20:05:39.414178 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet b437eca9eb0b4c60bf797121a5a8bad8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b437eca9eb0b4c60bf797121a5a8bad8 already in progress: creating tablet
I20250903 20:05:39.414434 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1175) took 3466 ms. Trace:
I20250903 20:05:39.415055 27687 rpcz_store.cc:276] 0903 20:05:35.947584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.947800 (+ 216us) service_pool.cc:225] Handling call
0903 20:05:39.414272 (+3466472us) ts_tablet_manager.cc:1145] Deleting tablet 435c002dffd24aeaa6221ab00ee5b24b
0903 20:05:39.414286 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.414416 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3466249}
I20250903 20:05:39.413120 27693 raft_consensus.cc:357] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
W20250903 20:05:39.415840 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 435c002dffd24aeaa6221ab00ee5b24b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 435c002dffd24aeaa6221ab00ee5b24b already in progress: creating tablet
I20250903 20:05:39.415884 27693 raft_consensus.cc:383] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.415957 27693 raft_consensus.cc:738] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
W20250903 20:05:39.416289 27183 leader_election.cc:343] T 886b8783aa9d4618aad498d4a1115843 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 886b8783aa9d4618aad498d4a1115843
I20250903 20:05:39.416267 27693 consensus_queue.cc:260] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:39.416908 27685 log.cc:1199] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/aa9f5e0964b348b7af184f60cd145666
I20250903 20:05:39.417889 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1176) took 3454 ms. Trace:
I20250903 20:05:39.418241 27695 tablet_bootstrap.cc:492] T 615f42e95bb24c8480f73d371fd63b9c P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:39.418416 27695 ts_tablet_manager.cc:1397] T 615f42e95bb24c8480f73d371fd63b9c P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.269s user 0.008s sys 0.045s
I20250903 20:05:39.418568 27687 rpcz_store.cc:276] 0903 20:05:35.963307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.963475 (+ 168us) service_pool.cc:225] Handling call
0903 20:05:39.417760 (+3454285us) ts_tablet_manager.cc:1145] Deleting tablet 7bb0da31c68440788dc296e67b38ec7c
0903 20:05:39.417771 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.417867 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3454085}
I20250903 20:05:39.419315 27695 raft_consensus.cc:357] T 615f42e95bb24c8480f73d371fd63b9c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.419555 27695 raft_consensus.cc:383] T 615f42e95bb24c8480f73d371fd63b9c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.419646 27695 raft_consensus.cc:738] T 615f42e95bb24c8480f73d371fd63b9c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.419864 29452 raft_consensus.cc:491] T 46f49fd718be4a2996d781ab31268217 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.419939 27695 consensus_queue.cc:260] T 615f42e95bb24c8480f73d371fd63b9c P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.420060 29452 raft_consensus.cc:513] T 46f49fd718be4a2996d781ab31268217 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.420518 27183 leader_election.cc:304] T 886b8783aa9d4618aad498d4a1115843 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:39.422048 27685 ts_tablet_manager.cc:1939] T aa9f5e0964b348b7af184f60cd145666 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:39.415161 27684 ts_tablet_manager.cc:1918] T 3f303233e6c347289650c6cf83f8127f P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:39.424052 27684 log.cc:1199] T 3f303233e6c347289650c6cf83f8127f P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/3f303233e6c347289650c6cf83f8127f
I20250903 20:05:39.424599 27684 ts_tablet_manager.cc:1939] T 3f303233e6c347289650c6cf83f8127f P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:39.425482 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1159) took 3370 ms. Trace:
I20250903 20:05:39.425552 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1055) took 5043 ms. Trace:
I20250903 20:05:39.425679 27685 rpcz_store.cc:276] 0903 20:05:34.381678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.381728 (+ 50us) service_pool.cc:225] Handling call
0903 20:05:39.340811 (+4959083us) ts_tablet_manager.cc:1145] Deleting tablet aa9f5e0964b348b7af184f60cd145666
0903 20:05:39.340819 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.384109 (+ 43290us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.408184 (+ 24075us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.425505 (+ 17321us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4958865}
I20250903 20:05:39.426172 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet aa9f5e0964b348b7af184f60cd145666 (table default.Table_14e02c1be9c74bd99f4184faa6753f52 [id=a6dc7550a6d643008c92b3a59ba022ce]) successfully deleted
I20250903 20:05:39.426297 27685 tablet_replica.cc:331] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:39.426927 27685 raft_consensus.cc:2241] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:39.427207 27684 rpcz_store.cc:276] 0903 20:05:36.054704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.054785 (+ 81us) service_pool.cc:225] Handling call
0903 20:05:39.394938 (+3340153us) ts_tablet_manager.cc:1145] Deleting tablet 3f303233e6c347289650c6cf83f8127f
0903 20:05:39.394951 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.408277 (+ 13326us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.415020 (+ 6743us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.425425 (+ 10405us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3339901}
I20250903 20:05:39.427659 27695 ts_tablet_manager.cc:1428] T 615f42e95bb24c8480f73d371fd63b9c P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20250903 20:05:39.427758 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 3f303233e6c347289650c6cf83f8127f (table default.Table_58569618f9d04fdfbd005b3d0e15c049 [id=1cbfdd865e51404688e08bc60eec2197]) successfully deleted
I20250903 20:05:39.427810 29589 raft_consensus.cc:2747] T 886b8783aa9d4618aad498d4a1115843 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:39.428006 27695 tablet_bootstrap.cc:492] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
W20250903 20:05:39.427963 27183 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 54312.3us
I20250903 20:05:39.428119 29443 raft_consensus.cc:491] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.428260 29443 raft_consensus.cc:513] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.428915 29443 leader_election.cc:290] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:39.429376 27510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de20d18d7f264811a0631aa1144aef2c" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:39.429579 27510 raft_consensus.cc:3058] T de20d18d7f264811a0631aa1144aef2c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:39.429963 27695 tablet_bootstrap.cc:654] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.430688 27684 tablet_replica.cc:331] T de20d18d7f264811a0631aa1144aef2c P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:39.433221 27510 raft_consensus.cc:2466] T de20d18d7f264811a0631aa1144aef2c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:39.433727 27684 raft_consensus.cc:2241] T de20d18d7f264811a0631aa1144aef2c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:39.434180 27684 raft_consensus.cc:2270] T de20d18d7f264811a0631aa1144aef2c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:39.434638 27687 tablet_replica.cc:331] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
W20250903 20:05:39.434751 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 7bb0da31c68440788dc296e67b38ec7c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7bb0da31c68440788dc296e67b38ec7c already in progress: creating tablet
I20250903 20:05:39.434911 27687 raft_consensus.cc:2241] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:39.435128 27684 ts_tablet_manager.cc:1905] T de20d18d7f264811a0631aa1144aef2c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:39.435611 27687 raft_consensus.cc:2270] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:39.436203 27687 ts_tablet_manager.cc:1905] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:39.420419 27693 ts_tablet_manager.cc:1428] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250903 20:05:39.429648 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17f16e9c8f794e03a2f8b3b6b35f8e84" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:39.438591 27693 tablet_bootstrap.cc:492] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
W20250903 20:05:39.439288 27462 leader_election.cc:343] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.440482 27693 tablet_bootstrap.cc:654] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.441618 27347 tablet_service.cc:1468] Processing CreateTablet for tablet ce3bb86fe2f74549934504421254183e (DEFAULT_TABLE table=default.Table_8a7162fd4e2540d5a4b2a7bccac7af08 [id=fe4a1e13e1b7435b9d1c7e2993cf9f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.442288 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce3bb86fe2f74549934504421254183e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:39.430423 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17f16e9c8f794e03a2f8b3b6b35f8e84" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:39.440986 27216 tablet_service.cc:1468] Processing CreateTablet for tablet d9e522ecaf6042448b49e126a147cd9d (DEFAULT_TABLE table=default.Table_c19f2a545c9e49848bfa6313fa9a9543 [id=4a9daa9c5645434ba4e9b5002df9b6f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.451592 27684 ts_tablet_manager.cc:1918] T de20d18d7f264811a0631aa1144aef2c P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:39.451701 27216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9e522ecaf6042448b49e126a147cd9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:39.451795 27684 log.cc:1199] T de20d18d7f264811a0631aa1144aef2c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/de20d18d7f264811a0631aa1144aef2c
W20250903 20:05:39.452289 27462 leader_election.cc:343] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.452399 27684 ts_tablet_manager.cc:1939] T de20d18d7f264811a0631aa1144aef2c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:39.452524 27462 leader_election.cc:304] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:39.454376 29443 raft_consensus.cc:2747] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:39.454792 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1160) took 3339 ms. Trace:
I20250903 20:05:39.454921 27684 rpcz_store.cc:276] 0903 20:05:36.115211 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.115296 (+ 85us) service_pool.cc:225] Handling call
0903 20:05:39.430597 (+3315301us) ts_tablet_manager.cc:1145] Deleting tablet de20d18d7f264811a0631aa1144aef2c
0903 20:05:39.430607 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.434014 (+ 3407us) spinlock_profiling.cc:243] Waited 2.75 ms on lock 0x6160001b2254. stack: 00007f71bc6f7dc6 00007f71bc6f7b80 00007f71bb6df3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f71c8181600 00007f71c9494b13 00007f71cbf772b7 00007f71cbfa9344 00007f71cbf8ada0 00007f71cbf8ab2d 0000000000a774dc 00007f71bc73e14f 00007f71bc74141b 00007f71bc741298
0903 20:05:39.447619 (+ 13605us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.451411 (+ 3792us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.454750 (+ 3339us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6333312,"tablet-delete.queue_time_us":3315005}
I20250903 20:05:39.455699 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet de20d18d7f264811a0631aa1144aef2c (table default.Table_d4e288e7a9f9417288b39f130bab4289 [id=1fdfd927e5654aad860585fc23ce5f8a]) successfully deleted
I20250903 20:05:39.456255 27684 tablet_replica.cc:331] T ee40566e497740789575275d0cdca0db P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:39.456637 27684 raft_consensus.cc:2241] T ee40566e497740789575275d0cdca0db P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:39.456887 27684 raft_consensus.cc:2270] T ee40566e497740789575275d0cdca0db P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:39.457746 27684 ts_tablet_manager.cc:1905] T ee40566e497740789575275d0cdca0db P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:39.464401 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2441aa3c2c748bba5360259fb663c81" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:39.464870 27684 ts_tablet_manager.cc:1918] T ee40566e497740789575275d0cdca0db P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:39.465353 27684 log.cc:1199] T ee40566e497740789575275d0cdca0db P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/ee40566e497740789575275d0cdca0db
I20250903 20:05:39.465899 27684 ts_tablet_manager.cc:1939] T ee40566e497740789575275d0cdca0db P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:39.466903 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1161) took 3348 ms. Trace:
I20250903 20:05:39.467056 27321 leader_election.cc:304] T e2441aa3c2c748bba5360259fb663c81 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:39.467085 27684 rpcz_store.cc:276] 0903 20:05:36.117950 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.118052 (+ 102us) service_pool.cc:225] Handling call
0903 20:05:39.456152 (+3338100us) ts_tablet_manager.cc:1145] Deleting tablet ee40566e497740789575275d0cdca0db
0903 20:05:39.456169 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.461429 (+ 5260us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.464715 (+ 3286us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.466830 (+ 2115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3337822}
I20250903 20:05:39.467633 28567 raft_consensus.cc:2747] T e2441aa3c2c748bba5360259fb663c81 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:39.467629 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet ee40566e497740789575275d0cdca0db (table default.Table_ee1ad081d0c846ca9fdc08acfb8318e9 [id=1794b6e2ac9443718c798588325d4a2d]) successfully deleted
I20250903 20:05:39.469306 27687 ts_tablet_manager.cc:1918] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:39.469372 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1175) took 3136 ms. Trace:
I20250903 20:05:39.469554 27684 rpcz_store.cc:276] 0903 20:05:36.333335 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.333419 (+ 84us) service_pool.cc:225] Handling call
0903 20:05:39.469156 (+3135737us) ts_tablet_manager.cc:1145] Deleting tablet fa544dfdfff04a1e97676e562aeb0f40
0903 20:05:39.469170 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.469313 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3135541}
W20250903 20:05:39.470175 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet fa544dfdfff04a1e97676e562aeb0f40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fa544dfdfff04a1e97676e562aeb0f40 already in progress: creating tablet
I20250903 20:05:39.470813 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1176) took 3137 ms. Trace:
I20250903 20:05:39.470989 27684 rpcz_store.cc:276] 0903 20:05:36.333653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.333720 (+ 67us) service_pool.cc:225] Handling call
0903 20:05:39.470625 (+3136905us) ts_tablet_manager.cc:1145] Deleting tablet 7c6780770f104f669102b051ee7faeb5
0903 20:05:39.470636 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.470737 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3136764}
I20250903 20:05:39.471328 27687 log.cc:1199] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/a3103e1595ec4c38ba65dca26fe1167c
I20250903 20:05:39.471468 27684 tablet_replica.cc:331] T 39c2ebe4ada5426590ec3769c8662644 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
W20250903 20:05:39.471592 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 7c6780770f104f669102b051ee7faeb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c6780770f104f669102b051ee7faeb5 already in progress: creating tablet
I20250903 20:05:39.471788 27684 raft_consensus.cc:2241] T 39c2ebe4ada5426590ec3769c8662644 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:39.471863 27687 ts_tablet_manager.cc:1939] T a3103e1595ec4c38ba65dca26fe1167c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:39.472812 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1183) took 3382 ms. Trace:
I20250903 20:05:39.472064 27684 raft_consensus.cc:2270] T 39c2ebe4ada5426590ec3769c8662644 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:39.478787 27684 ts_tablet_manager.cc:1905] T 39c2ebe4ada5426590ec3769c8662644 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:39.479733 27692 tablet_bootstrap.cc:492] T 8400e23873fd4be18de2998afc64a689 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:39.479934 27692 ts_tablet_manager.cc:1397] T 8400e23873fd4be18de2998afc64a689 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.074s user 0.004s sys 0.000s
W20250903 20:05:39.480289 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 361156ca43a444b1b07e1d1fab48c234 consensus state has no leader
I20250903 20:05:39.480729 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_c3efd8f7384b448483d44f2e5a41ef67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:39.480792 27692 raft_consensus.cc:357] T 8400e23873fd4be18de2998afc64a689 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.480970 27692 raft_consensus.cc:383] T 8400e23873fd4be18de2998afc64a689 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.481065 27692 raft_consensus.cc:738] T 8400e23873fd4be18de2998afc64a689 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
W20250903 20:05:39.481283 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3efd8f7384b448483d44f2e5a41ef67 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:39.481447 27692 consensus_queue.cc:260] T 8400e23873fd4be18de2998afc64a689 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.482331 27692 ts_tablet_manager.cc:1428] T 8400e23873fd4be18de2998afc64a689 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:39.482688 27692 tablet_bootstrap.cc:492] T 8218868ff40b405387d560f614427c08 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:39.483553 29443 raft_consensus.cc:491] T ef02f91c0964446daf62ac1dde0330ff P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.483726 29443 raft_consensus.cc:513] T ef02f91c0964446daf62ac1dde0330ff P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.484388 29443 leader_election.cc:290] T ef02f91c0964446daf62ac1dde0330ff P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:39.484571 27692 tablet_bootstrap.cc:654] T 8218868ff40b405387d560f614427c08 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.485726 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef02f91c0964446daf62ac1dde0330ff" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:39.486810 27462 leader_election.cc:343] T ef02f91c0964446daf62ac1dde0330ff P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.485219 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef02f91c0964446daf62ac1dde0330ff" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:39.490705 27687 rpcz_store.cc:276] 0903 20:05:36.090791 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.094268 (+ 3477us) service_pool.cc:225] Handling call
0903 20:05:39.434562 (+3340294us) ts_tablet_manager.cc:1145] Deleting tablet a3103e1595ec4c38ba65dca26fe1167c
0903 20:05:39.434579 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.451707 (+ 17128us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.469199 (+ 17492us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.472788 (+ 3589us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3339943}
W20250903 20:05:39.490886 27462 leader_election.cc:343] T ef02f91c0964446daf62ac1dde0330ff P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.491024 27462 leader_election.cc:304] T ef02f91c0964446daf62ac1dde0330ff P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:39.491202 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet a3103e1595ec4c38ba65dca26fe1167c (table default.Table_fdf0ad621407452eb74b6a9a56be2e85 [id=e4270da20ece47aaa8897f47a02d86d8]) successfully deleted
I20250903 20:05:39.491623 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1194) took 3184 ms. Trace:
I20250903 20:05:39.491549 29443 raft_consensus.cc:2747] T ef02f91c0964446daf62ac1dde0330ff P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:39.493076 27497 tablet_service.cc:1468] Processing CreateTablet for tablet ce3bb86fe2f74549934504421254183e (DEFAULT_TABLE table=default.Table_8a7162fd4e2540d5a4b2a7bccac7af08 [id=fe4a1e13e1b7435b9d1c7e2993cf9f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.493427 27347 tablet_service.cc:1468] Processing CreateTablet for tablet ea3d1d66d6eb4665aedc7758f5bb105e (DEFAULT_TABLE table=default.Table_ca622380edab4739afde521d34a1cd81 [id=83fa01ec87f2407f9251b57cfbd9ca6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.493516 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce3bb86fe2f74549934504421254183e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:39.493782 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea3d1d66d6eb4665aedc7758f5bb105e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:39.495350 27353 tablet_service.cc:1468] Processing CreateTablet for tablet 1853e8f8d4464603808f9ab32b98cdff (DEFAULT_TABLE table=default.Table_75162e2d436048639aa708c34fc68ec1 [id=f6504379bee24447ba57969ece15a01f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.495700 27353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1853e8f8d4464603808f9ab32b98cdff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:39.497148 27357 tablet_service.cc:1468] Processing CreateTablet for tablet b86ef44e7a564eee88d78a9afe72dec1 (DEFAULT_TABLE table=default.Table_dd4b90aea2be4292899c22877f734f3a [id=3919441a1a434f198df6c0a07de82d7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.497476 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b86ef44e7a564eee88d78a9afe72dec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:39.441519 29452 leader_election.cc:290] T 46f49fd718be4a2996d781ab31268217 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:39.498486 27349 tablet_service.cc:1468] Processing CreateTablet for tablet 0618314b3e5c4be6a7f801af4c281346 (DEFAULT_TABLE table=default.Table_3f0e8bdd369f4e8c89ae8ee2ef5dd22f [id=10d06100a5c54cd08dafd1db7a7474b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.498786 27349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0618314b3e5c4be6a7f801af4c281346. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:39.442879 27183 consensus_peers.cc:489] T 64c4dd2645034a7eaf243f5474603f10 P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 64c4dd2645034a7eaf243f5474603f10. This is attempt 1: this message will repeat every 5th retry.
W20250903 20:05:39.502694 27183 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 59877.3us
I20250903 20:05:39.445580 27685 raft_consensus.cc:2270] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:39.503575 27360 tablet_service.cc:1468] Processing CreateTablet for tablet 5ae2ea414bf74201ba378ff6c3663e27 (DEFAULT_TABLE table=default.Table_8a742d84a1dd47c1a154c8e4bd43dc79 [id=a557bcb450e9413e853926e3d0b18be4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.503723 27362 tablet_service.cc:1468] Processing CreateTablet for tablet 45c0b69ed3cb44aebb9f0ca93f6ea81c (DEFAULT_TABLE table=default.Table_74c673867bf9445195f037ede94c26b9 [id=6d40fd789f2a416a87b17aa0a6ebcd72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.503953 27360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ae2ea414bf74201ba378ff6c3663e27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:39.504158 27685 ts_tablet_manager.cc:1905] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:39.504175 27359 tablet_service.cc:1468] Processing CreateTablet for tablet 97bc32f5b49a4bcfb8f5702da4a8e722 (DEFAULT_TABLE table=default.Table_1153bf74b285448b88b364efe2169784 [id=f321a32c1ac043238c2fc92bfd5ec1ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.504459 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97bc32f5b49a4bcfb8f5702da4a8e722. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:39.506515 27365 tablet_service.cc:1468] Processing CreateTablet for tablet b5647d22ab97477db760931a8ced773e (DEFAULT_TABLE table=default.Table_ab6863cab5e04e57b58bc14d9a3d379d [id=d7e4c722b55345bbb7451e2e9a10542d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.506821 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5647d22ab97477db760931a8ced773e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:39.508440 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45c0b69ed3cb44aebb9f0ca93f6ea81c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:39.509032 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 3efca58ed9de439bacef7cbb1376c506 (DEFAULT_TABLE table=default.Table_041cccd0d04d430c8c06d135afd3b91e [id=3072e25c24334234914acaa1bd6aa58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.509332 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3efca58ed9de439bacef7cbb1376c506. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:39.514427 27503 tablet_service.cc:1468] Processing CreateTablet for tablet 97bc32f5b49a4bcfb8f5702da4a8e722 (DEFAULT_TABLE table=default.Table_1153bf74b285448b88b364efe2169784 [id=f321a32c1ac043238c2fc92bfd5ec1ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.514576 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 1853e8f8d4464603808f9ab32b98cdff (DEFAULT_TABLE table=default.Table_75162e2d436048639aa708c34fc68ec1 [id=f6504379bee24447ba57969ece15a01f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.514896 27503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97bc32f5b49a4bcfb8f5702da4a8e722. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:39.515157 27492 tablet_service.cc:1468] Processing CreateTablet for tablet b86ef44e7a564eee88d78a9afe72dec1 (DEFAULT_TABLE table=default.Table_dd4b90aea2be4292899c22877f734f3a [id=3919441a1a434f198df6c0a07de82d7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.515693 27492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b86ef44e7a564eee88d78a9afe72dec1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:39.449384 27184 leader_election.cc:343] T 46f49fd718be4a2996d781ab31268217 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 46f49fd718be4a2996d781ab31268217
W20250903 20:05:39.517977 27184 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 68617.7us
I20250903 20:05:39.518571 27500 tablet_service.cc:1468] Processing CreateTablet for tablet 0618314b3e5c4be6a7f801af4c281346 (DEFAULT_TABLE table=default.Table_3f0e8bdd369f4e8c89ae8ee2ef5dd22f [id=10d06100a5c54cd08dafd1db7a7474b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.518939 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0618314b3e5c4be6a7f801af4c281346. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:39.520354 27684 ts_tablet_manager.cc:1918] T 39c2ebe4ada5426590ec3769c8662644 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:39.520445 27684 log.cc:1199] T 39c2ebe4ada5426590ec3769c8662644 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/39c2ebe4ada5426590ec3769c8662644
I20250903 20:05:39.520790 27684 ts_tablet_manager.cc:1939] T 39c2ebe4ada5426590ec3769c8662644 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:39.521652 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1177) took 3187 ms. Trace:
I20250903 20:05:39.521740 27684 rpcz_store.cc:276] 0903 20:05:36.334069 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.334132 (+ 63us) service_pool.cc:225] Handling call
0903 20:05:39.471375 (+3137243us) ts_tablet_manager.cc:1145] Deleting tablet 39c2ebe4ada5426590ec3769c8662644
0903 20:05:39.471385 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.514602 (+ 43217us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.520273 (+ 5671us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.521628 (+ 1355us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3137057}
I20250903 20:05:39.522117 27684 tablet_replica.cc:331] T d4c703dafbca4e669dbb18946ffac699 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:39.522552 27490 tablet_service.cc:1468] Processing CreateTablet for tablet b5647d22ab97477db760931a8ced773e (DEFAULT_TABLE table=default.Table_ab6863cab5e04e57b58bc14d9a3d379d [id=d7e4c722b55345bbb7451e2e9a10542d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.522912 27490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5647d22ab97477db760931a8ced773e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:39.523864 29371 raft_consensus.cc:491] T 659a127405d341c7b074d3445854256f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.523983 29371 raft_consensus.cc:513] T 659a127405d341c7b074d3445854256f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.524600 29371 leader_election.cc:290] T 659a127405d341c7b074d3445854256f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:39.525683 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "659a127405d341c7b074d3445854256f" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:39.526582 27462 leader_election.cc:343] T 659a127405d341c7b074d3445854256f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.492789 27687 rpcz_store.cc:276] 0903 20:05:36.306828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.306928 (+ 100us) service_pool.cc:225] Handling call
0903 20:05:39.491493 (+3184565us) ts_tablet_manager.cc:1145] Deleting tablet fa544dfdfff04a1e97676e562aeb0f40
0903 20:05:39.491506 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.491605 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3161896}
W20250903 20:05:39.527109 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet fa544dfdfff04a1e97676e562aeb0f40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fa544dfdfff04a1e97676e562aeb0f40 already in progress: creating tablet
I20250903 20:05:39.527405 27687 tablet_replica.cc:331] T 4e34d6d5d68e4395aed78821ea077c1b P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:39.527571 27687 raft_consensus.cc:2241] T 4e34d6d5d68e4395aed78821ea077c1b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:39.527725 27687 raft_consensus.cc:2270] T 4e34d6d5d68e4395aed78821ea077c1b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:39.530098 27687 ts_tablet_manager.cc:1905] T 4e34d6d5d68e4395aed78821ea077c1b P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:39.530375 27505 tablet_service.cc:1468] Processing CreateTablet for tablet 5ae2ea414bf74201ba378ff6c3663e27 (DEFAULT_TABLE table=default.Table_8a742d84a1dd47c1a154c8e4bd43dc79 [id=a557bcb450e9413e853926e3d0b18be4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.530773 27505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ae2ea414bf74201ba378ff6c3663e27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:39.531514 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 1853e8f8d4464603808f9ab32b98cdff (DEFAULT_TABLE table=default.Table_75162e2d436048639aa708c34fc68ec1 [id=f6504379bee24447ba57969ece15a01f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.531888 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1853e8f8d4464603808f9ab32b98cdff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:39.533344 27684 raft_consensus.cc:2241] T d4c703dafbca4e669dbb18946ffac699 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:39.533447 29371 raft_consensus.cc:491] T 6d13884fff8c43b690836c6a341a2d92 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.533524 27684 raft_consensus.cc:2270] T d4c703dafbca4e669dbb18946ffac699 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:39.533563 27493 tablet_service.cc:1468] Processing CreateTablet for tablet 3efca58ed9de439bacef7cbb1376c506 (DEFAULT_TABLE table=default.Table_041cccd0d04d430c8c06d135afd3b91e [id=3072e25c24334234914acaa1bd6aa58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.533598 29371 raft_consensus.cc:513] T 6d13884fff8c43b690836c6a341a2d92 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.533931 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3efca58ed9de439bacef7cbb1376c506. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:39.534198 29371 leader_election.cc:290] T 6d13884fff8c43b690836c6a341a2d92 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:39.534999 27376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d13884fff8c43b690836c6a341a2d92" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:39.535527 27462 leader_election.cc:343] T 6d13884fff8c43b690836c6a341a2d92 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.538550 27495 tablet_service.cc:1468] Processing CreateTablet for tablet 45c0b69ed3cb44aebb9f0ca93f6ea81c (DEFAULT_TABLE table=default.Table_74c673867bf9445195f037ede94c26b9 [id=6d40fd789f2a416a87b17aa0a6ebcd72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.539008 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45c0b69ed3cb44aebb9f0ca93f6ea81c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:39.539700 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d13884fff8c43b690836c6a341a2d92" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:39.540298 27222 tablet_service.cc:1468] Processing CreateTablet for tablet b86ef44e7a564eee88d78a9afe72dec1 (DEFAULT_TABLE table=default.Table_dd4b90aea2be4292899c22877f734f3a [id=3919441a1a434f198df6c0a07de82d7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.540854 27222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b86ef44e7a564eee88d78a9afe72dec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:39.542433 27224 tablet_service.cc:1468] Processing CreateTablet for tablet ea3d1d66d6eb4665aedc7758f5bb105e (DEFAULT_TABLE table=default.Table_ca622380edab4739afde521d34a1cd81 [id=83fa01ec87f2407f9251b57cfbd9ca6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.542789 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea3d1d66d6eb4665aedc7758f5bb105e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:39.544703 27462 leader_election.cc:343] T 6d13884fff8c43b690836c6a341a2d92 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.544831 27462 leader_election.cc:304] T 6d13884fff8c43b690836c6a341a2d92 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:39.545284 29371 raft_consensus.cc:2747] T 6d13884fff8c43b690836c6a341a2d92 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:39.545473 27504 tablet_service.cc:1468] Processing CreateTablet for tablet ea3d1d66d6eb4665aedc7758f5bb105e (DEFAULT_TABLE table=default.Table_ca622380edab4739afde521d34a1cd81 [id=83fa01ec87f2407f9251b57cfbd9ca6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.545862 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea3d1d66d6eb4665aedc7758f5bb105e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:39.546041 27228 tablet_service.cc:1468] Processing CreateTablet for tablet 0618314b3e5c4be6a7f801af4c281346 (DEFAULT_TABLE table=default.Table_3f0e8bdd369f4e8c89ae8ee2ef5dd22f [id=10d06100a5c54cd08dafd1db7a7474b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.546447 27228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0618314b3e5c4be6a7f801af4c281346. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:39.546641 27684 ts_tablet_manager.cc:1905] T d4c703dafbca4e669dbb18946ffac699 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:39.549024 27693 tablet_bootstrap.cc:492] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:39.549190 27693 ts_tablet_manager.cc:1397] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.111s user 0.002s sys 0.001s
I20250903 20:05:39.549923 27693 raft_consensus.cc:357] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.550099 27693 raft_consensus.cc:383] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.550143 27693 raft_consensus.cc:738] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.550401 27693 consensus_queue.cc:260] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.551162 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1853e8f8d4464603808f9ab32b98cdff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:39.553517 27693 ts_tablet_manager.cc:1428] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:39.553943 27687 ts_tablet_manager.cc:1918] T 4e34d6d5d68e4395aed78821ea077c1b P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:39.554049 27687 log.cc:1199] T 4e34d6d5d68e4395aed78821ea077c1b P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/4e34d6d5d68e4395aed78821ea077c1b
I20250903 20:05:39.553957 27693 tablet_bootstrap.cc:492] T d4c703dafbca4e669dbb18946ffac699 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:39.556638 27693 tablet_bootstrap.cc:654] T d4c703dafbca4e669dbb18946ffac699 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.557281 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 39c2ebe4ada5426590ec3769c8662644 (table default.Table_d84f067d5a924a23964078d360422878 [id=5cb314181f104d63b1aed98c7198d5c3]) successfully deleted
I20250903 20:05:39.560740 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "659a127405d341c7b074d3445854256f" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:39.561275 27183 leader_election.cc:343] T 46f49fd718be4a2996d781ab31268217 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 46f49fd718be4a2996d781ab31268217
I20250903 20:05:39.561537 27183 leader_election.cc:304] T 46f49fd718be4a2996d781ab31268217 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:39.561842 27183 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.26.65.67:46231, user_credentials={real_user=slave}} blocked reactor thread for 58512.5us
I20250903 20:05:39.562300 27229 tablet_service.cc:1468] Processing CreateTablet for tablet b5647d22ab97477db760931a8ced773e (DEFAULT_TABLE table=default.Table_ab6863cab5e04e57b58bc14d9a3d379d [id=d7e4c722b55345bbb7451e2e9a10542d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.562458 27211 tablet_service.cc:1468] Processing CreateTablet for tablet 3efca58ed9de439bacef7cbb1376c506 (DEFAULT_TABLE table=default.Table_041cccd0d04d430c8c06d135afd3b91e [id=3072e25c24334234914acaa1bd6aa58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.562765 27211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3efca58ed9de439bacef7cbb1376c506. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:39.563367 27462 leader_election.cc:343] T 659a127405d341c7b074d3445854256f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.563499 27462 leader_election.cc:304] T 659a127405d341c7b074d3445854256f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:39.564196 29371 raft_consensus.cc:2747] T 659a127405d341c7b074d3445854256f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:39.564185 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 45c0b69ed3cb44aebb9f0ca93f6ea81c (DEFAULT_TABLE table=default.Table_74c673867bf9445195f037ede94c26b9 [id=6d40fd789f2a416a87b17aa0a6ebcd72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.564549 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45c0b69ed3cb44aebb9f0ca93f6ea81c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:39.564841 27218 tablet_service.cc:1468] Processing CreateTablet for tablet 97bc32f5b49a4bcfb8f5702da4a8e722 (DEFAULT_TABLE table=default.Table_1153bf74b285448b88b364efe2169784 [id=f321a32c1ac043238c2fc92bfd5ec1ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.565183 27218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97bc32f5b49a4bcfb8f5702da4a8e722. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:39.566883 27184 consensus_peers.cc:489] T 4e34d6d5d68e4395aed78821ea077c1b P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:39.567206 29411 consensus_queue.cc:1035] T 30cc0d4ba8d444f1b98232f906ea0c7b P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:39.572850 27684 ts_tablet_manager.cc:1918] T d4c703dafbca4e669dbb18946ffac699 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:39.573055 29461 raft_consensus.cc:2747] T 46f49fd718be4a2996d781ab31268217 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:39.573173 27229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5647d22ab97477db760931a8ced773e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:39.573240 27210 tablet_service.cc:1468] Processing CreateTablet for tablet 5ae2ea414bf74201ba378ff6c3663e27 (DEFAULT_TABLE table=default.Table_8a742d84a1dd47c1a154c8e4bd43dc79 [id=a557bcb450e9413e853926e3d0b18be4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.573570 27210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ae2ea414bf74201ba378ff6c3663e27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:39.579775 27216 tablet_service.cc:1468] Processing CreateTablet for tablet ce3bb86fe2f74549934504421254183e (DEFAULT_TABLE table=default.Table_8a7162fd4e2540d5a4b2a7bccac7af08 [id=fe4a1e13e1b7435b9d1c7e2993cf9f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:39.580281 27216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce3bb86fe2f74549934504421254183e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:39.569339 27320 consensus_peers.cc:489] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 47c1287bff1145a5b2ab6d14dbd608c7. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:39.587220 27379 raft_consensus.cc:3058] T 3919f7f95a954f59b68f56d0051c0109 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:39.579607 29170 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 20:05:39.589461 29170 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 20:05:39.589785 29170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 20:05:39.591629 27687 ts_tablet_manager.cc:1939] T 4e34d6d5d68e4395aed78821ea077c1b P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:39.593489 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1196) took 3186 ms. Trace:
I20250903 20:05:39.593672 27687 rpcz_store.cc:276] 0903 20:05:36.406905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.406963 (+ 58us) service_pool.cc:225] Handling call
0903 20:05:39.527361 (+3120398us) ts_tablet_manager.cc:1145] Deleting tablet 4e34d6d5d68e4395aed78821ea077c1b
0903 20:05:39.527370 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.550993 (+ 23623us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.553839 (+ 2846us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.593468 (+ 39629us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3120175}
I20250903 20:05:39.594206 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 4e34d6d5d68e4395aed78821ea077c1b (table default.Table_f2505e03caaa40c4b17d2e60b0196843 [id=08f0b93efef74aa59b1ec3c286b3b45b]) successfully deleted
I20250903 20:05:39.594854 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1197) took 3187 ms. Trace:
I20250903 20:05:39.594983 27687 rpcz_store.cc:276] 0903 20:05:36.407566 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.407617 (+ 51us) service_pool.cc:225] Handling call
0903 20:05:39.594719 (+3187102us) ts_tablet_manager.cc:1145] Deleting tablet 2eba38dd30454b26b2c0754e5fa05c16
0903 20:05:39.594730 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.594840 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3186951}
W20250903 20:05:39.595510 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 2eba38dd30454b26b2c0754e5fa05c16 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2eba38dd30454b26b2c0754e5fa05c16 already in progress: creating tablet
I20250903 20:05:39.595886 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1198) took 3187 ms. Trace:
I20250903 20:05:39.595990 27687 rpcz_store.cc:276] 0903 20:05:36.408523 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.408570 (+ 47us) service_pool.cc:225] Handling call
0903 20:05:39.595788 (+3187218us) ts_tablet_manager.cc:1145] Deleting tablet 2dc0f714cdb44ece90615de5a11bc71f
0903 20:05:39.595797 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.595871 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3187060}
W20250903 20:05:39.596422 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 2dc0f714cdb44ece90615de5a11bc71f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2dc0f714cdb44ece90615de5a11bc71f already in progress: creating tablet
I20250903 20:05:39.597216 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1199) took 3187 ms. Trace:
I20250903 20:05:39.597332 27687 rpcz_store.cc:276] 0903 20:05:36.409874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.409919 (+ 45us) service_pool.cc:225] Handling call
0903 20:05:39.597102 (+3187183us) ts_tablet_manager.cc:1145] Deleting tablet 17f16e9c8f794e03a2f8b3b6b35f8e84
0903 20:05:39.597120 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.597200 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3187023}
W20250903 20:05:39.597711 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 17f16e9c8f794e03a2f8b3b6b35f8e84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17f16e9c8f794e03a2f8b3b6b35f8e84 already in progress: creating tablet
I20250903 20:05:39.598055 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1195) took 3192 ms. Trace:
I20250903 20:05:39.598165 27687 rpcz_store.cc:276] 0903 20:05:36.406061 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.409366 (+ 3305us) service_pool.cc:225] Handling call
0903 20:05:39.597961 (+3188595us) ts_tablet_manager.cc:1145] Deleting tablet 7c6780770f104f669102b051ee7faeb5
0903 20:05:39.597972 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.598042 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3187580}
W20250903 20:05:39.598572 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 7c6780770f104f669102b051ee7faeb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c6780770f104f669102b051ee7faeb5 already in progress: creating tablet
I20250903 20:05:39.598930 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1201) took 2980 ms. Trace:
I20250903 20:05:39.599028 27687 rpcz_store.cc:276] 0903 20:05:36.642968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.648037 (+ 5069us) service_pool.cc:225] Handling call
0903 20:05:39.598836 (+2950799us) ts_tablet_manager.cc:1145] Deleting tablet 827d186faaa140bdab2f87cf421ca64c
0903 20:05:39.598845 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.598916 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2950521}
W20250903 20:05:39.599416 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 827d186faaa140bdab2f87cf421ca64c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 827d186faaa140bdab2f87cf421ca64c already in progress: creating tablet
I20250903 20:05:39.599685 27687 tablet_replica.cc:331] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:39.591053 29443 raft_consensus.cc:491] T 8400e23873fd4be18de2998afc64a689 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.614938 29443 raft_consensus.cc:513] T 8400e23873fd4be18de2998afc64a689 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.615535 27695 tablet_bootstrap.cc:492] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:39.615648 29443 leader_election.cc:290] T 8400e23873fd4be18de2998afc64a689 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:39.615693 27695 ts_tablet_manager.cc:1397] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.188s user 0.012s sys 0.009s
I20250903 20:05:39.616866 27695 raft_consensus.cc:357] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:39.617131 27684 log.cc:1199] T d4c703dafbca4e669dbb18946ffac699 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/d4c703dafbca4e669dbb18946ffac699
I20250903 20:05:39.616940 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8400e23873fd4be18de2998afc64a689" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:39.617507 27684 ts_tablet_manager.cc:1939] T d4c703dafbca4e669dbb18946ffac699 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:39.617990 27695 raft_consensus.cc:383] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.618167 27695 raft_consensus.cc:738] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
W20250903 20:05:39.617849 27462 leader_election.cc:343] T 8400e23873fd4be18de2998afc64a689 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.618526 27695 consensus_queue.cc:260] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:39.619079 27692 tablet_bootstrap.cc:492] T 8218868ff40b405387d560f614427c08 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:39.619666 27695 ts_tablet_manager.cc:1428] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:39.611224 27693 tablet_bootstrap.cc:492] T d4c703dafbca4e669dbb18946ffac699 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:39.620218 27695 tablet_bootstrap.cc:492] T 39c2ebe4ada5426590ec3769c8662644 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:39.620254 27693 ts_tablet_manager.cc:1397] T d4c703dafbca4e669dbb18946ffac699 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.066s user 0.003s sys 0.001s
I20250903 20:05:39.621098 27693 raft_consensus.cc:357] T d4c703dafbca4e669dbb18946ffac699 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.621315 27693 raft_consensus.cc:383] T d4c703dafbca4e669dbb18946ffac699 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.621374 27693 raft_consensus.cc:738] T d4c703dafbca4e669dbb18946ffac699 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.621803 29411 raft_consensus.cc:491] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.621745 27693 consensus_queue.cc:260] T d4c703dafbca4e669dbb18946ffac699 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.621949 29411 raft_consensus.cc:513] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:39.622364 27693 ts_tablet_manager.cc:1428] T d4c703dafbca4e669dbb18946ffac699 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250903 20:05:39.622638 29411 leader_election.cc:290] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:39.625697 27376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8400e23873fd4be18de2998afc64a689" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:39.626012 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6800e0b6269045f29682704fd847d5d1" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:39.626402 27687 raft_consensus.cc:2241] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:39.626839 27687 raft_consensus.cc:2270] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:39.627314 27184 leader_election.cc:343] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.627504 27687 ts_tablet_manager.cc:1905] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:39.619686 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1178) took 3285 ms. Trace:
I20250903 20:05:39.625516 27509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6800e0b6269045f29682704fd847d5d1" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:39.629580 27692 ts_tablet_manager.cc:1397] T 8218868ff40b405387d560f614427c08 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.147s user 0.016s sys 0.003s
I20250903 20:05:39.629690 27509 raft_consensus.cc:2466] T 6800e0b6269045f29682704fd847d5d1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:39.629594 27684 rpcz_store.cc:276] 0903 20:05:36.334343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.334392 (+ 49us) service_pool.cc:225] Handling call
0903 20:05:39.522067 (+3187675us) ts_tablet_manager.cc:1145] Deleting tablet d4c703dafbca4e669dbb18946ffac699
0903 20:05:39.522077 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.549384 (+ 27307us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.572781 (+ 23397us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.619662 (+ 46881us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3187545}
W20250903 20:05:39.626942 27462 leader_election.cc:343] T 8400e23873fd4be18de2998afc64a689 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.629922 27462 leader_election.cc:304] T 8400e23873fd4be18de2998afc64a689 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:39.630134 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet d4c703dafbca4e669dbb18946ffac699 (table default.Table_ccda245b73064dc2b5634f9593b1888c [id=e6c1c8665bd841b5a932b2d54f9be4e1]) successfully deleted
I20250903 20:05:39.630389 27692 raft_consensus.cc:357] T 8218868ff40b405387d560f614427c08 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.630599 27692 raft_consensus.cc:383] T 8218868ff40b405387d560f614427c08 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.630658 27692 raft_consensus.cc:738] T 8218868ff40b405387d560f614427c08 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.630895 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1180) took 3222 ms. Trace:
I20250903 20:05:39.630860 27692 consensus_queue.cc:260] T 8218868ff40b405387d560f614427c08 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.630973 27684 rpcz_store.cc:276] 0903 20:05:36.408777 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.408901 (+ 124us) service_pool.cc:225] Handling call
0903 20:05:39.630716 (+3221815us) ts_tablet_manager.cc:1145] Deleting tablet 2dc0f714cdb44ece90615de5a11bc71f
0903 20:05:39.630731 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.630879 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3221580}
I20250903 20:05:39.631100 29371 raft_consensus.cc:2747] T 8400e23873fd4be18de2998afc64a689 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:39.631448 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 2dc0f714cdb44ece90615de5a11bc71f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2dc0f714cdb44ece90615de5a11bc71f already in progress: creating tablet
I20250903 20:05:39.632112 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1179) took 3225 ms. Trace:
I20250903 20:05:39.632236 27684 rpcz_store.cc:276] 0903 20:05:36.406522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.409196 (+ 2674us) service_pool.cc:225] Handling call
0903 20:05:39.631685 (+3222489us) ts_tablet_manager.cc:1145] Deleting tablet 2eba38dd30454b26b2c0754e5fa05c16
0903 20:05:39.631695 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.632093 (+ 398us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3221966}
I20250903 20:05:39.632543 27183 leader_election.cc:304] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
W20250903 20:05:39.632779 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 2eba38dd30454b26b2c0754e5fa05c16 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2eba38dd30454b26b2c0754e5fa05c16 already in progress: creating tablet
I20250903 20:05:39.632855 29411 raft_consensus.cc:2802] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:39.632949 29411 raft_consensus.cc:491] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:39.633024 29411 raft_consensus.cc:3058] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:39.633188 27684 tablet_replica.cc:331] T 30cc0d4ba8d444f1b98232f906ea0c7b P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:39.633451 27684 raft_consensus.cc:2241] T 30cc0d4ba8d444f1b98232f906ea0c7b P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:39.633615 27684 raft_consensus.cc:2270] T 30cc0d4ba8d444f1b98232f906ea0c7b P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:39.634196 27684 ts_tablet_manager.cc:1905] T 30cc0d4ba8d444f1b98232f906ea0c7b P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:39.635919 29411 raft_consensus.cc:513] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:39.636593 29411 leader_election.cc:290] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:39.636866 28567 raft_consensus.cc:491] T 4739f10128d64e2c986257b1d9deb90f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.637017 28567 raft_consensus.cc:513] T 4739f10128d64e2c986257b1d9deb90f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.637636 28567 leader_election.cc:290] T 4739f10128d64e2c986257b1d9deb90f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:39.637665 27509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6800e0b6269045f29682704fd847d5d1" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:39.637830 27509 raft_consensus.cc:3058] T 6800e0b6269045f29682704fd847d5d1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:39.638384 27692 ts_tablet_manager.cc:1428] T 8218868ff40b405387d560f614427c08 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250903 20:05:39.638756 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6800e0b6269045f29682704fd847d5d1" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:39.639315 27692 tablet_bootstrap.cc:492] T 314856bf269f4df7aa25f5a30cec0d5b P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
W20250903 20:05:39.639304 27184 leader_election.cc:343] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.640609 27509 raft_consensus.cc:2466] T 6800e0b6269045f29682704fd847d5d1 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:39.641404 27509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4739f10128d64e2c986257b1d9deb90f" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:39.641602 27509 raft_consensus.cc:2391] T 4739f10128d64e2c986257b1d9deb90f P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 51578fb7bceb4e119b3ab1e58c4f5528 in current term 1: Already voted for candidate 89f8f0ee68b940db9ce250c0604f934a in this term.
I20250903 20:05:39.642310 27183 leader_election.cc:304] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:39.642779 29411 raft_consensus.cc:2802] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:39.643213 29411 raft_consensus.cc:695] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:39.645531 29411 consensus_queue.cc:237] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:39.648299 27693 tablet_bootstrap.cc:492] T 5b46bb34ac214b1ba23fba1a33f746ac P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:39.650291 27685 ts_tablet_manager.cc:1918] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:39.656162 27693 tablet_bootstrap.cc:654] T 5b46bb34ac214b1ba23fba1a33f746ac P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.657389 27685 log.cc:1199] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/548b50819f694fa3ad508da9f442a99c
I20250903 20:05:39.656630 27687 ts_tablet_manager.cc:1918] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 20:05:39.658617 27687 log.cc:1199] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/0f5b87c1c4a24f8abdbfd6194f3b2c4a
I20250903 20:05:39.659196 27687 ts_tablet_manager.cc:1939] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:39.659317 29443 raft_consensus.cc:491] T 615f42e95bb24c8480f73d371fd63b9c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.659520 29443 raft_consensus.cc:513] T 615f42e95bb24c8480f73d371fd63b9c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.660315 29443 leader_election.cc:290] T 615f42e95bb24c8480f73d371fd63b9c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:39.661470 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "615f42e95bb24c8480f73d371fd63b9c" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:39.661623 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "615f42e95bb24c8480f73d371fd63b9c" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:39.661685 27249 raft_consensus.cc:2466] T 615f42e95bb24c8480f73d371fd63b9c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
W20250903 20:05:39.662277 27462 leader_election.cc:343] T 615f42e95bb24c8480f73d371fd63b9c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.662478 27462 leader_election.cc:304] T 615f42e95bb24c8480f73d371fd63b9c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:39.662901 27685 ts_tablet_manager.cc:1939] T 548b50819f694fa3ad508da9f442a99c P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:39.662940 29443 raft_consensus.cc:2802] T 615f42e95bb24c8480f73d371fd63b9c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:39.663010 29443 raft_consensus.cc:491] T 615f42e95bb24c8480f73d371fd63b9c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:39.663075 29443 raft_consensus.cc:3058] T 615f42e95bb24c8480f73d371fd63b9c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:39.666934 29443 raft_consensus.cc:513] T 615f42e95bb24c8480f73d371fd63b9c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.667917 29443 leader_election.cc:290] T 615f42e95bb24c8480f73d371fd63b9c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:39.668707 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "615f42e95bb24c8480f73d371fd63b9c" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:39.668745 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "615f42e95bb24c8480f73d371fd63b9c" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:39.668991 27249 raft_consensus.cc:3058] T 615f42e95bb24c8480f73d371fd63b9c P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:39.669911 27379 raft_consensus.cc:3058] T d4c703dafbca4e669dbb18946ffac699 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:39.672016 27249 raft_consensus.cc:2466] T 615f42e95bb24c8480f73d371fd63b9c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
W20250903 20:05:39.669449 27462 leader_election.cc:343] T 615f42e95bb24c8480f73d371fd63b9c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.672771 27462 leader_election.cc:304] T 615f42e95bb24c8480f73d371fd63b9c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:39.673067 29371 raft_consensus.cc:2802] T 615f42e95bb24c8480f73d371fd63b9c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:39.676432 27692 tablet_bootstrap.cc:654] T 314856bf269f4df7aa25f5a30cec0d5b P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.677582 29452 consensus_queue.cc:1035] T d4c703dafbca4e669dbb18946ffac699 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:39.680711 27684 ts_tablet_manager.cc:1918] T 30cc0d4ba8d444f1b98232f906ea0c7b P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:39.682564 27684 log.cc:1199] T 30cc0d4ba8d444f1b98232f906ea0c7b P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/30cc0d4ba8d444f1b98232f906ea0c7b
W20250903 20:05:39.682920 27183 consensus_peers.cc:489] T d4c703dafbca4e669dbb18946ffac699 P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4c703dafbca4e669dbb18946ffac699. This is attempt 1: this message will repeat every 5th retry.
W20250903 20:05:39.684405 27183 consensus_peers.cc:489] T 30cc0d4ba8d444f1b98232f906ea0c7b P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:39.684832 27684 ts_tablet_manager.cc:1939] T 30cc0d4ba8d444f1b98232f906ea0c7b P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:39.686978 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4739f10128d64e2c986257b1d9deb90f" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:39.687873 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1056) took 5305 ms. Trace:
I20250903 20:05:39.688043 27685 rpcz_store.cc:276] 0903 20:05:34.382890 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.382939 (+ 49us) service_pool.cc:225] Handling call
0903 20:05:39.426174 (+5043235us) ts_tablet_manager.cc:1145] Deleting tablet 548b50819f694fa3ad508da9f442a99c
0903 20:05:39.426184 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.560299 (+134115us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.650107 (+ 89808us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.687855 (+ 37748us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41757,"tablet-delete.queue_time_us":5042977}
I20250903 20:05:39.688659 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 548b50819f694fa3ad508da9f442a99c (table default.Table_b838fda30d4043a5ad578ec527c1d105 [id=1eb6b859ca0049699466cc72e5a2b6f7]) successfully deleted
I20250903 20:05:39.689872 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1182) took 3256 ms. Trace:
I20250903 20:05:39.690044 27692 tablet_bootstrap.cc:492] T 314856bf269f4df7aa25f5a30cec0d5b P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:39.690591 27692 ts_tablet_manager.cc:1397] T 314856bf269f4df7aa25f5a30cec0d5b P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.006s
I20250903 20:05:39.690450 27684 rpcz_store.cc:276] 0903 20:05:36.433113 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.433172 (+ 59us) service_pool.cc:225] Handling call
0903 20:05:39.633140 (+3199968us) ts_tablet_manager.cc:1145] Deleting tablet 30cc0d4ba8d444f1b98232f906ea0c7b
0903 20:05:39.633149 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.656085 (+ 22936us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.680564 (+ 24479us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.689847 (+ 9283us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3199602}
I20250903 20:05:39.692443 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 30cc0d4ba8d444f1b98232f906ea0c7b (table default.Table_423c3bfc0b8c46bc9f81d7a9013e1468 [id=57668609851c4124b73a3797bb6d2eb8]) successfully deleted
I20250903 20:05:39.692559 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1202) took 3037 ms. Trace:
I20250903 20:05:39.691779 29371 raft_consensus.cc:695] T 615f42e95bb24c8480f73d371fd63b9c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:39.657572 27695 tablet_bootstrap.cc:654] T 39c2ebe4ada5426590ec3769c8662644 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.692868 27684 tablet_replica.cc:331] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:39.693681 29371 consensus_queue.cc:237] T 615f42e95bb24c8480f73d371fd63b9c P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.694413 27684 raft_consensus.cc:2241] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:39.696861 27692 raft_consensus.cc:357] T 314856bf269f4df7aa25f5a30cec0d5b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.697008 27684 raft_consensus.cc:2270] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:39.697050 27692 raft_consensus.cc:383] T 314856bf269f4df7aa25f5a30cec0d5b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.697556 27692 raft_consensus.cc:738] T 314856bf269f4df7aa25f5a30cec0d5b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.697757 27692 consensus_queue.cc:260] T 314856bf269f4df7aa25f5a30cec0d5b P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.698051 27684 ts_tablet_manager.cc:1905] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:39.698300 27692 ts_tablet_manager.cc:1428] T 314856bf269f4df7aa25f5a30cec0d5b P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250903 20:05:39.700944 27692 tablet_bootstrap.cc:492] T 0ca6e2165a9d41299c1fe0bfa9475915 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:39.695858 27321 leader_election.cc:304] T 4739f10128d64e2c986257b1d9deb90f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:39.701741 28567 raft_consensus.cc:3058] T 4739f10128d64e2c986257b1d9deb90f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:39.704669 28567 raft_consensus.cc:2747] T 4739f10128d64e2c986257b1d9deb90f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:39.705204 27687 rpcz_store.cc:276] 0903 20:05:36.655598 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.670433 (+ 14835us) service_pool.cc:225] Handling call
0903 20:05:39.599641 (+2929208us) ts_tablet_manager.cc:1145] Deleting tablet 0f5b87c1c4a24f8abdbfd6194f3b2c4a
0903 20:05:39.599649 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.648133 (+ 48484us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.656520 (+ 8387us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.692524 (+ 36004us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3770,"tablet-delete.queue_time_us":2884852}
I20250903 20:05:39.706063 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1222) took 2681 ms. Trace:
I20250903 20:05:39.706290 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 0f5b87c1c4a24f8abdbfd6194f3b2c4a (table default.Table_6e9cf5ead5a24831a6a14d131dd8397e [id=524a27cbe1ad4c10aa2fd4f4074105c6]) successfully deleted
I20250903 20:05:39.706327 27687 rpcz_store.cc:276] 0903 20:05:37.024581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:37.024824 (+ 243us) service_pool.cc:225] Handling call
0903 20:05:39.705894 (+2681070us) ts_tablet_manager.cc:1145] Deleting tablet 32164b4e88b94e1fbca2e714e6bddfc4
0903 20:05:39.705909 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.706045 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2680879}
W20250903 20:05:39.706751 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 32164b4e88b94e1fbca2e714e6bddfc4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32164b4e88b94e1fbca2e714e6bddfc4 already in progress: creating tablet
I20250903 20:05:39.707199 29612 raft_consensus.cc:491] T 43fb27c857d543b7b78ea6108c406085 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.707430 29612 raft_consensus.cc:513] T 43fb27c857d543b7b78ea6108c406085 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.708966 29612 leader_election.cc:290] T 43fb27c857d543b7b78ea6108c406085 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:39.710541 27693 tablet_bootstrap.cc:492] T 5b46bb34ac214b1ba23fba1a33f746ac P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:39.710772 27692 tablet_bootstrap.cc:654] T 0ca6e2165a9d41299c1fe0bfa9475915 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.710762 27693 ts_tablet_manager.cc:1397] T 5b46bb34ac214b1ba23fba1a33f746ac P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.063s user 0.004s sys 0.001s
I20250903 20:05:39.712090 27693 raft_consensus.cc:357] T 5b46bb34ac214b1ba23fba1a33f746ac P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:39.714160 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "43fb27c857d543b7b78ea6108c406085" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:39.714828 27462 leader_election.cc:343] T 43fb27c857d543b7b78ea6108c406085 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.716660 27693 raft_consensus.cc:383] T 5b46bb34ac214b1ba23fba1a33f746ac P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.716832 27693 raft_consensus.cc:738] T 5b46bb34ac214b1ba23fba1a33f746ac P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.717159 27693 consensus_queue.cc:260] T 5b46bb34ac214b1ba23fba1a33f746ac P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:39.718204 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1061) took 5253 ms. Trace:
I20250903 20:05:39.718335 27685 rpcz_store.cc:276] 0903 20:05:34.464295 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.464465 (+ 170us) service_pool.cc:225] Handling call
0903 20:05:39.717976 (+5253511us) ts_tablet_manager.cc:1145] Deleting tablet a2358eac86914f3ba51bfc6c299838c3
0903 20:05:39.717991 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.718180 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5253293}
I20250903 20:05:39.718796 27685 tablet_replica.cc:331] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:39.719033 27685 raft_consensus.cc:2241] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
W20250903 20:05:39.719019 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet a2358eac86914f3ba51bfc6c299838c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2358eac86914f3ba51bfc6c299838c3 already in progress: creating tablet
I20250903 20:05:39.719352 27685 pending_rounds.cc:70] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a: Trying to abort 1 pending ops.
I20250903 20:05:39.719408 27685 pending_rounds.cc:77] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196385019479416832 op_type: NO_OP noop_request { }
I20250903 20:05:39.719491 27685 raft_consensus.cc:2887] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:39.719574 27685 raft_consensus.cc:2270] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:39.719858 27693 ts_tablet_manager.cc:1428] T 5b46bb34ac214b1ba23fba1a33f746ac P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250903 20:05:39.720158 27685 ts_tablet_manager.cc:1905] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:39.720217 28567 raft_consensus.cc:491] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.720403 28567 raft_consensus.cc:513] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:39.720991 28567 leader_election.cc:290] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:39.721818 27509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d4b8d4bf29e408cbdf9557c8b0d8a4b" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:39.722043 27509 raft_consensus.cc:2466] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:39.722597 27320 leader_election.cc:304] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:39.722882 28567 raft_consensus.cc:2802] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:39.722947 28567 raft_consensus.cc:491] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:39.723006 28567 raft_consensus.cc:3058] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:39.725589 28567 raft_consensus.cc:513] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:39.726606 28567 leader_election.cc:290] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:39.727283 27509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d4b8d4bf29e408cbdf9557c8b0d8a4b" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:39.727460 27509 raft_consensus.cc:3058] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:39.730180 27509 raft_consensus.cc:2466] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:39.730922 27320 leader_election.cc:304] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:39.731316 28567 raft_consensus.cc:2802] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:39.731493 28567 raft_consensus.cc:695] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:39.731685 27695 tablet_bootstrap.cc:492] T 39c2ebe4ada5426590ec3769c8662644 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:39.731894 27695 ts_tablet_manager.cc:1397] T 39c2ebe4ada5426590ec3769c8662644 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.112s user 0.001s sys 0.003s
I20250903 20:05:39.732201 28567 consensus_queue.cc:237] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:39.732844 27695 raft_consensus.cc:357] T 39c2ebe4ada5426590ec3769c8662644 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.734416 27692 tablet_bootstrap.cc:492] T 0ca6e2165a9d41299c1fe0bfa9475915 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:39.734548 27695 raft_consensus.cc:383] T 39c2ebe4ada5426590ec3769c8662644 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.734788 27695 raft_consensus.cc:738] T 39c2ebe4ada5426590ec3769c8662644 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.735386 27379 raft_consensus.cc:3058] T e2441aa3c2c748bba5360259fb663c81 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:39.735327 27695 consensus_queue.cc:260] T 39c2ebe4ada5426590ec3769c8662644 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.736748 27692 ts_tablet_manager.cc:1397] T 0ca6e2165a9d41299c1fe0bfa9475915 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.000s
I20250903 20:05:39.735692 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "43fb27c857d543b7b78ea6108c406085" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:39.737423 27462 leader_election.cc:343] T 43fb27c857d543b7b78ea6108c406085 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.737540 27462 leader_election.cc:304] T 43fb27c857d543b7b78ea6108c406085 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:39.738037 29371 raft_consensus.cc:2747] T 43fb27c857d543b7b78ea6108c406085 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:39.741328 27695 ts_tablet_manager.cc:1428] T 39c2ebe4ada5426590ec3769c8662644 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20250903 20:05:39.745498 27684 ts_tablet_manager.cc:1918] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:39.745628 27684 log.cc:1199] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/17f16e9c8f794e03a2f8b3b6b35f8e84
I20250903 20:05:39.745947 27684 ts_tablet_manager.cc:1939] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:39.742583 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d4b8d4bf29e408cbdf9557c8b0d8a4b" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
W20250903 20:05:39.746490 27321 leader_election.cc:343] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.748828 27695 tablet_bootstrap.cc:492] T f9375fff5f5945078c4412aa21c06f0d P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:39.748294 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d4b8d4bf29e408cbdf9557c8b0d8a4b" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:39.735918 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1221) took 2711 ms. Trace:
I20250903 20:05:39.751041 27687 rpcz_store.cc:276] 0903 20:05:37.024499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:37.025043 (+ 544us) service_pool.cc:225] Handling call
0903 20:05:39.710979 (+2685936us) ts_tablet_manager.cc:1145] Deleting tablet 04094bb7dd324b53a865980c53d4a17a
0903 20:05:39.722569 (+ 11590us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.735897 (+ 13328us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2685803}
I20250903 20:05:39.737545 27692 raft_consensus.cc:357] T 0ca6e2165a9d41299c1fe0bfa9475915 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.751586 27692 raft_consensus.cc:383] T 0ca6e2165a9d41299c1fe0bfa9475915 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250903 20:05:39.751577 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 04094bb7dd324b53a865980c53d4a17a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04094bb7dd324b53a865980c53d4a17a already in progress: creating tablet
I20250903 20:05:39.751649 27692 raft_consensus.cc:738] T 0ca6e2165a9d41299c1fe0bfa9475915 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.736395 27693 tablet_bootstrap.cc:492] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:39.751852 27692 consensus_queue.cc:260] T 0ca6e2165a9d41299c1fe0bfa9475915 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:39.753176 27321 leader_election.cc:343] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.756043 27692 ts_tablet_manager.cc:1428] T 0ca6e2165a9d41299c1fe0bfa9475915 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20250903 20:05:39.759207 27693 tablet_bootstrap.cc:654] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.759635 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1220) took 2735 ms. Trace:
I20250903 20:05:39.760612 27687 rpcz_store.cc:276] 0903 20:05:37.024393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:37.030510 (+ 6117us) service_pool.cc:225] Handling call
0903 20:05:39.759496 (+2728986us) ts_tablet_manager.cc:1145] Deleting tablet 4978cd831361454d86b23c1a52ab1c3b
0903 20:05:39.759507 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.759615 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2728739}
W20250903 20:05:39.761178 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 4978cd831361454d86b23c1a52ab1c3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4978cd831361454d86b23c1a52ab1c3b already in progress: creating tablet
I20250903 20:05:39.761809 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1226) took 2680 ms. Trace:
I20250903 20:05:39.761925 27687 rpcz_store.cc:276] 0903 20:05:37.081213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:37.081282 (+ 69us) service_pool.cc:225] Handling call
0903 20:05:39.761670 (+2680388us) ts_tablet_manager.cc:1145] Deleting tablet 5bff3d77088c4b69a90365950641e300
0903 20:05:39.761680 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.761792 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2680127}
W20250903 20:05:39.762524 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 5bff3d77088c4b69a90365950641e300 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bff3d77088c4b69a90365950641e300 already in progress: creating tablet
I20250903 20:05:39.762998 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1227) took 2681 ms. Trace:
I20250903 20:05:39.763123 27687 rpcz_store.cc:276] 0903 20:05:37.081749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:37.081796 (+ 47us) service_pool.cc:225] Handling call
0903 20:05:39.762870 (+2681074us) ts_tablet_manager.cc:1145] Deleting tablet cc71346bf84f4a708bf17b5aba174582
0903 20:05:39.762881 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.762979 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2680939}
W20250903 20:05:39.763652 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet cc71346bf84f4a708bf17b5aba174582 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc71346bf84f4a708bf17b5aba174582 already in progress: creating tablet
W20250903 20:05:39.763990 27065 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e7fd7cbf5a3048558720c7e258be308c consensus state has no leader
I20250903 20:05:39.764039 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1231) took 2677 ms. Trace:
I20250903 20:05:39.764290 27695 tablet_bootstrap.cc:654] T f9375fff5f5945078c4412aa21c06f0d P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.764541 27687 rpcz_store.cc:276] 0903 20:05:37.086860 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:37.086925 (+ 65us) service_pool.cc:225] Handling call
0903 20:05:39.763931 (+2677006us) ts_tablet_manager.cc:1145] Deleting tablet 4a4198b4841b4c2585e6c2f5c6a655d4
0903 20:05:39.763939 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.764021 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2676625}
W20250903 20:05:39.764992 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 4a4198b4841b4c2585e6c2f5c6a655d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a4198b4841b4c2585e6c2f5c6a655d4 already in progress: creating tablet
I20250903 20:05:39.765581 27687 tablet_replica.cc:331] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:39.765825 27687 raft_consensus.cc:2241] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:39.766435 27060 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_78e7a5b76ebc469396d8a3d670ab63ee [id=fb43441ca8f2415d99a628e81ef1642f] alter complete (version 2)
W20250903 20:05:39.768075 27060 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.26.65.65:44885 (request call id 198) took 994 ms (client timeout 1000 ms). Trace:
W20250903 20:05:39.768169 27060 rpcz_store.cc:269] 0903 20:05:38.773710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.829724 (+ 56014us) service_pool.cc:225] Handling call
0903 20:05:38.861496 (+ 31772us) spinlock_profiling.cc:243] Waited 30.6 ms on lock 0x616000040f54. stack: 00007f4b9d6c2dc6 00007f4b9d6c2b80 00007f4b9c6aa3ad 0000000000972d94 0000000000972d5d 00007f4bac49014d 00007f4bac490243 00007f4ba914f941 00007f4bb0cdaa7d 00007f4bb0e5e6f8 00007f4ba7c98c86 00007f4ba7c989b5 00007f4ba44730be 00007f4ba4471f9e 00007f4ba44781d5 00007f4ba4478aff
0903 20:05:39.768052 (+906556us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:39.474355 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:39.474452 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:39.474457 (+ 5us) write_op.cc:435] Acquired schema lock
0903 20:05:39.474459 (+ 2us) tablet.cc:660] Decoding operations
0903 20:05:39.474541 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:39.474550 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:39.474554 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0903 20:05:39.474605 (+ 51us) tablet.cc:699] Row locks acquired
0903 20:05:39.474608 (+ 3us) write_op.cc:260] PREPARE: finished
0903 20:05:39.474639 (+ 31us) write_op.cc:270] Start()
0903 20:05:39.474665 (+ 26us) write_op.cc:276] Timestamp: P: 1756929939474612 usec, L: 0
0903 20:05:39.474669 (+ 4us) op_driver.cc:348] REPLICATION: starting
0903 20:05:39.474852 (+ 183us) log.cc:844] Serialized 356 byte log entry
0903 20:05:39.759865 (+285013us) op_driver.cc:464] REPLICATION: finished
0903 20:05:39.761120 (+ 1255us) write_op.cc:301] APPLY: starting
0903 20:05:39.761129 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:39.761162 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:39.761181 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:39.761211 (+ 30us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:39.761231 (+ 20us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0903 20:05:39.761238 (+ 7us) write_op.cc:312] APPLY: finished
0903 20:05:39.761374 (+ 136us) log.cc:844] Serialized 23 byte log entry
0903 20:05:39.761439 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:39.761454 (+ 15us) write_op.cc:454] Released schema lock
0903 20:05:39.761461 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0903 20:05:39.761544 (+ 83us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:39.761549 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":15,"spinlock_wait_cycles":70351232,"child_traces":[["op",{"apply.queue_time_us":1230,"apply.run_cpu_time_us":411,"apply.run_wall_time_us":410,"num_ops":1,"prepare.queue_time_us":172618,"prepare.run_cpu_time_us":612,"prepare.run_wall_time_us":709,"raft.queue_time_us":35,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":285182,"spinlock_wait_cycles":77056}]]}
I20250903 20:05:39.768945 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_fd55f02a3038460fb305e900be86ef28" } new_table_name: "default.Table_f4791f9e8f0643ffa7b74d16fb1ad130" modify_external_catalogs: true
W20250903 20:05:39.770733 27063 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.26.65.66:53547 (request call id 161) took 1064 ms (client timeout 1000 ms). Trace:
W20250903 20:05:39.770824 27063 rpcz_store.cc:269] 0903 20:05:38.706424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.707361 (+ 937us) service_pool.cc:225] Handling call
0903 20:05:39.770708 (+1063347us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:38.891653 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:38.891826 (+ 173us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:38.891832 (+ 6us) write_op.cc:435] Acquired schema lock
0903 20:05:38.891835 (+ 3us) tablet.cc:660] Decoding operations
0903 20:05:38.891961 (+ 126us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:38.891974 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:38.891979 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0903 20:05:38.892058 (+ 79us) tablet.cc:699] Row locks acquired
0903 20:05:38.892060 (+ 2us) write_op.cc:260] PREPARE: finished
0903 20:05:38.893046 (+ 986us) write_op.cc:270] Start()
0903 20:05:38.893112 (+ 66us) write_op.cc:276] Timestamp: P: 1756929938892074 usec, L: 0
0903 20:05:38.893117 (+ 5us) op_driver.cc:348] REPLICATION: starting
0903 20:05:38.893429 (+ 312us) log.cc:844] Serialized 530 byte log entry
0903 20:05:38.919419 (+ 25990us) op_driver.cc:464] REPLICATION: finished
0903 20:05:38.919501 (+ 82us) write_op.cc:301] APPLY: starting
0903 20:05:38.919515 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:38.919550 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:38.919573 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:38.919620 (+ 47us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:38.919647 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0903 20:05:38.919655 (+ 8us) write_op.cc:312] APPLY: finished
0903 20:05:38.919729 (+ 74us) log.cc:844] Serialized 23 byte log entry
0903 20:05:38.919781 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:38.919797 (+ 16us) write_op.cc:454] Released schema lock
0903 20:05:38.919804 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0903 20:05:38.924378 (+ 4574us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:38.924387 (+ 9us) write_op.cc:454] Released schema lock
Related trace 'op':
0903 20:05:39.067062 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:39.067190 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:39.067197 (+ 7us) write_op.cc:435] Acquired schema lock
0903 20:05:39.067199 (+ 2us) tablet.cc:660] Decoding operations
0903 20:05:39.067307 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:39.067319 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:39.067324 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0903 20:05:39.067385 (+ 61us) tablet.cc:699] Row locks acquired
0903 20:05:39.067388 (+ 3us) write_op.cc:260] PREPARE: finished
0903 20:05:39.067423 (+ 35us) write_op.cc:270] Start()
0903 20:05:39.067454 (+ 31us) write_op.cc:276] Timestamp: P: 1756929939067387 usec, L: 0
0903 20:05:39.067458 (+ 4us) op_driver.cc:348] REPLICATION: starting
0903 20:05:39.072670 (+ 5212us) log.cc:844] Serialized 356 byte log entry
0903 20:05:39.083817 (+ 11147us) op_driver.cc:464] REPLICATION: finished
0903 20:05:39.084360 (+ 543us) write_op.cc:301] APPLY: starting
0903 20:05:39.084377 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:39.084420 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:39.084446 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:39.084500 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:39.084540 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0903 20:05:39.084549 (+ 9us) write_op.cc:312] APPLY: finished
0903 20:05:39.084622 (+ 73us) log.cc:844] Serialized 23 byte log entry
0903 20:05:39.084686 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:39.084701 (+ 15us) write_op.cc:454] Released schema lock
0903 20:05:39.084707 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0903 20:05:39.084771 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:39.084775 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":12,"child_traces":[["op",{"apply.queue_time_us":39,"apply.run_cpu_time_us":358,"apply.run_wall_time_us":357,"num_ops":1,"prepare.queue_time_us":90833,"prepare.run_cpu_time_us":1014,"prepare.run_wall_time_us":32705,"raft.queue_time_us":114,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":112,"replication_time_us":26279}],["op",{"apply.queue_time_us":462,"apply.run_cpu_time_us":411,"apply.run_wall_time_us":411,"num_ops":1,"prepare.queue_time_us":145536,"prepare.run_cpu_time_us":784,"prepare.run_wall_time_us":11037,"raft.queue_time_us":40,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":7,"replication_time_us":16328}]]}
I20250903 20:05:39.771844 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1cbf3c0f4048429680ea8fbf4b60a761" } modify_external_catalogs: true
I20250903 20:05:39.775267 27692 tablet_bootstrap.cc:492] T bc98762790f441038972366b1cc2c55b P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:39.775915 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1181) took 3344 ms. Trace:
I20250903 20:05:39.776104 27684 rpcz_store.cc:276] 0903 20:05:36.431062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.434433 (+ 3371us) service_pool.cc:225] Handling call
0903 20:05:39.692809 (+3258376us) ts_tablet_manager.cc:1145] Deleting tablet 17f16e9c8f794e03a2f8b3b6b35f8e84
0903 20:05:39.692822 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.742155 (+ 49333us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.745397 (+ 3242us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.775890 (+ 30493us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3258066}
I20250903 20:05:39.776680 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 17f16e9c8f794e03a2f8b3b6b35f8e84 (table default.Table_c9ba436d242a498582b9009fa5d299e4 [id=d7a41e621fde497d8fcfb842c1e6f9e7]) successfully deleted
I20250903 20:05:39.777077 27692 tablet_bootstrap.cc:654] T bc98762790f441038972366b1cc2c55b P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.766203 27687 raft_consensus.cc:2270] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:39.779481 27693 tablet_bootstrap.cc:492] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:39.779524 27064 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 1fce05bd4e6c425e952001ffdc25c78a
I20250903 20:05:39.779688 27693 ts_tablet_manager.cc:1397] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.044s user 0.001s sys 0.004s
I20250903 20:05:39.780064 27064 catalog_manager.cc:6652] ReplaceTablet: tablet 1fce05bd4e6c425e952001ffdc25c78a deleted and replaced by tablet 491f42ede56a40608aec2c7db90ccb7b
I20250903 20:05:39.781404 27693 raft_consensus.cc:357] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.781589 27693 raft_consensus.cc:383] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.781637 27693 raft_consensus.cc:738] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.781831 27693 consensus_queue.cc:260] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.783416 27693 ts_tablet_manager.cc:1428] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250903 20:05:39.783439 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 72fcb7b0cba34313a7dc2b8c29e120fc
I20250903 20:05:39.783766 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 23f315a1fa3d4c749994de7d0a0df775
I20250903 20:05:39.784178 27693 tablet_bootstrap.cc:492] T 1bf570855b204b96acd070587c434463 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:39.784435 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 6853839e73e343a0a54aa400a23db35c
I20250903 20:05:39.785288 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_8172cc8c300c4e1ebcef3dc1fadf1295"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:39.786361 27687 ts_tablet_manager.cc:1905] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:39.786378 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1cbf3c0f4048429680ea8fbf4b60a761" } modify_external_catalogs: true
I20250903 20:05:39.786890 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 23f315a1fa3d4c749994de7d0a0df775 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:39 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:39.786940 27347 tablet_service.cc:1515] Processing DeleteTablet for tablet 6853839e73e343a0a54aa400a23db35c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:39.786975 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet 1fce05bd4e6c425e952001ffdc25c78a with delete_type TABLET_DATA_DELETED (replaced by tablet 491f42ede56a40608aec2c7db90ccb7b) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:39.788031 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 6853839e73e343a0a54aa400a23db35c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:39.787984 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 23f315a1fa3d4c749994de7d0a0df775 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:39 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:39.777086 27684 tablet_replica.cc:331] T 13496efe34944b7280e7a684f180e6e0 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:39.792954 27684 raft_consensus.cc:2241] T 13496efe34944b7280e7a684f180e6e0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:39.793089 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_59e7e14a9ba945bf9969e024a530cb07" } modify_external_catalogs: true
I20250903 20:05:39.793258 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_59e7e14a9ba945bf9969e024a530cb07" } modify_external_catalogs: true
I20250903 20:05:39.793263 27684 raft_consensus.cc:2270] T 13496efe34944b7280e7a684f180e6e0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:39.793955 27684 ts_tablet_manager.cc:1905] T 13496efe34944b7280e7a684f180e6e0 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:39.794531 27061 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:45936 (request call id 1470) took 1168 ms. Trace:
I20250903 20:05:39.794612 27061 rpcz_store.cc:276] 0903 20:05:38.625741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.903081 (+277340us) service_pool.cc:225] Handling call
0903 20:05:38.917847 (+ 14766us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x616000040f54. stack: 00007f4b9d6c2dc6 00007f4b9d6c2b80 00007f4b9c6aa3ad 0000000000972d94 0000000000972d5d 00007f4bac49014d 00007f4bac490243 00007f4ba914f941 00007f4bb0cdaa7d 00007f4bb0e64ea6 00007f4ba7c9d466 00007f4ba7c9d195 00007f4ba44730be 00007f4ba4471f9e 00007f4ba44781d5 00007f4ba4478aff
0903 20:05:38.918156 (+ 309us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0903 20:05:38.918304 (+ 148us) catalog_manager.cc:2790] Modifying in-memory table state
0903 20:05:38.918363 (+ 59us) catalog_manager.cc:2799] Locking tablets
0903 20:05:39.323362 (+404999us) catalog_manager.cc:2813] Removing table and tablets from system table
0903 20:05:39.769330 (+445968us) catalog_manager.cc:2831] Removing table from by-name map
0903 20:05:39.784369 (+ 15039us) catalog_manager.cc:2847] Committing in-memory state
0903 20:05:39.784411 (+ 42us) catalog_manager.cc:2851] Aborting table tasks
0903 20:05:39.794511 (+ 10100us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:39.476520 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:39.476611 (+ 91us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:39.476616 (+ 5us) write_op.cc:435] Acquired schema lock
0903 20:05:39.476618 (+ 2us) tablet.cc:660] Decoding operations
0903 20:05:39.476707 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:39.476715 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:39.476720 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0903 20:05:39.476795 (+ 75us) tablet.cc:699] Row locks acquired
0903 20:05:39.476798 (+ 3us) write_op.cc:260] PREPARE: finished
0903 20:05:39.476822 (+ 24us) write_op.cc:270] Start()
0903 20:05:39.476847 (+ 25us) write_op.cc:276] Timestamp: P: 1756929939476798 usec, L: 0
0903 20:05:39.476850 (+ 3us) op_driver.cc:348] REPLICATION: starting
0903 20:05:39.476997 (+ 147us) log.cc:844] Serialized 759 byte log entry
0903 20:05:39.759974 (+282977us) op_driver.cc:464] REPLICATION: finished
0903 20:05:39.762465 (+ 2491us) write_op.cc:301] APPLY: starting
0903 20:05:39.762474 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:39.762501 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:39.762522 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:39.762564 (+ 42us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:39.762586 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0903 20:05:39.762593 (+ 7us) write_op.cc:312] APPLY: finished
0903 20:05:39.762636 (+ 43us) log.cc:844] Serialized 29 byte log entry
0903 20:05:39.762666 (+ 30us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:39.762677 (+ 11us) write_op.cc:454] Released schema lock
0903 20:05:39.762681 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0903 20:05:39.762731 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:39.762735 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":33340672,"child_traces":[["op",{"apply.queue_time_us":2464,"apply.run_cpu_time_us":260,"apply.run_wall_time_us":260,"num_ops":2,"prepare.queue_time_us":152442,"prepare.run_cpu_time_us":587,"prepare.run_wall_time_us":638,"raft.queue_time_us":33,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":283107}]]}
I20250903 20:05:39.795401 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_6f310ec4054e4ac7a3ae073de1b63c19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:39.795921 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f310ec4054e4ac7a3ae073de1b63c19 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:39.798288 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8172cc8c300c4e1ebcef3dc1fadf1295 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:39.798659 27692 tablet_bootstrap.cc:492] T bc98762790f441038972366b1cc2c55b P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:39.798827 27692 ts_tablet_manager.cc:1397] T bc98762790f441038972366b1cc2c55b P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20250903 20:05:39.791031 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_33be5adb5dee4e248075bea00bd101d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:39.799538 27692 raft_consensus.cc:357] T bc98762790f441038972366b1cc2c55b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.799685 27692 raft_consensus.cc:383] T bc98762790f441038972366b1cc2c55b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.799731 27692 raft_consensus.cc:738] T bc98762790f441038972366b1cc2c55b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
W20250903 20:05:39.799892 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33be5adb5dee4e248075bea00bd101d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:39.800062 27692 consensus_queue.cc:260] T bc98762790f441038972366b1cc2c55b P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.802044 27692 ts_tablet_manager.cc:1428] T bc98762790f441038972366b1cc2c55b P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250903 20:05:39.802598 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet 72fcb7b0cba34313a7dc2b8c29e120fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:39 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:39.802676 27692 tablet_bootstrap.cc:492] T f23f90aac8bc49889b1b882df186ef50 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:39.806365 27065 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 6d4b8d4bf29e408cbdf9557c8b0d8a4b (table default.Table_ccdb4130549b4be5888f17097ae35106 [id=cbcfc7067aca4796b39e3eb195f38903]). Expected version 2 got 0
I20250903 20:05:39.806682 27065 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 249362a0b8304719863bcb82c47273b3 (table default.Table_1dd1b4a75c914e6a88058266d2a7e98a [id=565dc96c38de4efeb4328c01e030eb8c]). Expected version 1 got 0
I20250903 20:05:39.806949 27065 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet f4d3e1cfdfab4be1a9143b78bf3bc46d (table default.Table_5e672ed0c8c142029a79b72f799f3f59 [id=386b8d4e7c4042d1947c19ac464c75a0]). Expected version 2 got 0
I20250903 20:05:39.808885 27687 ts_tablet_manager.cc:1918] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:39.811117 27692 tablet_bootstrap.cc:654] T f23f90aac8bc49889b1b882df186ef50 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.811825 27695 tablet_bootstrap.cc:492] T f9375fff5f5945078c4412aa21c06f0d P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:39.812150 27695 ts_tablet_manager.cc:1397] T f9375fff5f5945078c4412aa21c06f0d P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.063s user 0.009s sys 0.000s
I20250903 20:05:39.812868 27695 raft_consensus.cc:357] T f9375fff5f5945078c4412aa21c06f0d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.813113 27695 raft_consensus.cc:383] T f9375fff5f5945078c4412aa21c06f0d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.813206 27695 raft_consensus.cc:738] T f9375fff5f5945078c4412aa21c06f0d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.813357 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 548b50819f694fa3ad508da9f442a99c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:39.813500 27695 consensus_queue.cc:260] T f9375fff5f5945078c4412aa21c06f0d P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.813752 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet aa9f5e0964b348b7af184f60cd145666 with delete_type TABLET_DATA_DELETED (replaced by tablet 26decef5841a4a3b85f2a5f578ae95dc) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:39.814003 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 1fce05bd4e6c425e952001ffdc25c78a with delete_type TABLET_DATA_DELETED (replaced by tablet 491f42ede56a40608aec2c7db90ccb7b) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:39.814369 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 72fcb7b0cba34313a7dc2b8c29e120fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:39 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:39.814463 27695 ts_tablet_manager.cc:1428] T f9375fff5f5945078c4412aa21c06f0d P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:39.814842 27695 tablet_bootstrap.cc:492] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:39.815016 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 23f315a1fa3d4c749994de7d0a0df775 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:39 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:39.815630 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 6853839e73e343a0a54aa400a23db35c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:38 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:39.817103 27695 tablet_bootstrap.cc:654] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.821391 27695 tablet_bootstrap.cc:492] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:39.821393 27684 ts_tablet_manager.cc:1918] T 13496efe34944b7280e7a684f180e6e0 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:39.821503 27684 log.cc:1199] T 13496efe34944b7280e7a684f180e6e0 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/13496efe34944b7280e7a684f180e6e0
I20250903 20:05:39.824167 27695 ts_tablet_manager.cc:1397] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20250903 20:05:39.825560 27695 raft_consensus.cc:357] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.826531 27695 raft_consensus.cc:383] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.826712 27695 raft_consensus.cc:738] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.828114 27695 consensus_queue.cc:260] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.809552 27693 tablet_bootstrap.cc:654] T 1bf570855b204b96acd070587c434463 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.812752 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet f326ec776ba446be8d5dda5c1dc5b6ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:39.812880 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e34d6d5d68e4395aed78821ea077c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:39.812992 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet a3103e1595ec4c38ba65dca26fe1167c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:39.835127 27695 ts_tablet_manager.cc:1428] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250903 20:05:39.835974 27695 tablet_bootstrap.cc:492] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:39.838683 27695 tablet_bootstrap.cc:654] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.838707 27687 log.cc:1199] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/ee40566e497740789575275d0cdca0db
I20250903 20:05:39.839219 27687 ts_tablet_manager.cc:1939] T ee40566e497740789575275d0cdca0db P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:39.841567 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1267) took 2206 ms. Trace:
I20250903 20:05:39.843683 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 6ff8d4902e6844d191038a6f4e4af4b1
I20250903 20:05:39.843904 27687 rpcz_store.cc:276] 0903 20:05:37.635058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:37.641885 (+ 6827us) service_pool.cc:225] Handling call
0903 20:05:39.765419 (+2123534us) ts_tablet_manager.cc:1145] Deleting tablet ee40566e497740789575275d0cdca0db
0903 20:05:39.765432 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.805725 (+ 40293us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.808769 (+ 3044us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.841536 (+ 32767us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2123169}
I20250903 20:05:39.824105 27692 tablet_bootstrap.cc:492] T f23f90aac8bc49889b1b882df186ef50 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:39.844506 27692 ts_tablet_manager.cc:1397] T f23f90aac8bc49889b1b882df186ef50 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20250903 20:05:39.844532 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e29b515650234a5685ac744105156384"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:39.845134 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e29b515650234a5685ac744105156384 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:39.845515 27695 tablet_bootstrap.cc:492] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:39.845719 27692 raft_consensus.cc:357] T f23f90aac8bc49889b1b882df186ef50 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.845944 27692 raft_consensus.cc:383] T f23f90aac8bc49889b1b882df186ef50 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.846001 27692 raft_consensus.cc:738] T f23f90aac8bc49889b1b882df186ef50 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
W20250903 20:05:39.846063 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7c200f2becd84e3482e528191cbc2868 consensus state has no leader
I20250903 20:05:39.846293 27692 consensus_queue.cc:260] T f23f90aac8bc49889b1b882df186ef50 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.846488 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_93c4817ac1c64d589e71a4590d32adb2" } new_table_name: "default.Table_cc82debd3c624d588c7959ed6f045065" modify_external_catalogs: true
I20250903 20:05:39.846681 27695 ts_tablet_manager.cc:1397] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20250903 20:05:39.846798 27692 ts_tablet_manager.cc:1428] T f23f90aac8bc49889b1b882df186ef50 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:39.847328 27692 tablet_bootstrap.cc:492] T 8e12f4c420874ab382d433982bddbf05 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:39.849187 27692 tablet_bootstrap.cc:654] T 8e12f4c420874ab382d433982bddbf05 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.849321 27695 raft_consensus.cc:357] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.849529 27695 raft_consensus.cc:383] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.849591 27695 raft_consensus.cc:738] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.849882 27695 consensus_queue.cc:260] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.850467 27684 ts_tablet_manager.cc:1939] T 13496efe34944b7280e7a684f180e6e0 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:39.851117 27687 tablet_replica.cc:331] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:39.851361 27687 raft_consensus.cc:2241] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:39.851687 27687 raft_consensus.cc:2270] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:39.851961 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_0f25b427fe144098afa055eaec8a1cfe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:39.852662 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f25b427fe144098afa055eaec8a1cfe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:39.852710 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1183) took 3295 ms. Trace:
I20250903 20:05:39.852823 27684 rpcz_store.cc:276] 0903 20:05:36.557402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.557483 (+ 81us) service_pool.cc:225] Handling call
0903 20:05:39.777032 (+3219549us) ts_tablet_manager.cc:1145] Deleting tablet 13496efe34944b7280e7a684f180e6e0
0903 20:05:39.777044 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.817650 (+ 40606us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.821279 (+ 3629us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.852676 (+ 31397us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2942,"tablet-delete.queue_time_us":3219305}
I20250903 20:05:39.853217 27684 tablet_replica.cc:331] T e2441aa3c2c748bba5360259fb663c81 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:39.853428 27684 raft_consensus.cc:2241] T e2441aa3c2c748bba5360259fb663c81 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:39.853598 27684 raft_consensus.cc:2270] T e2441aa3c2c748bba5360259fb663c81 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:39.854300 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 13496efe34944b7280e7a684f180e6e0 (table default.Table_1d4af1f6213b4b4db3bfaee9e1ede2d0 [id=41f9673d6e604f45a8def10948ce21ac]) successfully deleted
I20250903 20:05:39.855270 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet ee40566e497740789575275d0cdca0db (table default.Table_ee1ad081d0c846ca9fdc08acfb8318e9 [id=1794b6e2ac9443718c798588325d4a2d]) successfully deleted
I20250903 20:05:39.855427 27687 ts_tablet_manager.cc:1905] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:39.856427 29411 consensus_queue.cc:1035] T 2385d3006f90486890ebe43e16a53d7a P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:39.856495 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_556002da3b4142b9b0c87900520db2bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:39.857013 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_556002da3b4142b9b0c87900520db2bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:39.857523 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ff8d4902e6844d191038a6f4e4af4b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:39 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:39.858580 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet ef7dd1ec6fcc47ec81facd6ae0207b97
I20250903 20:05:39.859223 29612 raft_consensus.cc:491] T 36e1061915574407b6e7695be8b1e55a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.859310 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ec4b7aba6eaa402586095c0237bdb4d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:39.859408 29612 raft_consensus.cc:513] T 36e1061915574407b6e7695be8b1e55a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:39.859860 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec4b7aba6eaa402586095c0237bdb4d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:39.860249 29612 leader_election.cc:290] T 36e1061915574407b6e7695be8b1e55a P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:39.861614 27057 master_service.cc:948] ReplaceTablet: received request to replace tablet fa2014bccb5e4f0a91cbc9845df4634d from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:39.862352 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e1061915574407b6e7695be8b1e55a" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:39.863080 27462 leader_election.cc:343] T 36e1061915574407b6e7695be8b1e55a P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.863866 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet ef7dd1ec6fcc47ec81facd6ae0207b97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:39 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:39.840454 27693 tablet_bootstrap.cc:492] T 1bf570855b204b96acd070587c434463 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:39.855854 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ff8d4902e6844d191038a6f4e4af4b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:39 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:39.861508 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36e1061915574407b6e7695be8b1e55a" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:39.865422 27462 leader_election.cc:343] T 36e1061915574407b6e7695be8b1e55a P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.865569 27462 leader_election.cc:304] T 36e1061915574407b6e7695be8b1e55a P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:39.866278 29612 raft_consensus.cc:2747] T 36e1061915574407b6e7695be8b1e55a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:39.863694 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet ef7dd1ec6fcc47ec81facd6ae0207b97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:39 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:39.866899 27693 ts_tablet_manager.cc:1397] T 1bf570855b204b96acd070587c434463 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.083s user 0.003s sys 0.003s
I20250903 20:05:39.868292 27061 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5b8fcb7885294903a6849c86ad91ccca" } new_table_name: "default.Table_7e439005e9d4471e94e67a047e287b3c" modify_external_catalogs: true
I20250903 20:05:39.868345 27693 raft_consensus.cc:357] T 1bf570855b204b96acd070587c434463 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.868577 27693 raft_consensus.cc:383] T 1bf570855b204b96acd070587c434463 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.868731 27693 raft_consensus.cc:738] T 1bf570855b204b96acd070587c434463 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.869011 27693 consensus_queue.cc:260] T 1bf570855b204b96acd070587c434463 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.870117 27693 ts_tablet_manager.cc:1428] T 1bf570855b204b96acd070587c434463 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250903 20:05:39.870517 27693 tablet_bootstrap.cc:492] T 6800e0b6269045f29682704fd847d5d1 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:39.871280 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet ef7dd1ec6fcc47ec81facd6ae0207b97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:39 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:39.825513 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet 1fce05bd4e6c425e952001ffdc25c78a with delete_type TABLET_DATA_DELETED (replaced by tablet 491f42ede56a40608aec2c7db90ccb7b) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:39.871999 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ff8d4902e6844d191038a6f4e4af4b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:39 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:39.872905 29455 consensus_queue.cc:1035] T 2385d3006f90486890ebe43e16a53d7a P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:39.829177 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 72fcb7b0cba34313a7dc2b8c29e120fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:39 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:39.879767 27693 tablet_bootstrap.cc:654] T 6800e0b6269045f29682704fd847d5d1 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:39.880193 27183 consensus_peers.cc:489] T 3f303233e6c347289650c6cf83f8127f P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f303233e6c347289650c6cf83f8127f. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:39.881096 27692 tablet_bootstrap.cc:492] T 8e12f4c420874ab382d433982bddbf05 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:39.882581 27692 ts_tablet_manager.cc:1397] T 8e12f4c420874ab382d433982bddbf05 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.000s
I20250903 20:05:39.884914 27695 ts_tablet_manager.cc:1428] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.038s user 0.002s sys 0.000s
I20250903 20:05:39.885278 27695 tablet_bootstrap.cc:492] T ed4902c002ef4198802375ced39ef2ad P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:39.886411 27692 raft_consensus.cc:357] T 8e12f4c420874ab382d433982bddbf05 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.887607 27695 tablet_bootstrap.cc:654] T ed4902c002ef4198802375ced39ef2ad P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.892831 27692 raft_consensus.cc:383] T 8e12f4c420874ab382d433982bddbf05 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.892964 27692 raft_consensus.cc:738] T 8e12f4c420874ab382d433982bddbf05 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.893283 27692 consensus_queue.cc:260] T 8e12f4c420874ab382d433982bddbf05 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.854192 27684 ts_tablet_manager.cc:1905] T e2441aa3c2c748bba5360259fb663c81 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:39.890381 29371 raft_consensus.cc:491] T 69f8eef52bee431a86f0eef45fd11f9f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.893687 29371 raft_consensus.cc:513] T 69f8eef52bee431a86f0eef45fd11f9f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.893697 27065 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_3edadd8954444273b91ea4d39cf3a834 [id=45cd523acd3c45b994d1b9cc6eb569ea] alter complete (version 4)
I20250903 20:05:39.894376 29371 leader_election.cc:290] T 69f8eef52bee431a86f0eef45fd11f9f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:39.895586 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f8eef52bee431a86f0eef45fd11f9f" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:39.896209 27462 leader_election.cc:343] T 69f8eef52bee431a86f0eef45fd11f9f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.895370 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f8eef52bee431a86f0eef45fd11f9f" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:39.898300 27693 tablet_bootstrap.cc:492] T 6800e0b6269045f29682704fd847d5d1 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:39.899765 27695 tablet_bootstrap.cc:492] T ed4902c002ef4198802375ced39ef2ad P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:39.900439 27695 ts_tablet_manager.cc:1397] T ed4902c002ef4198802375ced39ef2ad P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20250903 20:05:39.904515 27695 raft_consensus.cc:357] T ed4902c002ef4198802375ced39ef2ad P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:39.905098 27462 leader_election.cc:343] T 69f8eef52bee431a86f0eef45fd11f9f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.905230 27462 leader_election.cc:304] T 69f8eef52bee431a86f0eef45fd11f9f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:39.906476 27695 raft_consensus.cc:383] T ed4902c002ef4198802375ced39ef2ad P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.906507 29371 raft_consensus.cc:2747] T 69f8eef52bee431a86f0eef45fd11f9f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:39.906755 29612 raft_consensus.cc:491] T f23f90aac8bc49889b1b882df186ef50 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.906875 27695 raft_consensus.cc:738] T ed4902c002ef4198802375ced39ef2ad P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.906863 29612 raft_consensus.cc:513] T f23f90aac8bc49889b1b882df186ef50 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.907254 27695 consensus_queue.cc:260] T ed4902c002ef4198802375ced39ef2ad P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.907435 29612 leader_election.cc:290] T f23f90aac8bc49889b1b882df186ef50 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:39.907531 29452 raft_consensus.cc:491] T f9375fff5f5945078c4412aa21c06f0d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.907673 29452 raft_consensus.cc:513] T f9375fff5f5945078c4412aa21c06f0d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.907804 27695 ts_tablet_manager.cc:1428] T ed4902c002ef4198802375ced39ef2ad P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250903 20:05:39.908136 27695 tablet_bootstrap.cc:492] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:39.908227 29452 leader_election.cc:290] T f9375fff5f5945078c4412aa21c06f0d P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:39.909062 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f23f90aac8bc49889b1b882df186ef50" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:39.909940 27695 tablet_bootstrap.cc:654] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.909883 27509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9375fff5f5945078c4412aa21c06f0d" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:39.910084 27509 raft_consensus.cc:2466] T f9375fff5f5945078c4412aa21c06f0d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
W20250903 20:05:39.910706 27462 leader_election.cc:343] T f23f90aac8bc49889b1b882df186ef50 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.910827 27183 leader_election.cc:304] T f9375fff5f5945078c4412aa21c06f0d P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:39.911202 29461 raft_consensus.cc:2802] T f9375fff5f5945078c4412aa21c06f0d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:39.911341 29461 raft_consensus.cc:491] T f9375fff5f5945078c4412aa21c06f0d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:39.911455 29461 raft_consensus.cc:3058] T f9375fff5f5945078c4412aa21c06f0d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:39.913457 27692 ts_tablet_manager.cc:1428] T 8e12f4c420874ab382d433982bddbf05 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20250903 20:05:39.903862 28567 consensus_queue.cc:1035] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:39.909065 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9375fff5f5945078c4412aa21c06f0d" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:39.914609 27184 leader_election.cc:343] T f9375fff5f5945078c4412aa21c06f0d P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.911486 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f23f90aac8bc49889b1b882df186ef50" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:39.915460 27462 leader_election.cc:343] T f23f90aac8bc49889b1b882df186ef50 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.915578 27462 leader_election.cc:304] T f23f90aac8bc49889b1b882df186ef50 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:39.916524 29371 raft_consensus.cc:2747] T f23f90aac8bc49889b1b882df186ef50 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:39.916558 29461 raft_consensus.cc:513] T f9375fff5f5945078c4412aa21c06f0d P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.917321 29487 consensus_queue.cc:1035] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:39.918071 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9375fff5f5945078c4412aa21c06f0d" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:39.918350 27693 ts_tablet_manager.cc:1397] T 6800e0b6269045f29682704fd847d5d1 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.002s
W20250903 20:05:39.918555 27184 leader_election.cc:343] T f9375fff5f5945078c4412aa21c06f0d P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.919651 27693 raft_consensus.cc:357] T 6800e0b6269045f29682704fd847d5d1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:39.920480 27692 tablet_bootstrap.cc:492] T 847d0a0a3d2b45938cec9058c77f6231 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:39.924973 27510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9375fff5f5945078c4412aa21c06f0d" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:39.925223 27510 raft_consensus.cc:3058] T f9375fff5f5945078c4412aa21c06f0d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:39.928299 27692 tablet_bootstrap.cc:654] T 847d0a0a3d2b45938cec9058c77f6231 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.929318 27693 raft_consensus.cc:383] T 6800e0b6269045f29682704fd847d5d1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.929307 27510 raft_consensus.cc:2466] T f9375fff5f5945078c4412aa21c06f0d P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:39.929590 27693 raft_consensus.cc:738] T 6800e0b6269045f29682704fd847d5d1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.930137 27183 leader_election.cc:304] T f9375fff5f5945078c4412aa21c06f0d P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:39.930572 29452 raft_consensus.cc:2802] T f9375fff5f5945078c4412aa21c06f0d P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:39.930519 27693 consensus_queue.cc:260] T 6800e0b6269045f29682704fd847d5d1 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:39.930687 29452 raft_consensus.cc:695] T f9375fff5f5945078c4412aa21c06f0d P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:39.930958 29452 consensus_queue.cc:237] T f9375fff5f5945078c4412aa21c06f0d P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.931095 27693 ts_tablet_manager.cc:1428] T 6800e0b6269045f29682704fd847d5d1 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250903 20:05:39.933426 27695 tablet_bootstrap.cc:492] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:39.933619 27695 ts_tablet_manager.cc:1397] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.004s
I20250903 20:05:39.933939 27693 tablet_bootstrap.cc:492] T 60c32370a5274347afafc6a7e1e94270 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:39.934587 27695 raft_consensus.cc:357] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:39.934764 27695 raft_consensus.cc:383] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.934809 27695 raft_consensus.cc:738] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.935002 27695 consensus_queue.cc:260] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:39.935590 27695 ts_tablet_manager.cc:1428] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:39.935876 27695 tablet_bootstrap.cc:492] T ea3f82f4e4244219ab0919f3e341aead P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:39.936735 27249 raft_consensus.cc:3058] T 39c2ebe4ada5426590ec3769c8662644 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:39.938297 28567 raft_consensus.cc:491] T 249362a0b8304719863bcb82c47273b3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.938525 28567 raft_consensus.cc:513] T 249362a0b8304719863bcb82c47273b3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.939229 28567 leader_election.cc:290] T 249362a0b8304719863bcb82c47273b3 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:39.940316 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "249362a0b8304719863bcb82c47273b3" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:39.941344 27321 leader_election.cc:343] T 249362a0b8304719863bcb82c47273b3 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.941823 27510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "249362a0b8304719863bcb82c47273b3" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:39.942117 27510 raft_consensus.cc:2466] T 249362a0b8304719863bcb82c47273b3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:39.942657 27320 leader_election.cc:304] T 249362a0b8304719863bcb82c47273b3 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:39.942895 27693 tablet_bootstrap.cc:654] T 60c32370a5274347afafc6a7e1e94270 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.943308 28567 raft_consensus.cc:2802] T 249362a0b8304719863bcb82c47273b3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:39.943393 28567 raft_consensus.cc:491] T 249362a0b8304719863bcb82c47273b3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:39.943477 28567 raft_consensus.cc:3058] T 249362a0b8304719863bcb82c47273b3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:39.946398 28567 raft_consensus.cc:513] T 249362a0b8304719863bcb82c47273b3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.947312 27684 ts_tablet_manager.cc:1918] T e2441aa3c2c748bba5360259fb663c81 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:39.947887 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "249362a0b8304719863bcb82c47273b3" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:39.948416 27516 raft_consensus.cc:3058] T 249362a0b8304719863bcb82c47273b3 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:39.948740 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "249362a0b8304719863bcb82c47273b3" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
W20250903 20:05:39.949244 27321 leader_election.cc:343] T 249362a0b8304719863bcb82c47273b3 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.949406 28567 leader_election.cc:290] T 249362a0b8304719863bcb82c47273b3 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:39.951401 27695 tablet_bootstrap.cc:654] T ea3f82f4e4244219ab0919f3e341aead P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.951418 27516 raft_consensus.cc:2466] T 249362a0b8304719863bcb82c47273b3 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:39.948272 27692 tablet_bootstrap.cc:492] T 847d0a0a3d2b45938cec9058c77f6231 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:39.951957 27692 ts_tablet_manager.cc:1397] T 847d0a0a3d2b45938cec9058c77f6231 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20250903 20:05:39.951972 27320 leader_election.cc:304] T 249362a0b8304719863bcb82c47273b3 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:39.953092 27692 raft_consensus.cc:357] T 847d0a0a3d2b45938cec9058c77f6231 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.953483 27692 raft_consensus.cc:383] T 847d0a0a3d2b45938cec9058c77f6231 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.953612 27692 raft_consensus.cc:738] T 847d0a0a3d2b45938cec9058c77f6231 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.953948 27692 consensus_queue.cc:260] T 847d0a0a3d2b45938cec9058c77f6231 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.954181 28567 raft_consensus.cc:2802] T 249362a0b8304719863bcb82c47273b3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:39.955040 27692 ts_tablet_manager.cc:1428] T 847d0a0a3d2b45938cec9058c77f6231 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:39.955376 27692 tablet_bootstrap.cc:492] T b84f487773c24735a61d0fd0898b2bcb P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:39.960005 27684 log.cc:1199] T e2441aa3c2c748bba5360259fb663c81 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/e2441aa3c2c748bba5360259fb663c81
I20250903 20:05:39.960860 27684 ts_tablet_manager.cc:1939] T e2441aa3c2c748bba5360259fb663c81 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:39.962076 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1184) took 3379 ms. Trace:
I20250903 20:05:39.962329 27684 rpcz_store.cc:276] 0903 20:05:36.582192 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.582282 (+ 90us) service_pool.cc:225] Handling call
0903 20:05:39.853158 (+3270876us) ts_tablet_manager.cc:1145] Deleting tablet e2441aa3c2c748bba5360259fb663c81
0903 20:05:39.853171 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.922865 (+ 69694us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.947183 (+ 24318us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:39.962049 (+ 14866us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3270646}
I20250903 20:05:39.963099 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet e2441aa3c2c748bba5360259fb663c81 (table default.Table_64dabe496cb94cfb9061d0d7eefcbd3a [id=741df86b1206422b9467c06cab14ddce]) successfully deleted
I20250903 20:05:39.963649 27684 tablet_replica.cc:331] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:39.963992 27684 raft_consensus.cc:2241] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:39.967717 29455 raft_consensus.cc:491] T ed4902c002ef4198802375ced39ef2ad P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.967883 29455 raft_consensus.cc:513] T ed4902c002ef4198802375ced39ef2ad P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.968575 29455 leader_election.cc:290] T ed4902c002ef4198802375ced39ef2ad P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:39.969818 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed4902c002ef4198802375ced39ef2ad" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:39.970525 27184 leader_election.cc:343] T ed4902c002ef4198802375ced39ef2ad P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.973400 27693 tablet_bootstrap.cc:492] T 60c32370a5274347afafc6a7e1e94270 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:39.973629 27693 ts_tablet_manager.cc:1397] T 60c32370a5274347afafc6a7e1e94270 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
W20250903 20:05:39.973711 27320 consensus_peers.cc:489] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:39.974782 27693 raft_consensus.cc:357] T 60c32370a5274347afafc6a7e1e94270 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.977506 27693 raft_consensus.cc:383] T 60c32370a5274347afafc6a7e1e94270 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:39.977663 27693 raft_consensus.cc:738] T 60c32370a5274347afafc6a7e1e94270 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:39.978034 27693 consensus_queue.cc:260] T 60c32370a5274347afafc6a7e1e94270 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.979378 27693 ts_tablet_manager.cc:1428] T 60c32370a5274347afafc6a7e1e94270 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250903 20:05:39.980176 27693 tablet_bootstrap.cc:492] T 13496efe34944b7280e7a684f180e6e0 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:39.982587 27693 tablet_bootstrap.cc:654] T 13496efe34944b7280e7a684f180e6e0 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.967278 29443 raft_consensus.cc:491] T 8e12f4c420874ab382d433982bddbf05 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.983654 29443 raft_consensus.cc:513] T 8e12f4c420874ab382d433982bddbf05 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.984449 29443 leader_election.cc:290] T 8e12f4c420874ab382d433982bddbf05 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:39.969563 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed4902c002ef4198802375ced39ef2ad" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:39.985309 27516 raft_consensus.cc:2466] T ed4902c002ef4198802375ced39ef2ad P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:39.985723 27684 pending_rounds.cc:70] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f: Trying to abort 1 pending ops.
I20250903 20:05:39.985827 27684 pending_rounds.cc:77] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196385029557157888 op_type: NO_OP noop_request { }
I20250903 20:05:39.986014 27684 raft_consensus.cc:2887] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:39.986179 27684 raft_consensus.cc:2270] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:39.987512 27183 leader_election.cc:304] T ed4902c002ef4198802375ced39ef2ad P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:39.988018 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e12f4c420874ab382d433982bddbf05" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:39.988814 27462 leader_election.cc:343] T 8e12f4c420874ab382d433982bddbf05 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:39.989089 29455 raft_consensus.cc:2802] T ed4902c002ef4198802375ced39ef2ad P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:39.989162 29455 raft_consensus.cc:491] T ed4902c002ef4198802375ced39ef2ad P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:39.989219 29455 raft_consensus.cc:3058] T ed4902c002ef4198802375ced39ef2ad P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:39.990446 27692 tablet_bootstrap.cc:654] T b84f487773c24735a61d0fd0898b2bcb P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:39.991549 28567 raft_consensus.cc:695] T 249362a0b8304719863bcb82c47273b3 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:39.991827 28567 consensus_queue.cc:237] T 249362a0b8304719863bcb82c47273b3 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.992096 27684 ts_tablet_manager.cc:1905] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:39.998176 29443 raft_consensus.cc:491] T 0ca6e2165a9d41299c1fe0bfa9475915 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:39.998343 29443 raft_consensus.cc:513] T 0ca6e2165a9d41299c1fe0bfa9475915 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:39.998874 29443 leader_election.cc:290] T 0ca6e2165a9d41299c1fe0bfa9475915 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:39.999256 27693 tablet_bootstrap.cc:492] T 13496efe34944b7280e7a684f180e6e0 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:39.999435 27693 ts_tablet_manager.cc:1397] T 13496efe34944b7280e7a684f180e6e0 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20250903 20:05:40.000001 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0ca6e2165a9d41299c1fe0bfa9475915" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:40.000666 27462 leader_election.cc:343] T 0ca6e2165a9d41299c1fe0bfa9475915 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.001116 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0ca6e2165a9d41299c1fe0bfa9475915" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:40.001570 27462 leader_election.cc:343] T 0ca6e2165a9d41299c1fe0bfa9475915 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.001675 27462 leader_election.cc:304] T 0ca6e2165a9d41299c1fe0bfa9475915 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:40.002070 29443 raft_consensus.cc:2747] T 0ca6e2165a9d41299c1fe0bfa9475915 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:40.003172 29455 raft_consensus.cc:513] T ed4902c002ef4198802375ced39ef2ad P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.004216 29455 leader_election.cc:290] T ed4902c002ef4198802375ced39ef2ad P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:40.004328 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed4902c002ef4198802375ced39ef2ad" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:40.004513 27516 raft_consensus.cc:3058] T ed4902c002ef4198802375ced39ef2ad P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:40.007908 27516 raft_consensus.cc:2466] T ed4902c002ef4198802375ced39ef2ad P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:40.008280 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed4902c002ef4198802375ced39ef2ad" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
W20250903 20:05:40.008924 27184 leader_election.cc:343] T ed4902c002ef4198802375ced39ef2ad P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.009094 27184 leader_election.cc:304] T ed4902c002ef4198802375ced39ef2ad P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:40.009407 29669 raft_consensus.cc:491] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:40.009516 29589 raft_consensus.cc:2802] T ed4902c002ef4198802375ced39ef2ad P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:40.009632 29589 raft_consensus.cc:695] T ed4902c002ef4198802375ced39ef2ad P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:40.009588 29669 raft_consensus.cc:513] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.009872 29589 consensus_queue.cc:237] T ed4902c002ef4198802375ced39ef2ad P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.011606 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4d3e1cfdfab4be1a9143b78bf3bc46d" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:40.011772 27240 raft_consensus.cc:2466] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:40.012152 27321 leader_election.cc:304] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:40.012549 29669 leader_election.cc:290] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:40.013073 29669 raft_consensus.cc:2802] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:40.013173 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e12f4c420874ab382d433982bddbf05" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:40.013520 27320 consensus_peers.cc:489] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39c2ebe4ada5426590ec3769c8662644. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:40.013621 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4d3e1cfdfab4be1a9143b78bf3bc46d" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:40.013878 27516 raft_consensus.cc:2466] T f4d3e1cfdfab4be1a9143b78bf3bc46d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
W20250903 20:05:40.014055 27462 leader_election.cc:343] T 8e12f4c420874ab382d433982bddbf05 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.014168 27462 leader_election.cc:304] T 8e12f4c420874ab382d433982bddbf05 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:40.014731 29443 raft_consensus.cc:2747] T 8e12f4c420874ab382d433982bddbf05 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:40.015571 27692 tablet_bootstrap.cc:492] T b84f487773c24735a61d0fd0898b2bcb P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:40.015704 27692 ts_tablet_manager.cc:1397] T b84f487773c24735a61d0fd0898b2bcb P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.060s user 0.002s sys 0.001s
I20250903 20:05:40.016474 27692 raft_consensus.cc:357] T b84f487773c24735a61d0fd0898b2bcb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.016638 27692 raft_consensus.cc:383] T b84f487773c24735a61d0fd0898b2bcb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:40.016692 27692 raft_consensus.cc:738] T b84f487773c24735a61d0fd0898b2bcb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:40.016913 27692 consensus_queue.cc:260] T b84f487773c24735a61d0fd0898b2bcb P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.017606 27692 ts_tablet_manager.cc:1428] T b84f487773c24735a61d0fd0898b2bcb P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:40.017894 27692 tablet_bootstrap.cc:492] T 05e949476a834762973cb98923420f69 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:40.023630 27695 tablet_bootstrap.cc:492] T ea3f82f4e4244219ab0919f3e341aead P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:40.023873 27695 ts_tablet_manager.cc:1397] T ea3f82f4e4244219ab0919f3e341aead P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.088s user 0.007s sys 0.005s
I20250903 20:05:40.024938 27693 raft_consensus.cc:357] T 13496efe34944b7280e7a684f180e6e0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:40.024964 27695 raft_consensus.cc:357] T ea3f82f4e4244219ab0919f3e341aead P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.025141 27695 raft_consensus.cc:383] T ea3f82f4e4244219ab0919f3e341aead P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:40.025209 27695 raft_consensus.cc:738] T ea3f82f4e4244219ab0919f3e341aead P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:40.025485 27695 consensus_queue.cc:260] T ea3f82f4e4244219ab0919f3e341aead P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.026206 27695 ts_tablet_manager.cc:1428] T ea3f82f4e4244219ab0919f3e341aead P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:40.026584 27695 tablet_bootstrap.cc:492] T 54f1ce5192f547f595dea23a9eabf8e6 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:40.026916 27693 raft_consensus.cc:383] T 13496efe34944b7280e7a684f180e6e0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:40.027119 27693 raft_consensus.cc:738] T 13496efe34944b7280e7a684f180e6e0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:40.013195 29669 raft_consensus.cc:491] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:40.027513 29669 raft_consensus.cc:3058] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:40.027554 27693 consensus_queue.cc:260] T 13496efe34944b7280e7a684f180e6e0 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:40.029620 27695 tablet_bootstrap.cc:654] T 54f1ce5192f547f595dea23a9eabf8e6 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:40.029815 27053 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 7c200f2becd84e3482e528191cbc2868 (table default.Table_f4791f9e8f0643ffa7b74d16fb1ad130 [id=52a15cfac52043afb5e3bbdf0fb01c1e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7c200f2becd84e3482e528191cbc2868 consensus state has no leader
I20250903 20:05:40.030198 27692 tablet_bootstrap.cc:654] T 05e949476a834762973cb98923420f69 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:40.032200 27693 ts_tablet_manager.cc:1428] T 13496efe34944b7280e7a684f180e6e0 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.033s user 0.002s sys 0.000s
I20250903 20:05:40.032589 27693 tablet_bootstrap.cc:492] T 7d5a3fbe93e44c228222ab1b1e78c187 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:40.036509 27684 ts_tablet_manager.cc:1918] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:40.036742 27684 log.cc:1199] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/0fef9aa7153a4902abce522e3ff0df3f
I20250903 20:05:40.037180 27684 ts_tablet_manager.cc:1939] T 0fef9aa7153a4902abce522e3ff0df3f P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:40.038184 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1185) took 3429 ms. Trace:
I20250903 20:05:40.038376 27684 rpcz_store.cc:276] 0903 20:05:36.608450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.608980 (+ 530us) service_pool.cc:225] Handling call
0903 20:05:39.963579 (+3354599us) ts_tablet_manager.cc:1145] Deleting tablet 0fef9aa7153a4902abce522e3ff0df3f
0903 20:05:39.963594 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:39.995932 (+ 32338us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.036365 (+ 40433us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.038160 (+ 1795us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3354319}
I20250903 20:05:40.038908 29669 raft_consensus.cc:513] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.041831 27063 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_447c4407775d4be6b8b3bcb7b6849333"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:40.042533 27063 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_447c4407775d4be6b8b3bcb7b6849333 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:40.043577 29669 leader_election.cc:290] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:40.043845 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b3d9491f777b422da4b9fca803beabff consensus state has no leader
I20250903 20:05:40.044003 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4d3e1cfdfab4be1a9143b78bf3bc46d" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:40.044189 27249 raft_consensus.cc:3058] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:40.044162 27060 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_48ce0908f41148948ef0a0d8907a8b45" } new_table_name: "default.Table_b457240e12b14e14a471517d02b189d8" modify_external_catalogs: true
I20250903 20:05:40.046691 27693 tablet_bootstrap.cc:654] T 7d5a3fbe93e44c228222ab1b1e78c187 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:40.046902 27249 raft_consensus.cc:2466] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:40.047349 27321 leader_election.cc:304] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:40.043074 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4d3e1cfdfab4be1a9143b78bf3bc46d" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:40.048354 27516 raft_consensus.cc:3058] T f4d3e1cfdfab4be1a9143b78bf3bc46d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:40.048611 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet fa2014bccb5e4f0a91cbc9845df4634d
I20250903 20:05:40.049330 27057 catalog_manager.cc:6652] ReplaceTablet: tablet fa2014bccb5e4f0a91cbc9845df4634d deleted and replaced by tablet deb52499131a4ed18e1c4777cb99b2f0
I20250903 20:05:40.049536 29589 consensus_queue.cc:1035] T 4739f10128d64e2c986257b1d9deb90f P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:40.049683 27057 master_service.cc:948] ReplaceTablet: received request to replace tablet f27e2aa2d0bf4de48429a030146df3d8 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:40.050922 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet fa2014bccb5e4f0a91cbc9845df4634d with delete_type TABLET_DATA_DELETED (replaced by tablet deb52499131a4ed18e1c4777cb99b2f0) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:40.051231 27516 raft_consensus.cc:2466] T f4d3e1cfdfab4be1a9143b78bf3bc46d P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:40.051656 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet fa2014bccb5e4f0a91cbc9845df4634d with delete_type TABLET_DATA_DELETED (replaced by tablet deb52499131a4ed18e1c4777cb99b2f0) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:40.051980 29461 leader_election.cc:290] T f9375fff5f5945078c4412aa21c06f0d P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:40.052556 29669 raft_consensus.cc:2802] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:40.048183 27692 tablet_bootstrap.cc:492] T 05e949476a834762973cb98923420f69 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:40.052918 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet fa2014bccb5e4f0a91cbc9845df4634d with delete_type TABLET_DATA_DELETED (replaced by tablet deb52499131a4ed18e1c4777cb99b2f0) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:40.053078 27692 ts_tablet_manager.cc:1397] T 05e949476a834762973cb98923420f69 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.001s
I20250903 20:05:40.054164 27692 raft_consensus.cc:357] T 05e949476a834762973cb98923420f69 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.054484 27692 raft_consensus.cc:383] T 05e949476a834762973cb98923420f69 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:40.054550 27692 raft_consensus.cc:738] T 05e949476a834762973cb98923420f69 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
W20250903 20:05:40.054634 27061 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f268cdd43a4048ddbfeb7a2ec87fcb03 has no consensus state
I20250903 20:05:40.054912 27061 master_service.cc:948] ReplaceTablet: received request to replace tablet 4ae0446b76d041c98fa76dfeadab5276 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:40.054818 27692 consensus_queue.cc:260] T 05e949476a834762973cb98923420f69 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.055552 27692 ts_tablet_manager.cc:1428] T 05e949476a834762973cb98923420f69 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:40.056499 27692 tablet_bootstrap.cc:492] T 6d21f38759d743af972635f8b5b8d8d2 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:40.057369 27685 ts_tablet_manager.cc:1918] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:40.057508 27685 log.cc:1199] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/f1d22e76ea72441db51f7e9db26eb520
I20250903 20:05:40.057916 29669 raft_consensus.cc:695] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:40.059190 28567 raft_consensus.cc:491] T 13496efe34944b7280e7a684f180e6e0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:40.059345 28567 raft_consensus.cc:513] T 13496efe34944b7280e7a684f180e6e0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:40.059490 27692 tablet_bootstrap.cc:654] T 6d21f38759d743af972635f8b5b8d8d2 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:40.060138 29589 raft_consensus.cc:491] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:40.060300 29589 raft_consensus.cc:513] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.060426 28567 leader_election.cc:290] T 13496efe34944b7280e7a684f180e6e0 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:40.060951 29589 leader_election.cc:290] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
W20250903 20:05:40.061678 27320 leader_election.cc:343] T 13496efe34944b7280e7a684f180e6e0 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 13496efe34944b7280e7a684f180e6e0
I20250903 20:05:40.061686 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 0fef9aa7153a4902abce522e3ff0df3f (table default.Table_c304f6eaf4014caba10a5b376c13972d [id=f8544f28af82478aa3762f84b040a6d4]) successfully deleted
I20250903 20:05:40.062403 29589 consensus_queue.cc:1035] T 1bf570855b204b96acd070587c434463 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:40.062512 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17f16e9c8f794e03a2f8b3b6b35f8e84" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:40.062690 27386 raft_consensus.cc:2466] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:40.063078 27184 leader_election.cc:304] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:40.064483 29589 raft_consensus.cc:2802] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:40.064569 29589 raft_consensus.cc:491] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:40.064631 29589 raft_consensus.cc:3058] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:40.062142 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1186) took 3439 ms. Trace:
I20250903 20:05:40.064707 27684 rpcz_store.cc:276] 0903 20:05:36.622644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.622705 (+ 61us) service_pool.cc:225] Handling call
0903 20:05:40.062017 (+3439312us) ts_tablet_manager.cc:1145] Deleting tablet 827d186faaa140bdab2f87cf421ca64c
0903 20:05:40.062030 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.062123 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3439090}
W20250903 20:05:40.065475 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 827d186faaa140bdab2f87cf421ca64c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 827d186faaa140bdab2f87cf421ca64c already in progress: creating tablet
I20250903 20:05:40.058331 29669 consensus_queue.cc:237] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.066354 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13496efe34944b7280e7a684f180e6e0" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:40.067054 27321 leader_election.cc:343] T 13496efe34944b7280e7a684f180e6e0 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.067159 27321 leader_election.cc:304] T 13496efe34944b7280e7a684f180e6e0 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:40.067526 28567 raft_consensus.cc:2747] T 13496efe34944b7280e7a684f180e6e0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:40.067461 27684 tablet_replica.cc:331] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:40.068068 27684 raft_consensus.cc:2241] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:40.068339 27684 raft_consensus.cc:2270] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
W20250903 20:05:40.068322 27183 leader_election.cc:343] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 17f16e9c8f794e03a2f8b3b6b35f8e84
I20250903 20:05:40.069164 27684 ts_tablet_manager.cc:1905] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:40.070041 27386 raft_consensus.cc:3058] T 1bf570855b204b96acd070587c434463 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:40.070732 29589 raft_consensus.cc:513] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:40.072547 27183 leader_election.cc:343] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 17f16e9c8f794e03a2f8b3b6b35f8e84
I20250903 20:05:40.073267 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17f16e9c8f794e03a2f8b3b6b35f8e84" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:40.073449 27385 raft_consensus.cc:3058] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:40.073465 29589 leader_election.cc:290] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:40.075453 27058 master_service.cc:948] ReplaceTablet: received request to replace tablet 7dc5214512af449bbc537f111a3d8dbc from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:40.075863 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5f0199a660844c15a254a062c1540498" } new_table_name: "default.Table_aabc7101f3e24a189aee5e863a629263" modify_external_catalogs: true
I20250903 20:05:40.076087 27385 raft_consensus.cc:2466] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:40.076481 27184 leader_election.cc:304] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:40.076784 29461 raft_consensus.cc:2802] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:40.077279 29461 raft_consensus.cc:695] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:40.077438 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_7b633e99fc1749d19a0a58bce6de6000"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:40.077581 29461 consensus_queue.cc:237] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:40.077948 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b633e99fc1749d19a0a58bce6de6000 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:40.069806 29443 raft_consensus.cc:491] T 351208baafc94f5eab531101d52e3b3f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:40.080346 29443 raft_consensus.cc:513] T 351208baafc94f5eab531101d52e3b3f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.080730 29589 consensus_queue.cc:1035] T 1bf570855b204b96acd070587c434463 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:40.080909 29443 leader_election.cc:290] T 351208baafc94f5eab531101d52e3b3f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:40.081768 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "351208baafc94f5eab531101d52e3b3f" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:40.083005 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "351208baafc94f5eab531101d52e3b3f" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:40.083122 27462 leader_election.cc:343] T 351208baafc94f5eab531101d52e3b3f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:40.083371 27462 leader_election.cc:343] T 351208baafc94f5eab531101d52e3b3f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.083462 27462 leader_election.cc:304] T 351208baafc94f5eab531101d52e3b3f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:40.083910 29443 raft_consensus.cc:2747] T 351208baafc94f5eab531101d52e3b3f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:40.089128 27060 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b256c6340bf453180048ed621c3df8c consensus state has no leader
I20250903 20:05:40.089604 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_092cc68e6e774d91ba49dc1622f26b0b" } modify_external_catalogs: true
I20250903 20:05:40.089780 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_092cc68e6e774d91ba49dc1622f26b0b" } modify_external_catalogs: true
I20250903 20:05:40.092635 27695 tablet_bootstrap.cc:492] T 54f1ce5192f547f595dea23a9eabf8e6 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:40.092774 27695 ts_tablet_manager.cc:1397] T 54f1ce5192f547f595dea23a9eabf8e6 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.066s user 0.005s sys 0.001s
I20250903 20:05:40.093568 27695 raft_consensus.cc:357] T 54f1ce5192f547f595dea23a9eabf8e6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.093720 27695 raft_consensus.cc:383] T 54f1ce5192f547f595dea23a9eabf8e6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:40.093762 27695 raft_consensus.cc:738] T 54f1ce5192f547f595dea23a9eabf8e6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:40.094062 27695 consensus_queue.cc:260] T 54f1ce5192f547f595dea23a9eabf8e6 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.094311 27685 ts_tablet_manager.cc:1939] T f1d22e76ea72441db51f7e9db26eb520 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:40.094653 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1060) took 5630 ms. Trace:
I20250903 20:05:40.094698 27695 ts_tablet_manager.cc:1428] T 54f1ce5192f547f595dea23a9eabf8e6 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250903 20:05:40.094733 27685 rpcz_store.cc:276] 0903 20:05:34.464181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.464958 (+ 777us) service_pool.cc:225] Handling call
0903 20:05:39.718748 (+5253790us) ts_tablet_manager.cc:1145] Deleting tablet f1d22e76ea72441db51f7e9db26eb520
0903 20:05:39.718757 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.049193 (+330436us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.057225 (+ 8032us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.094638 (+ 37413us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5253618}
I20250903 20:05:40.095424 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet f1d22e76ea72441db51f7e9db26eb520 (table default.Table_258f358903dd4f6fa7e310d72562a570 [id=ae5e27c159a9462482d09a79e148f795]) successfully deleted
I20250903 20:05:40.095847 27695 tablet_bootstrap.cc:492] T 5b46bb34ac214b1ba23fba1a33f746ac P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:40.097546 27695 tablet_bootstrap.cc:654] T 5b46bb34ac214b1ba23fba1a33f746ac P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:40.097802 29443 raft_consensus.cc:491] T bc98762790f441038972366b1cc2c55b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:40.097929 29443 raft_consensus.cc:513] T bc98762790f441038972366b1cc2c55b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.098308 29411 consensus_queue.cc:1035] T 4739f10128d64e2c986257b1d9deb90f P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:40.098544 29443 leader_election.cc:290] T bc98762790f441038972366b1cc2c55b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:40.100009 27376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc98762790f441038972366b1cc2c55b" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:40.100518 27462 leader_election.cc:343] T bc98762790f441038972366b1cc2c55b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.099865 27693 tablet_bootstrap.cc:492] T 7d5a3fbe93e44c228222ab1b1e78c187 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:40.100754 27693 ts_tablet_manager.cc:1397] T 7d5a3fbe93e44c228222ab1b1e78c187 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.068s user 0.004s sys 0.000s
I20250903 20:05:40.101853 27693 raft_consensus.cc:357] T 7d5a3fbe93e44c228222ab1b1e78c187 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.102164 27693 raft_consensus.cc:383] T 7d5a3fbe93e44c228222ab1b1e78c187 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:40.102246 27693 raft_consensus.cc:738] T 7d5a3fbe93e44c228222ab1b1e78c187 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:40.102452 27693 consensus_queue.cc:260] T 7d5a3fbe93e44c228222ab1b1e78c187 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.102981 27693 ts_tablet_manager.cc:1428] T 7d5a3fbe93e44c228222ab1b1e78c187 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250903 20:05:40.103696 27693 tablet_bootstrap.cc:492] T f9375fff5f5945078c4412aa21c06f0d P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:40.104431 27063 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1828d17a24374724b13ff20d37532044" } modify_external_catalogs: true
I20250903 20:05:40.104537 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc98762790f441038972366b1cc2c55b" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:40.104671 27063 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_1828d17a24374724b13ff20d37532044" } modify_external_catalogs: true
W20250903 20:05:40.105249 27462 leader_election.cc:343] T bc98762790f441038972366b1cc2c55b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.105398 27462 leader_election.cc:304] T bc98762790f441038972366b1cc2c55b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:40.105849 29612 raft_consensus.cc:2747] T bc98762790f441038972366b1cc2c55b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:40.105996 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0a4db1d67753479197d68a85fb3d6c5c" } modify_external_catalogs: true
I20250903 20:05:40.107007 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_9ea78ec0f6314be48893b91681dd50f3" } modify_external_catalogs: true
I20250903 20:05:40.107177 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_9ea78ec0f6314be48893b91681dd50f3" } modify_external_catalogs: true
I20250903 20:05:40.109781 27687 ts_tablet_manager.cc:1918] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:40.111187 27693 tablet_bootstrap.cc:654] T f9375fff5f5945078c4412aa21c06f0d P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:40.106184 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_0a4db1d67753479197d68a85fb3d6c5c" } modify_external_catalogs: true
I20250903 20:05:40.118592 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1059) took 5654 ms. Trace:
I20250903 20:05:40.119668 29612 raft_consensus.cc:491] T ea3f82f4e4244219ab0919f3e341aead P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:40.119828 29612 raft_consensus.cc:513] T ea3f82f4e4244219ab0919f3e341aead P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.119874 27685 rpcz_store.cc:276] 0903 20:05:34.464083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.465163 (+ 1080us) service_pool.cc:225] Handling call
0903 20:05:40.118418 (+5653255us) ts_tablet_manager.cc:1145] Deleting tablet 2035344c7f6d478b89a89981d89f22ca
0903 20:05:40.118432 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.118563 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5653122}
W20250903 20:05:40.120324 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 2035344c7f6d478b89a89981d89f22ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2035344c7f6d478b89a89981d89f22ca already in progress: creating tablet
I20250903 20:05:40.120483 29612 leader_election.cc:290] T ea3f82f4e4244219ab0919f3e341aead P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:40.120772 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet ac1b0dc95fca49b4af6697c0a70e9730
I20250903 20:05:40.120982 27685 tablet_replica.cc:331] T 5557dfdd953a41e9bd3a8be4cd689604 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:40.121192 27685 raft_consensus.cc:2241] T 5557dfdd953a41e9bd3a8be4cd689604 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:40.121490 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet ac1b0dc95fca49b4af6697c0a70e9730 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:40 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:40.121516 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea3f82f4e4244219ab0919f3e341aead" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:40.122053 27462 leader_election.cc:343] T ea3f82f4e4244219ab0919f3e341aead P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.122494 27687 log.cc:1199] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/47c1287bff1145a5b2ab6d14dbd608c7
I20250903 20:05:40.123008 27687 ts_tablet_manager.cc:1939] T 47c1287bff1145a5b2ab6d14dbd608c7 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:40.122933 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7ad5f88730c24cbbb97feda4328aa2ec" } new_table_name: "default.Table_5ae61d1284b44382bac99dec4273846c" modify_external_catalogs: true
I20250903 20:05:40.122473 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet ac1b0dc95fca49b4af6697c0a70e9730 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:40 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:40.123057 29612 raft_consensus.cc:491] T 8218868ff40b405387d560f614427c08 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:40.122068 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea3f82f4e4244219ab0919f3e341aead" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:40.123175 29612 raft_consensus.cc:513] T 8218868ff40b405387d560f614427c08 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.123473 27249 raft_consensus.cc:2466] T ea3f82f4e4244219ab0919f3e341aead P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 0.
I20250903 20:05:40.123775 29612 leader_election.cc:290] T 8218868ff40b405387d560f614427c08 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:40.121373 27685 raft_consensus.cc:2270] T 5557dfdd953a41e9bd3a8be4cd689604 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:40.123993 27462 leader_election.cc:304] T ea3f82f4e4244219ab0919f3e341aead P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:40.124809 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8218868ff40b405387d560f614427c08" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:40.124996 29612 raft_consensus.cc:2802] T ea3f82f4e4244219ab0919f3e341aead P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:40.125100 29612 raft_consensus.cc:491] T ea3f82f4e4244219ab0919f3e341aead P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:40.125181 29612 raft_consensus.cc:3058] T ea3f82f4e4244219ab0919f3e341aead P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:40.125496 27462 leader_election.cc:343] T 8218868ff40b405387d560f614427c08 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.125737 27695 tablet_bootstrap.cc:492] T 5b46bb34ac214b1ba23fba1a33f746ac P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:40.125931 27695 ts_tablet_manager.cc:1397] T 5b46bb34ac214b1ba23fba1a33f746ac P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.005s
I20250903 20:05:40.125880 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8218868ff40b405387d560f614427c08" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:40.126940 27685 ts_tablet_manager.cc:1905] T 5557dfdd953a41e9bd3a8be4cd689604 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:40.127161 27462 leader_election.cc:343] T 8218868ff40b405387d560f614427c08 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.127269 27462 leader_election.cc:304] T 8218868ff40b405387d560f614427c08 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:40.127624 29443 raft_consensus.cc:2747] T 8218868ff40b405387d560f614427c08 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:40.128159 29612 raft_consensus.cc:513] T ea3f82f4e4244219ab0919f3e341aead P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.129832 29612 leader_election.cc:290] T ea3f82f4e4244219ab0919f3e341aead P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:40.130142 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea3f82f4e4244219ab0919f3e341aead" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
W20250903 20:05:40.130841 27462 leader_election.cc:343] T ea3f82f4e4244219ab0919f3e341aead P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.132903 27693 tablet_bootstrap.cc:492] T f9375fff5f5945078c4412aa21c06f0d P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:40.133093 27693 ts_tablet_manager.cc:1397] T f9375fff5f5945078c4412aa21c06f0d P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.003s
I20250903 20:05:40.134752 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1266) took 2499 ms. Trace:
I20250903 20:05:40.134893 27687 rpcz_store.cc:276] 0903 20:05:37.634948 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:37.642199 (+ 7251us) service_pool.cc:225] Handling call
0903 20:05:39.851060 (+2208861us) ts_tablet_manager.cc:1145] Deleting tablet 47c1287bff1145a5b2ab6d14dbd608c7
0903 20:05:39.851070 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.064440 (+213370us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.109648 (+ 45208us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.134722 (+ 25074us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2208609}
I20250903 20:05:40.135465 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1270) took 2465 ms. Trace:
I20250903 20:05:40.135537 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 47c1287bff1145a5b2ab6d14dbd608c7 (table default.Table_0cd65e00bd5a40bd9dc9db12fd0bda4d [id=15369700921549be8faa450d2ca98949]) successfully deleted
I20250903 20:05:40.135569 27687 rpcz_store.cc:276] 0903 20:05:37.669890 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:37.669953 (+ 63us) service_pool.cc:225] Handling call
0903 20:05:40.135307 (+2465354us) ts_tablet_manager.cc:1145] Deleting tablet 74eab6ad2fe64b18bd00c0d5524428c6
0903 20:05:40.135321 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.135448 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2464385}
I20250903 20:05:40.135960 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1269) took 2476 ms. Trace:
W20250903 20:05:40.136013 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 74eab6ad2fe64b18bd00c0d5524428c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74eab6ad2fe64b18bd00c0d5524428c6 already in progress: creating tablet
I20250903 20:05:40.136050 27687 rpcz_store.cc:276] 0903 20:05:37.659917 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:37.662988 (+ 3071us) service_pool.cc:225] Handling call
0903 20:05:40.135848 (+2472860us) ts_tablet_manager.cc:1145] Deleting tablet e4ec4cc1ef914cffa2d304cdf701cf33
0903 20:05:40.135859 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.135944 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2464094}
I20250903 20:05:40.136443 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1268) took 2476 ms. Trace:
W20250903 20:05:40.136512 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet e4ec4cc1ef914cffa2d304cdf701cf33 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4ec4cc1ef914cffa2d304cdf701cf33 already in progress: creating tablet
I20250903 20:05:40.136535 27687 rpcz_store.cc:276] 0903 20:05:37.659747 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:37.678420 (+ 18673us) service_pool.cc:225] Handling call
0903 20:05:40.136336 (+2457916us) ts_tablet_manager.cc:1145] Deleting tablet 503dba32175c4459892c62d3f80693fb
0903 20:05:40.136346 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.136426 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2457620}
I20250903 20:05:40.136909 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1271) took 2448 ms. Trace:
W20250903 20:05:40.136978 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 503dba32175c4459892c62d3f80693fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 503dba32175c4459892c62d3f80693fb already in progress: creating tablet
I20250903 20:05:40.136993 27687 rpcz_store.cc:276] 0903 20:05:37.688150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:37.689295 (+ 1145us) service_pool.cc:225] Handling call
0903 20:05:40.136802 (+2447507us) ts_tablet_manager.cc:1145] Deleting tablet d7d37849132d4420bb5fcb57c11cfb6b
0903 20:05:40.136813 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.136894 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2447140}
I20250903 20:05:40.137368 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1272) took 2250 ms. Trace:
W20250903 20:05:40.137413 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet d7d37849132d4420bb5fcb57c11cfb6b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7d37849132d4420bb5fcb57c11cfb6b already in progress: creating tablet
I20250903 20:05:40.137461 27687 rpcz_store.cc:276] 0903 20:05:37.886658 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:37.886773 (+ 115us) service_pool.cc:225] Handling call
0903 20:05:40.137261 (+2250488us) ts_tablet_manager.cc:1145] Deleting tablet e9b5ab312f8744af99d557c0733bf71d
0903 20:05:40.137272 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.137352 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2250162}
I20250903 20:05:40.128654 27695 raft_consensus.cc:357] T 5b46bb34ac214b1ba23fba1a33f746ac P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:40.137825 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1273) took 2018 ms. Trace:
W20250903 20:05:40.137877 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet e9b5ab312f8744af99d557c0733bf71d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9b5ab312f8744af99d557c0733bf71d already in progress: creating tablet
I20250903 20:05:40.137905 27687 rpcz_store.cc:276] 0903 20:05:38.119313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.119381 (+ 68us) service_pool.cc:225] Handling call
0903 20:05:40.137724 (+2018343us) ts_tablet_manager.cc:1145] Deleting tablet a7c8657489734199b346033bd4afb02e
0903 20:05:40.137734 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.137810 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2018122}
I20250903 20:05:40.138051 29612 raft_consensus.cc:491] T 5c42d84a6a5649208b0539a10da21aec P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250903 20:05:40.138332 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet a7c8657489734199b346033bd4afb02e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7c8657489734199b346033bd4afb02e already in progress: creating tablet
I20250903 20:05:40.138357 29612 raft_consensus.cc:513] T 5c42d84a6a5649208b0539a10da21aec P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.139709 29612 leader_election.cc:290] T 5c42d84a6a5649208b0539a10da21aec P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:40.140862 27693 raft_consensus.cc:357] T f9375fff5f5945078c4412aa21c06f0d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.129940 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea3f82f4e4244219ab0919f3e341aead" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:40.140231 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c42d84a6a5649208b0539a10da21aec" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:40.141849 27695 raft_consensus.cc:383] T 5b46bb34ac214b1ba23fba1a33f746ac P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:40.141927 27695 raft_consensus.cc:738] T 5b46bb34ac214b1ba23fba1a33f746ac P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:40.141933 27249 raft_consensus.cc:3058] T ea3f82f4e4244219ab0919f3e341aead P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:40.142498 27462 leader_election.cc:343] T 5c42d84a6a5649208b0539a10da21aec P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.142971 27692 tablet_bootstrap.cc:492] T 6d21f38759d743af972635f8b5b8d8d2 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:40.143081 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet ac1b0dc95fca49b4af6697c0a70e9730 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:40 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:40.143143 27692 ts_tablet_manager.cc:1397] T 6d21f38759d743af972635f8b5b8d8d2 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.087s user 0.005s sys 0.000s
I20250903 20:05:40.143982 27692 raft_consensus.cc:357] T 6d21f38759d743af972635f8b5b8d8d2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.144205 27692 raft_consensus.cc:383] T 6d21f38759d743af972635f8b5b8d8d2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:40.144307 27692 raft_consensus.cc:738] T 6d21f38759d743af972635f8b5b8d8d2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:40.144580 27692 consensus_queue.cc:260] T 6d21f38759d743af972635f8b5b8d8d2 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.144738 27249 raft_consensus.cc:2466] T ea3f82f4e4244219ab0919f3e341aead P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72ef1dbcc5143cf9f04a808f101c80f in term 1.
I20250903 20:05:40.145155 27462 leader_election.cc:304] T ea3f82f4e4244219ab0919f3e341aead P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:40.145592 29612 raft_consensus.cc:2802] T ea3f82f4e4244219ab0919f3e341aead P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:40.145804 29612 raft_consensus.cc:695] T ea3f82f4e4244219ab0919f3e341aead P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Becoming Leader. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Running, Role: LEADER
I20250903 20:05:40.145849 29461 raft_consensus.cc:491] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:40.145979 29461 raft_consensus.cc:513] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.145931 27695 consensus_queue.cc:260] T 5b46bb34ac214b1ba23fba1a33f746ac P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:40.146541 29461 leader_election.cc:290] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
W20250903 20:05:40.147521 27183 leader_election.cc:343] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 45f7068bafaf48d290ded9d1272c0e46
I20250903 20:05:40.146064 29612 consensus_queue.cc:237] T ea3f82f4e4244219ab0919f3e341aead P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:40.147869 27184 leader_election.cc:343] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 45f7068bafaf48d290ded9d1272c0e46
I20250903 20:05:40.147977 27184 leader_election.cc:304] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:40.148339 29461 raft_consensus.cc:2747] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:40.148896 27695 ts_tablet_manager.cc:1428] T 5b46bb34ac214b1ba23fba1a33f746ac P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20250903 20:05:40.149606 27695 tablet_bootstrap.cc:492] T 13496efe34944b7280e7a684f180e6e0 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
W20250903 20:05:40.151150 27066 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ab380b2765b142c890c6010fe8d00948 consensus state has no leader
I20250903 20:05:40.151923 27692 ts_tablet_manager.cc:1428] T 6d21f38759d743af972635f8b5b8d8d2 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250903 20:05:40.152247 27692 tablet_bootstrap.cc:492] T 5f2eefeebbfc47249e2f72b7c52e905d P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:40.152908 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_d88ebd3d2c7f4a93bd404a8f39ae6682"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:40.153631 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d88ebd3d2c7f4a93bd404a8f39ae6682 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:40.154026 27692 tablet_bootstrap.cc:654] T 5f2eefeebbfc47249e2f72b7c52e905d P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:40.155791 27684 ts_tablet_manager.cc:1918] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 20:05:40.155906 27684 log.cc:1199] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/0f5b87c1c4a24f8abdbfd6194f3b2c4a
I20250903 20:05:40.156271 27684 ts_tablet_manager.cc:1939] T 0f5b87c1c4a24f8abdbfd6194f3b2c4a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:40.156994 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1187) took 3501 ms. Trace:
I20250903 20:05:40.157081 27684 rpcz_store.cc:276] 0903 20:05:36.655935 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.655988 (+ 53us) service_pool.cc:225] Handling call
0903 20:05:40.067291 (+3411303us) ts_tablet_manager.cc:1145] Deleting tablet 0f5b87c1c4a24f8abdbfd6194f3b2c4a
0903 20:05:40.067409 (+ 118us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.093071 (+ 25662us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.155686 (+ 62615us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.156977 (+ 1291us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3378015}
I20250903 20:05:40.157446 27684 tablet_replica.cc:331] T 6800e0b6269045f29682704fd847d5d1 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:40.157634 27684 raft_consensus.cc:2241] T 6800e0b6269045f29682704fd847d5d1 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:40.157773 27684 raft_consensus.cc:2270] T 6800e0b6269045f29682704fd847d5d1 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:40.141654 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c42d84a6a5649208b0539a10da21aec" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:40.158316 27693 raft_consensus.cc:383] T f9375fff5f5945078c4412aa21c06f0d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:40.158413 27693 raft_consensus.cc:738] T f9375fff5f5945078c4412aa21c06f0d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:40.158692 27693 consensus_queue.cc:260] T f9375fff5f5945078c4412aa21c06f0d P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:40.158875 27462 leader_election.cc:343] T 5c42d84a6a5649208b0539a10da21aec P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.159004 27462 leader_election.cc:304] T 5c42d84a6a5649208b0539a10da21aec P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:40.159220 29443 raft_consensus.cc:2747] T 5c42d84a6a5649208b0539a10da21aec P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:40.159644 27693 ts_tablet_manager.cc:1428] T f9375fff5f5945078c4412aa21c06f0d P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20250903 20:05:40.160391 27693 tablet_bootstrap.cc:492] T 5c42d84a6a5649208b0539a10da21aec P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:40.161623 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 0f5b87c1c4a24f8abdbfd6194f3b2c4a (table default.Table_6e9cf5ead5a24831a6a14d131dd8397e [id=524a27cbe1ad4c10aa2fd4f4074105c6]) successfully deleted
I20250903 20:05:40.161715 27695 tablet_bootstrap.cc:654] T 13496efe34944b7280e7a684f180e6e0 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:40.164469 27693 tablet_bootstrap.cc:654] T 5c42d84a6a5649208b0539a10da21aec P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:40.169191 27687 tablet_replica.cc:331] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:40.170765 27687 raft_consensus.cc:2241] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:40.171314 27687 raft_consensus.cc:2270] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:40.172531 27687 ts_tablet_manager.cc:1905] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:40.179172 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_0100346787c743eeb7b8c2484d7d1f95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:40.181771 27685 ts_tablet_manager.cc:1918] T 5557dfdd953a41e9bd3a8be4cd689604 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:40.182415 27684 ts_tablet_manager.cc:1905] T 6800e0b6269045f29682704fd847d5d1 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:40.183610 27695 tablet_bootstrap.cc:492] T 13496efe34944b7280e7a684f180e6e0 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:40.184095 27695 ts_tablet_manager.cc:1397] T 13496efe34944b7280e7a684f180e6e0 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.004s
I20250903 20:05:40.185005 27695 raft_consensus.cc:357] T 13496efe34944b7280e7a684f180e6e0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:40.185652 27695 raft_consensus.cc:383] T 13496efe34944b7280e7a684f180e6e0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:40.185779 27695 raft_consensus.cc:738] T 13496efe34944b7280e7a684f180e6e0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:40.183917 27685 log.cc:1199] T 5557dfdd953a41e9bd3a8be4cd689604 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/5557dfdd953a41e9bd3a8be4cd689604
I20250903 20:05:40.186403 27685 ts_tablet_manager.cc:1939] T 5557dfdd953a41e9bd3a8be4cd689604 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:40.186789 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1058) took 5722 ms. Trace:
I20250903 20:05:40.188709 27693 tablet_bootstrap.cc:492] T 5c42d84a6a5649208b0539a10da21aec P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:40.189765 27693 ts_tablet_manager.cc:1397] T 5c42d84a6a5649208b0539a10da21aec P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.001s
W20250903 20:05:40.189957 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0100346787c743eeb7b8c2484d7d1f95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:40.191145 28567 raft_consensus.cc:491] T 6800e0b6269045f29682704fd847d5d1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:40.191275 28567 raft_consensus.cc:513] T 6800e0b6269045f29682704fd847d5d1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:40.192449 28567 leader_election.cc:290] T 6800e0b6269045f29682704fd847d5d1 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:40.193861 27509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6800e0b6269045f29682704fd847d5d1" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:40.195258 27320 leader_election.cc:343] T 6800e0b6269045f29682704fd847d5d1 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Illegal state: Tablet not RUNNING: STOPPED
I20250903 20:05:40.196465 27692 tablet_bootstrap.cc:492] T 5f2eefeebbfc47249e2f72b7c52e905d P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:40.196673 27692 ts_tablet_manager.cc:1397] T 5f2eefeebbfc47249e2f72b7c52e905d P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.045s user 0.000s sys 0.004s
I20250903 20:05:40.197616 27692 raft_consensus.cc:357] T 5f2eefeebbfc47249e2f72b7c52e905d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.197846 27692 raft_consensus.cc:383] T 5f2eefeebbfc47249e2f72b7c52e905d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:40.195140 27693 raft_consensus.cc:357] T 5c42d84a6a5649208b0539a10da21aec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.197913 27692 raft_consensus.cc:738] T 5f2eefeebbfc47249e2f72b7c52e905d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:40.198467 27693 raft_consensus.cc:383] T 5c42d84a6a5649208b0539a10da21aec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:40.198609 27692 consensus_queue.cc:260] T 5f2eefeebbfc47249e2f72b7c52e905d P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.198704 27693 raft_consensus.cc:738] T 5c42d84a6a5649208b0539a10da21aec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:40.198998 27695 consensus_queue.cc:260] T 13496efe34944b7280e7a684f180e6e0 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:40.199118 27692 ts_tablet_manager.cc:1428] T 5f2eefeebbfc47249e2f72b7c52e905d P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20250903 20:05:40.199404 27693 consensus_queue.cc:260] T 5c42d84a6a5649208b0539a10da21aec P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.199994 27693 ts_tablet_manager.cc:1428] T 5c42d84a6a5649208b0539a10da21aec P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250903 20:05:40.200713 27693 tablet_bootstrap.cc:492] T f4dec89fc6bb436ba6542c267eaf9c15 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:40.202327 27693 tablet_bootstrap.cc:654] T f4dec89fc6bb436ba6542c267eaf9c15 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:40.205288 27066 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_90b62da1b4554249986db49d13942d17" } modify_external_catalogs: true
I20250903 20:05:40.205551 27066 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_90b62da1b4554249986db49d13942d17" } modify_external_catalogs: true
I20250903 20:05:40.206444 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6800e0b6269045f29682704fd847d5d1" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:40.207134 27321 leader_election.cc:304] T 6800e0b6269045f29682704fd847d5d1 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:40.207639 28567 raft_consensus.cc:2747] T 6800e0b6269045f29682704fd847d5d1 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:40.186944 27685 rpcz_store.cc:276] 0903 20:05:34.463952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.477795 (+ 13843us) service_pool.cc:225] Handling call
0903 20:05:40.120912 (+5643117us) ts_tablet_manager.cc:1145] Deleting tablet 5557dfdd953a41e9bd3a8be4cd689604
0903 20:05:40.120926 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.157422 (+ 36496us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.181616 (+ 24194us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.186772 (+ 5156us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5642863}
I20250903 20:05:40.208076 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1057) took 5765 ms. Trace:
I20250903 20:05:40.208156 27685 rpcz_store.cc:276] 0903 20:05:34.442248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.521281 (+ 79033us) service_pool.cc:225] Handling call
0903 20:05:40.207927 (+5686646us) ts_tablet_manager.cc:1145] Deleting tablet 13137053365a428e9d1e8d8852a53b6a
0903 20:05:40.207937 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.208059 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5686374}
I20250903 20:05:40.208391 27685 tablet_replica.cc:331] T 54f1ce5192f547f595dea23a9eabf8e6 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:40.209296 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 5557dfdd953a41e9bd3a8be4cd689604 (table default.Table_672b60689df64b1d8c9ebd27468d68d4 [id=6d5615e3adc24110a36e1486e69cde7b]) successfully deleted
W20250903 20:05:40.209523 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 13137053365a428e9d1e8d8852a53b6a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13137053365a428e9d1e8d8852a53b6a already in progress: creating tablet
I20250903 20:05:40.209764 27685 raft_consensus.cc:2241] T 54f1ce5192f547f595dea23a9eabf8e6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:40.209928 27685 raft_consensus.cc:2270] T 54f1ce5192f547f595dea23a9eabf8e6 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:40.210636 27685 ts_tablet_manager.cc:1905] T 54f1ce5192f547f595dea23a9eabf8e6 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:40.210717 27695 ts_tablet_manager.cc:1428] T 13496efe34944b7280e7a684f180e6e0 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.026s user 0.000s sys 0.001s
I20250903 20:05:40.211428 27695 tablet_bootstrap.cc:492] T b4b15d77d07c4fa7856d9c198cf15450 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:40.213161 27695 tablet_bootstrap.cc:654] T b4b15d77d07c4fa7856d9c198cf15450 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:40.221524 29461 raft_consensus.cc:491] T 5b46bb34ac214b1ba23fba1a33f746ac P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:40.221714 29461 raft_consensus.cc:513] T 5b46bb34ac214b1ba23fba1a33f746ac P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:40.221997 27685 ts_tablet_manager.cc:1918] T 54f1ce5192f547f595dea23a9eabf8e6 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:40.223037 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b46bb34ac214b1ba23fba1a33f746ac" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:40.223248 27385 raft_consensus.cc:2466] T 5b46bb34ac214b1ba23fba1a33f746ac P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:40.224040 27509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b46bb34ac214b1ba23fba1a33f746ac" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:40.224222 27509 raft_consensus.cc:2466] T 5b46bb34ac214b1ba23fba1a33f746ac P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:40.227296 29443 raft_consensus.cc:491] T 05e949476a834762973cb98923420f69 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:40.227538 27692 tablet_bootstrap.cc:492] T 54602d7f723d4c10af12408a43ed1f02 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:40.227447 29443 raft_consensus.cc:513] T 05e949476a834762973cb98923420f69 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.229387 27692 tablet_bootstrap.cc:654] T 54602d7f723d4c10af12408a43ed1f02 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:40.229703 29443 leader_election.cc:290] T 05e949476a834762973cb98923420f69 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:40.230775 28567 raft_consensus.cc:491] T 5c42d84a6a5649208b0539a10da21aec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:40.230929 28567 raft_consensus.cc:513] T 5c42d84a6a5649208b0539a10da21aec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.231572 28567 leader_election.cc:290] T 5c42d84a6a5649208b0539a10da21aec P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:40.232960 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05e949476a834762973cb98923420f69" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:40.233767 27462 leader_election.cc:343] T 05e949476a834762973cb98923420f69 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.234404 27509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c42d84a6a5649208b0539a10da21aec" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:40.234679 27509 raft_consensus.cc:2466] T 5c42d84a6a5649208b0539a10da21aec P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:40.235184 27320 leader_election.cc:304] T 5c42d84a6a5649208b0539a10da21aec P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:40.235491 28567 raft_consensus.cc:2802] T 5c42d84a6a5649208b0539a10da21aec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:40.235569 28567 raft_consensus.cc:491] T 5c42d84a6a5649208b0539a10da21aec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:40.235634 28567 raft_consensus.cc:3058] T 5c42d84a6a5649208b0539a10da21aec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:40.238343 28567 raft_consensus.cc:513] T 5c42d84a6a5649208b0539a10da21aec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.239554 28567 leader_election.cc:290] T 5c42d84a6a5649208b0539a10da21aec P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:40.239526 27509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c42d84a6a5649208b0539a10da21aec" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:40.244916 27684 ts_tablet_manager.cc:1918] T 6800e0b6269045f29682704fd847d5d1 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:40.222528 29461 leader_election.cc:290] T 5b46bb34ac214b1ba23fba1a33f746ac P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:40.224735 27183 leader_election.cc:304] T 5b46bb34ac214b1ba23fba1a33f746ac P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:40.230607 27240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05e949476a834762973cb98923420f69" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:40.232472 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c42d84a6a5649208b0539a10da21aec" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:40.239656 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c42d84a6a5649208b0539a10da21aec" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
W20250903 20:05:40.246335 27321 leader_election.cc:343] T 5c42d84a6a5649208b0539a10da21aec P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:40.246721 27321 leader_election.cc:343] T 5c42d84a6a5649208b0539a10da21aec P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.247732 29461 raft_consensus.cc:2802] T 5b46bb34ac214b1ba23fba1a33f746ac P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:40.247838 29461 raft_consensus.cc:491] T 5b46bb34ac214b1ba23fba1a33f746ac P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:40.247923 29461 raft_consensus.cc:3058] T 5b46bb34ac214b1ba23fba1a33f746ac P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:40.248145 29707 tablet.cc:1722] T 7345305820be4d32a3ceb227f045f009 P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:40.251562 29461 raft_consensus.cc:513] T 5b46bb34ac214b1ba23fba1a33f746ac P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:40.252995 29461 leader_election.cc:290] T 5b46bb34ac214b1ba23fba1a33f746ac P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:40.253377 27685 log.cc:1199] T 54f1ce5192f547f595dea23a9eabf8e6 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/54f1ce5192f547f595dea23a9eabf8e6
I20250903 20:05:40.253937 27685 ts_tablet_manager.cc:1939] T 54f1ce5192f547f595dea23a9eabf8e6 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:40.255779 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1062) took 5728 ms. Trace:
I20250903 20:05:40.258396 29704 tablet.cc:1722] T 7345305820be4d32a3ceb227f045f009 P 51578fb7bceb4e119b3ab1e58c4f5528: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:40.262120 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b46bb34ac214b1ba23fba1a33f746ac" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:40.262358 27379 raft_consensus.cc:3058] T 5b46bb34ac214b1ba23fba1a33f746ac P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:40.264513 27685 rpcz_store.cc:276] 0903 20:05:34.527592 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.528222 (+ 630us) service_pool.cc:225] Handling call
0903 20:05:40.208354 (+5680132us) ts_tablet_manager.cc:1145] Deleting tablet 54f1ce5192f547f595dea23a9eabf8e6
0903 20:05:40.208361 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.213984 (+ 5623us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.221861 (+ 7877us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.255731 (+ 33870us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5679836}
I20250903 20:05:40.266049 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 54f1ce5192f547f595dea23a9eabf8e6 (table default.Table_7849ab155085483f9e0ba20afc6f34a9 [id=4195535fe553487084ba71bc439c1930]) successfully deleted
I20250903 20:05:40.266124 27685 tablet_replica.cc:331] T 886b8783aa9d4618aad498d4a1115843 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:40.266588 27685 raft_consensus.cc:2241] T 886b8783aa9d4618aad498d4a1115843 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:40.268522 27685 raft_consensus.cc:2270] T 886b8783aa9d4618aad498d4a1115843 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:40.269394 27685 ts_tablet_manager.cc:1905] T 886b8783aa9d4618aad498d4a1115843 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:40.271016 27379 raft_consensus.cc:2466] T 5b46bb34ac214b1ba23fba1a33f746ac P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
W20250903 20:05:40.246003 27462 leader_election.cc:343] T 05e949476a834762973cb98923420f69 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.271412 27462 leader_election.cc:304] T 05e949476a834762973cb98923420f69 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:40.271844 29443 raft_consensus.cc:2747] T 05e949476a834762973cb98923420f69 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:40.261236 27510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b46bb34ac214b1ba23fba1a33f746ac" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:40.273954 27510 raft_consensus.cc:3058] T 5b46bb34ac214b1ba23fba1a33f746ac P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:40.275720 27684 log.cc:1199] T 6800e0b6269045f29682704fd847d5d1 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/6800e0b6269045f29682704fd847d5d1
I20250903 20:05:40.276434 27684 ts_tablet_manager.cc:1939] T 6800e0b6269045f29682704fd847d5d1 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:40.275480 27184 leader_election.cc:304] T 5b46bb34ac214b1ba23fba1a33f746ac P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:40.278645 29461 raft_consensus.cc:2802] T 5b46bb34ac214b1ba23fba1a33f746ac P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:40.279708 29461 raft_consensus.cc:695] T 5b46bb34ac214b1ba23fba1a33f746ac P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:40.280165 29461 consensus_queue.cc:237] T 5b46bb34ac214b1ba23fba1a33f746ac P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:40.282192 27695 tablet_bootstrap.cc:492] T b4b15d77d07c4fa7856d9c198cf15450 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:40.282420 27695 ts_tablet_manager.cc:1397] T b4b15d77d07c4fa7856d9c198cf15450 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.071s user 0.003s sys 0.004s
I20250903 20:05:40.285430 27695 raft_consensus.cc:357] T b4b15d77d07c4fa7856d9c198cf15450 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.285687 27695 raft_consensus.cc:383] T b4b15d77d07c4fa7856d9c198cf15450 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:40.285776 27695 raft_consensus.cc:738] T b4b15d77d07c4fa7856d9c198cf15450 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:40.286036 27695 consensus_queue.cc:260] T b4b15d77d07c4fa7856d9c198cf15450 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.286592 27695 ts_tablet_manager.cc:1428] T b4b15d77d07c4fa7856d9c198cf15450 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250903 20:05:40.286937 27695 tablet_bootstrap.cc:492] T 314856bf269f4df7aa25f5a30cec0d5b P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:40.298938 27687 ts_tablet_manager.cc:1918] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 20:05:40.297988 27064 catalog_manager.cc:5582] T 2385d3006f90486890ebe43e16a53d7a P 89f8f0ee68b940db9ce250c0604f934a reported cstate change: term changed from 0 to 1, leader changed from <none> to 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65). New cstate: current_term: 1 leader_uuid: "89f8f0ee68b940db9ce250c0604f934a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: HEALTHY } } }
I20250903 20:05:40.299705 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1188) took 3487 ms. Trace:
I20250903 20:05:40.299813 27684 rpcz_store.cc:276] 0903 20:05:36.797627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.797705 (+ 78us) service_pool.cc:225] Handling call
0903 20:05:40.157402 (+3359697us) ts_tablet_manager.cc:1145] Deleting tablet 6800e0b6269045f29682704fd847d5d1
0903 20:05:40.157410 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.185893 (+ 28483us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.244786 (+ 58893us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.285153 (+ 40367us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3359458}
I20250903 20:05:40.300256 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1206) took 3355 ms. Trace:
I20250903 20:05:40.300319 27059 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_0a4db1d67753479197d68a85fb3d6c5c [id=7d6d5a3d8906427e83ab3153b7bb4235] alter complete (version 1)
I20250903 20:05:40.300344 27684 rpcz_store.cc:276] 0903 20:05:36.944951 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.945014 (+ 63us) service_pool.cc:225] Handling call
0903 20:05:40.300111 (+3355097us) ts_tablet_manager.cc:1145] Deleting tablet 4978cd831361454d86b23c1a52ab1c3b
0903 20:05:40.300124 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.300237 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3354866}
I20250903 20:05:40.300627 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1207) took 3347 ms. Trace:
I20250903 20:05:40.300688 27684 rpcz_store.cc:276] 0903 20:05:36.953439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.953501 (+ 62us) service_pool.cc:225] Handling call
0903 20:05:40.300542 (+3347041us) ts_tablet_manager.cc:1145] Deleting tablet 04094bb7dd324b53a865980c53d4a17a
0903 20:05:40.300550 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.300616 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3346817}
I20250903 20:05:40.300959 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1208) took 3346 ms. Trace:
I20250903 20:05:40.301020 27684 rpcz_store.cc:276] 0903 20:05:36.954443 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:36.954596 (+ 153us) service_pool.cc:225] Handling call
0903 20:05:40.300862 (+3346266us) ts_tablet_manager.cc:1145] Deleting tablet 32164b4e88b94e1fbca2e714e6bddfc4
0903 20:05:40.300871 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.300944 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3346114}
I20250903 20:05:40.301260 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 6800e0b6269045f29682704fd847d5d1 (table default.Table_6656b728e12a482fa9e7893ef4842ac7 [id=6788b524a8654e34a4d9ee8cc8fc84d3]) successfully deleted
I20250903 20:05:40.301334 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1212) took 3124 ms. Trace:
I20250903 20:05:40.301391 27684 rpcz_store.cc:276] 0903 20:05:37.176763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:37.176823 (+ 60us) service_pool.cc:225] Handling call
0903 20:05:40.301222 (+3124399us) ts_tablet_manager.cc:1145] Deleting tablet 5bff3d77088c4b69a90365950641e300
0903 20:05:40.301230 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.301320 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3124151}
W20250903 20:05:40.301523 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 4978cd831361454d86b23c1a52ab1c3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4978cd831361454d86b23c1a52ab1c3b already in progress: creating tablet
I20250903 20:05:40.301654 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1213) took 3124 ms. Trace:
W20250903 20:05:40.301735 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 04094bb7dd324b53a865980c53d4a17a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04094bb7dd324b53a865980c53d4a17a already in progress: creating tablet
I20250903 20:05:40.301713 27684 rpcz_store.cc:276] 0903 20:05:37.177274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:37.177319 (+ 45us) service_pool.cc:225] Handling call
0903 20:05:40.301568 (+3124249us) ts_tablet_manager.cc:1145] Deleting tablet cc71346bf84f4a708bf17b5aba174582
0903 20:05:40.301576 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.301640 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3124045}
I20250903 20:05:40.302028 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1217) took 2873 ms. Trace:
W20250903 20:05:40.302245 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 32164b4e88b94e1fbca2e714e6bddfc4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32164b4e88b94e1fbca2e714e6bddfc4 already in progress: creating tablet
W20250903 20:05:40.302484 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 5bff3d77088c4b69a90365950641e300 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bff3d77088c4b69a90365950641e300 already in progress: creating tablet
W20250903 20:05:40.302693 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet cc71346bf84f4a708bf17b5aba174582 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc71346bf84f4a708bf17b5aba174582 already in progress: creating tablet
I20250903 20:05:40.306387 27687 log.cc:1199] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/06a79706f1184fa49d10182ba669b0ab
I20250903 20:05:40.306973 27687 ts_tablet_manager.cc:1939] T 06a79706f1184fa49d10182ba669b0ab P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:40.311347 29703 tablet.cc:1722] T 7345305820be4d32a3ceb227f045f009 P e72ef1dbcc5143cf9f04a808f101c80f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250903 20:05:40.311736 27510 raft_consensus.cc:2466] T 5b46bb34ac214b1ba23fba1a33f746ac P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:40.312384 27059 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_28a829a9988442049d50e6ae254e6274 [id=b27f08d56c3c43e6a09a465fdda4d3db] alter complete (version 1)
W20250903 20:05:40.314908 27059 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.26.65.67:58355 (request call id 178) took 998 ms (client timeout 1000 ms). Trace:
W20250903 20:05:40.315016 27059 rpcz_store.cc:269] 0903 20:05:39.316542 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:39.473526 (+156984us) service_pool.cc:225] Handling call
0903 20:05:40.314882 (+841356us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:39.773292 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:39.773427 (+ 135us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:39.773434 (+ 7us) write_op.cc:435] Acquired schema lock
0903 20:05:39.773439 (+ 5us) tablet.cc:660] Decoding operations
0903 20:05:39.773550 (+ 111us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:39.773563 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:39.773568 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0903 20:05:39.773642 (+ 74us) tablet.cc:699] Row locks acquired
0903 20:05:39.773644 (+ 2us) write_op.cc:260] PREPARE: finished
0903 20:05:39.773689 (+ 45us) write_op.cc:270] Start()
0903 20:05:39.773731 (+ 42us) write_op.cc:276] Timestamp: P: 1756929939773631 usec, L: 0
0903 20:05:39.773736 (+ 5us) op_driver.cc:348] REPLICATION: starting
0903 20:05:39.773958 (+ 222us) log.cc:844] Serialized 356 byte log entry
0903 20:05:39.841729 (+ 67771us) op_driver.cc:464] REPLICATION: finished
0903 20:05:39.842825 (+ 1096us) write_op.cc:301] APPLY: starting
0903 20:05:39.842835 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:39.842862 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:39.842884 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:39.842922 (+ 38us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:39.842944 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0903 20:05:39.842951 (+ 7us) write_op.cc:312] APPLY: finished
0903 20:05:39.842994 (+ 43us) log.cc:844] Serialized 23 byte log entry
0903 20:05:39.843029 (+ 35us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:39.843040 (+ 11us) write_op.cc:454] Released schema lock
0903 20:05:39.843044 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0903 20:05:39.843100 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:39.843104 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0903 20:05:40.304160 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:40.304316 (+ 156us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:40.304322 (+ 6us) write_op.cc:435] Acquired schema lock
0903 20:05:40.304324 (+ 2us) tablet.cc:660] Decoding operations
0903 20:05:40.304428 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:40.304440 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:40.304445 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0903 20:05:40.304510 (+ 65us) tablet.cc:699] Row locks acquired
0903 20:05:40.304512 (+ 2us) write_op.cc:260] PREPARE: finished
0903 20:05:40.304549 (+ 37us) write_op.cc:270] Start()
0903 20:05:40.304582 (+ 33us) write_op.cc:276] Timestamp: P: 1756929940304495 usec, L: 0
0903 20:05:40.304585 (+ 3us) op_driver.cc:348] REPLICATION: starting
0903 20:05:40.304761 (+ 176us) log.cc:844] Serialized 356 byte log entry
0903 20:05:40.310783 (+ 6022us) op_driver.cc:464] REPLICATION: finished
0903 20:05:40.310915 (+ 132us) write_op.cc:301] APPLY: starting
0903 20:05:40.310937 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:40.310997 (+ 60us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:40.311045 (+ 48us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:40.311121 (+ 76us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:40.311166 (+ 45us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0903 20:05:40.311177 (+ 11us) write_op.cc:312] APPLY: finished
0903 20:05:40.311262 (+ 85us) log.cc:844] Serialized 23 byte log entry
0903 20:05:40.311322 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:40.311339 (+ 17us) write_op.cc:454] Released schema lock
0903 20:05:40.311345 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0903 20:05:40.311415 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:40.311418 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":29,"child_traces":[["op",{"apply.queue_time_us":1066,"apply.run_cpu_time_us":269,"apply.run_wall_time_us":269,"num_ops":1,"prepare.queue_time_us":62,"prepare.run_cpu_time_us":779,"prepare.run_wall_time_us":782,"raft.queue_time_us":78,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":67973}],["op",{"apply.queue_time_us":69,"apply.run_cpu_time_us":495,"apply.run_wall_time_us":494,"num_ops":1,"prepare.queue_time_us":999,"prepare.run_cpu_time_us":932,"prepare.run_wall_time_us":933,"raft.queue_time_us":746,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":221,"replication_time_us":6158}]]}
I20250903 20:05:40.318379 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_07ba1128ccb74ca6994a357e77664c41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:40.318969 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07ba1128ccb74ca6994a357e77664c41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:40.321412 27360 tablet_service.cc:1515] Processing DeleteTablet for tablet 3919f7f95a954f59b68f56d0051c0109 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:19 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:40.321693 27347 tablet_service.cc:1515] Processing DeleteTablet for tablet de20d18d7f264811a0631aa1144aef2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:40.321849 27357 tablet_service.cc:1515] Processing DeleteTablet for tablet ee40566e497740789575275d0cdca0db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:40.322021 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 47c1287bff1145a5b2ab6d14dbd608c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:15 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:40.322170 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet 4739f10128d64e2c986257b1d9deb90f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:40.322330 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet d4c703dafbca4e669dbb18946ffac699 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:40.322449 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet e2441aa3c2c748bba5360259fb663c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:40.332362 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1274) took 2195 ms. Trace:
I20250903 20:05:40.333709 27687 rpcz_store.cc:276] 0903 20:05:38.136920 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.136982 (+ 62us) service_pool.cc:225] Handling call
0903 20:05:40.138156 (+2001174us) ts_tablet_manager.cc:1145] Deleting tablet 06a79706f1184fa49d10182ba669b0ab
0903 20:05:40.138166 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.177444 (+ 39278us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.298815 (+121371us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.332324 (+ 33509us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2000028}
I20250903 20:05:40.302098 27684 rpcz_store.cc:276] 0903 20:05:37.428761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:37.428931 (+ 170us) service_pool.cc:225] Handling call
0903 20:05:40.301904 (+2872973us) ts_tablet_manager.cc:1145] Deleting tablet 4a4198b4841b4c2585e6c2f5c6a655d4
0903 20:05:40.301914 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.302013 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2845702}
I20250903 20:05:40.335783 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 06a79706f1184fa49d10182ba669b0ab (table default.Table_1153bf74b285448b88b364efe2169784 [id=f321a32c1ac043238c2fc92bfd5ec1ae]) successfully deleted
I20250903 20:05:40.336310 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1252) took 2598 ms. Trace:
I20250903 20:05:40.336828 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1276) took 2150 ms. Trace:
I20250903 20:05:40.337232 27684 rpcz_store.cc:276] 0903 20:05:37.738152 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:37.743671 (+ 5519us) service_pool.cc:225] Handling call
0903 20:05:40.336175 (+2592504us) ts_tablet_manager.cc:1145] Deleting tablet 09fa32312c9648fbbd6fd327e9738066
0903 20:05:40.336188 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.336284 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2592265}
I20250903 20:05:40.337980 27687 rpcz_store.cc:276] 0903 20:05:38.186680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.186759 (+ 79us) service_pool.cc:225] Handling call
0903 20:05:40.336695 (+2149936us) ts_tablet_manager.cc:1145] Deleting tablet ed096b452ba747f89e91edc15e6e3ac1
0903 20:05:40.336711 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.336803 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2149701}
I20250903 20:05:40.338742 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1253) took 2571 ms. Trace:
W20250903 20:05:40.339236 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet ed096b452ba747f89e91edc15e6e3ac1 because the tablet was not found. No further retry: Not found: Tablet not found: ed096b452ba747f89e91edc15e6e3ac1
I20250903 20:05:40.340104 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1277) took 2152 ms. Trace:
I20250903 20:05:40.340261 27684 rpcz_store.cc:276] 0903 20:05:37.767215 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:37.767309 (+ 94us) service_pool.cc:225] Handling call
0903 20:05:40.338623 (+2571314us) ts_tablet_manager.cc:1145] Deleting tablet 503dba32175c4459892c62d3f80693fb
0903 20:05:40.338634 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.338727 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2570972}
I20250903 20:05:40.340927 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_bc9263f60c774a0a83de0c0f785c42b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:40.342314 27687 rpcz_store.cc:276] 0903 20:05:38.188047 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.188122 (+ 75us) service_pool.cc:225] Handling call
0903 20:05:40.339611 (+2151489us) ts_tablet_manager.cc:1145] Deleting tablet 24eadc70e10d4d02bcb643d45c9e0e36
0903 20:05:40.339623 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.340084 (+ 461us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2151199}
W20250903 20:05:40.342603 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc9263f60c774a0a83de0c0f785c42b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:40.343429 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1254) took 2572 ms. Trace:
W20250903 20:05:40.344061 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 24eadc70e10d4d02bcb643d45c9e0e36 because the tablet was not found. No further retry: Not found: Tablet not found: 24eadc70e10d4d02bcb643d45c9e0e36
I20250903 20:05:40.345083 27687 tablet_replica.cc:331] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:40.345371 27684 rpcz_store.cc:276] 0903 20:05:37.771043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:37.771094 (+ 51us) service_pool.cc:225] Handling call
0903 20:05:40.343259 (+2572165us) ts_tablet_manager.cc:1145] Deleting tablet e4ec4cc1ef914cffa2d304cdf701cf33
0903 20:05:40.343275 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.343407 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2571745}
I20250903 20:05:40.345983 27687 raft_consensus.cc:2241] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:40.347533 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1255) took 2574 ms. Trace:
I20250903 20:05:40.348215 27687 raft_consensus.cc:2270] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:40.348685 27684 rpcz_store.cc:276] 0903 20:05:37.772683 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:37.772733 (+ 50us) service_pool.cc:225] Handling call
0903 20:05:40.347342 (+2574609us) ts_tablet_manager.cc:1145] Deleting tablet 74eab6ad2fe64b18bd00c0d5524428c6
0903 20:05:40.347359 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.347511 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2574320}
I20250903 20:05:40.349920 27687 ts_tablet_manager.cc:1905] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:40.351516 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet c0343d2226e345669d42375de2545986
I20250903 20:05:40.352065 29461 raft_consensus.cc:491] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:40.352247 29461 raft_consensus.cc:513] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:40.352633 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_48a0a980c12f40128907043e0ad8a7a8" } new_table_name: "default.Table_2329720f2aaa431fb0a3ad5faf6d62ac" modify_external_catalogs: true
I20250903 20:05:40.353050 29461 leader_election.cc:290] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:40.354707 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1256) took 2576 ms. Trace:
I20250903 20:05:40.355207 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet c0343d2226e345669d42375de2545986 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:40 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:40.356226 29461 raft_consensus.cc:491] T 13496efe34944b7280e7a684f180e6e0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:40.356139 27684 rpcz_store.cc:276] 0903 20:05:37.778010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:37.778082 (+ 72us) service_pool.cc:225] Handling call
0903 20:05:40.354556 (+2576474us) ts_tablet_manager.cc:1145] Deleting tablet d7d37849132d4420bb5fcb57c11cfb6b
0903 20:05:40.354568 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.354687 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2576173}
I20250903 20:05:40.356400 29461 raft_consensus.cc:513] T 13496efe34944b7280e7a684f180e6e0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:40.357084 29461 leader_election.cc:290] T 13496efe34944b7280e7a684f180e6e0 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:40.354348 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d4b8d4bf29e408cbdf9557c8b0d8a4b" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:40.354768 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet c0343d2226e345669d42375de2545986 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:40 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:40.358816 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d4b8d4bf29e408cbdf9557c8b0d8a4b" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
W20250903 20:05:40.358871 27183 leader_election.cc:343] T 13496efe34944b7280e7a684f180e6e0 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 13496efe34944b7280e7a684f180e6e0
I20250903 20:05:40.358879 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13496efe34944b7280e7a684f180e6e0" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:40.359138 27379 raft_consensus.cc:2466] T 13496efe34944b7280e7a684f180e6e0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:40.359622 27184 leader_election.cc:304] T 13496efe34944b7280e7a684f180e6e0 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:40.359895 27516 raft_consensus.cc:2391] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89f8f0ee68b940db9ce250c0604f934a in current term 1: Already voted for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in this term.
I20250903 20:05:40.360262 27183 leader_election.cc:304] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:40.360426 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 0fef9aa7153a4902abce522e3ff0df3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:40.360608 27216 tablet_service.cc:1515] Processing DeleteTablet for tablet f1d22e76ea72441db51f7e9db26eb520 with delete_type TABLET_DATA_DELETED (replaced by tablet 069a9e21c417467fa68bac210dd64e4f) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:40.360651 29411 raft_consensus.cc:2802] T 13496efe34944b7280e7a684f180e6e0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:40.360713 29411 raft_consensus.cc:491] T 13496efe34944b7280e7a684f180e6e0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:40.360764 29461 raft_consensus.cc:3058] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:40.360780 29411 raft_consensus.cc:3058] T 13496efe34944b7280e7a684f180e6e0 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:40.361516 27216 tablet_service.cc:1515] Processing DeleteTablet for tablet 615f42e95bb24c8480f73d371fd63b9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:40.362955 27216 tablet_service.cc:1515] Processing DeleteTablet for tablet 39c2ebe4ada5426590ec3769c8662644 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c772ab7c50c4994aacde5a05bb0f08e) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:40.363437 29461 raft_consensus.cc:2747] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:40.363368 29411 raft_consensus.cc:513] T 13496efe34944b7280e7a684f180e6e0 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:40.363786 27216 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bf570855b204b96acd070587c434463 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:40.363986 29411 leader_election.cc:290] T 13496efe34944b7280e7a684f180e6e0 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:40.359706 27692 tablet_bootstrap.cc:492] T 54602d7f723d4c10af12408a43ed1f02 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:40.364758 27216 tablet_service.cc:1515] Processing DeleteTablet for tablet 4739f10128d64e2c986257b1d9deb90f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:40.364881 27692 ts_tablet_manager.cc:1397] T 54602d7f723d4c10af12408a43ed1f02 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.137s user 0.001s sys 0.009s
W20250903 20:05:40.365243 27183 leader_election.cc:343] T 13496efe34944b7280e7a684f180e6e0 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 13496efe34944b7280e7a684f180e6e0
I20250903 20:05:40.365882 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13496efe34944b7280e7a684f180e6e0" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:40.366123 27379 raft_consensus.cc:3058] T 13496efe34944b7280e7a684f180e6e0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:40.366007 27064 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_c0a41c4a69a84977906c16593147161d" } new_table_name: "default.Table_847fdaca757242839ea3be30b2111387" modify_external_catalogs: true
W20250903 20:05:40.368515 27183 consensus_peers.cc:489] T e2441aa3c2c748bba5360259fb663c81 P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e2441aa3c2c748bba5360259fb663c81. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:40.316071 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_ea61d93e87e8453f86d0161425e68876"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:40.368958 27379 raft_consensus.cc:2466] T 13496efe34944b7280e7a684f180e6e0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:40.369369 27184 leader_election.cc:304] T 13496efe34944b7280e7a684f180e6e0 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters: e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:40.369764 29411 raft_consensus.cc:2802] T 13496efe34944b7280e7a684f180e6e0 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
W20250903 20:05:40.369804 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea61d93e87e8453f86d0161425e68876 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:40.369876 29411 raft_consensus.cc:695] T 13496efe34944b7280e7a684f180e6e0 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:40.370105 29411 consensus_queue.cc:237] T 13496efe34944b7280e7a684f180e6e0 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:40.370314 27692 raft_consensus.cc:357] T 54602d7f723d4c10af12408a43ed1f02 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.370479 27692 raft_consensus.cc:383] T 54602d7f723d4c10af12408a43ed1f02 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:40.370532 27692 raft_consensus.cc:738] T 54602d7f723d4c10af12408a43ed1f02 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:40.370723 27692 consensus_queue.cc:260] T 54602d7f723d4c10af12408a43ed1f02 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.371397 27692 ts_tablet_manager.cc:1428] T 54602d7f723d4c10af12408a43ed1f02 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250903 20:05:40.371886 27692 tablet_bootstrap.cc:492] T 5132e6d4288c447e80f0608ba366b69c P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:40.373697 27692 tablet_bootstrap.cc:654] T 5132e6d4288c447e80f0608ba366b69c P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:40.374498 27685 ts_tablet_manager.cc:1918] T 886b8783aa9d4618aad498d4a1115843 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:40.379882 29589 consensus_queue.cc:1035] T de20d18d7f264811a0631aa1144aef2c P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:40.390583 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1257) took 2511 ms. Trace:
I20250903 20:05:40.390861 27684 rpcz_store.cc:276] 0903 20:05:37.878996 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:37.879095 (+ 99us) service_pool.cc:225] Handling call
0903 20:05:40.373136 (+2494041us) ts_tablet_manager.cc:1145] Deleting tablet e9b5ab312f8744af99d557c0733bf71d
0903 20:05:40.380963 (+ 7827us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.390548 (+ 9585us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2493842}
I20250903 20:05:40.391533 27684 tablet_replica.cc:331] T 259bc73ca9df47e9874692725a60bb1c P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:40.391803 27684 raft_consensus.cc:2241] T 259bc73ca9df47e9874692725a60bb1c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:40.392062 27684 raft_consensus.cc:2270] T 259bc73ca9df47e9874692725a60bb1c P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:40.392243 27685 log.cc:1199] T 886b8783aa9d4618aad498d4a1115843 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/886b8783aa9d4618aad498d4a1115843
I20250903 20:05:40.392817 27684 ts_tablet_manager.cc:1905] T 259bc73ca9df47e9874692725a60bb1c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:40.393451 27362 tablet_service.cc:1468] Processing CreateTablet for tablet f268cdd43a4048ddbfeb7a2ec87fcb03 (DEFAULT_TABLE table=default.Table_7e439005e9d4471e94e67a047e287b3c [id=26e3b9eecf454a87a8159eb79641e807]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.393816 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f268cdd43a4048ddbfeb7a2ec87fcb03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.395006 27685 ts_tablet_manager.cc:1939] T 886b8783aa9d4618aad498d4a1115843 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:40.392251 27692 tablet_bootstrap.cc:492] T 5132e6d4288c447e80f0608ba366b69c P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:40.395835 27692 ts_tablet_manager.cc:1397] T 5132e6d4288c447e80f0608ba366b69c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.000s
I20250903 20:05:40.395910 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1063) took 5843 ms. Trace:
I20250903 20:05:40.395980 27685 rpcz_store.cc:276] 0903 20:05:34.552980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.555404 (+ 2424us) service_pool.cc:225] Handling call
0903 20:05:40.266043 (+5710639us) ts_tablet_manager.cc:1145] Deleting tablet 886b8783aa9d4618aad498d4a1115843
0903 20:05:40.266060 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.304961 (+ 38901us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.374394 (+ 69433us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.395889 (+ 21495us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5710353}
I20250903 20:05:40.396528 27685 tablet_replica.cc:331] T 950a6fcd35f64f2d889831fe9cdf503a P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:40.396603 27692 raft_consensus.cc:357] T 5132e6d4288c447e80f0608ba366b69c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.396714 27685 raft_consensus.cc:2241] T 950a6fcd35f64f2d889831fe9cdf503a P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:40.396755 27692 raft_consensus.cc:383] T 5132e6d4288c447e80f0608ba366b69c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:40.396802 27692 raft_consensus.cc:738] T 5132e6d4288c447e80f0608ba366b69c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:40.396873 27685 raft_consensus.cc:2270] T 950a6fcd35f64f2d889831fe9cdf503a P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:40.397511 27685 ts_tablet_manager.cc:1905] T 950a6fcd35f64f2d889831fe9cdf503a P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:40.398666 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 886b8783aa9d4618aad498d4a1115843 (table default.Table_a540eea45dfb46ec92b02a3f9a2b3aed [id=5fe055478b724744a65fcaa2f25bef0d]) successfully deleted
I20250903 20:05:40.399453 27692 consensus_queue.cc:260] T 5132e6d4288c447e80f0608ba366b69c P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.402031 27692 ts_tablet_manager.cc:1428] T 5132e6d4288c447e80f0608ba366b69c P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250903 20:05:40.405776 27692 tablet_bootstrap.cc:492] T 112c0dccb60c4c7c9efc164d9c8443ea P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
W20250903 20:05:40.406708 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1ed519e2e9c94fa3a134dcfc8d5364e6 consensus state has no leader
I20250903 20:05:40.407300 27692 tablet_bootstrap.cc:654] T 112c0dccb60c4c7c9efc164d9c8443ea P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:40.407186 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b75452e91c004d2380f6a8ee068eb26f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:40.407766 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b75452e91c004d2380f6a8ee068eb26f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:40.410241 27695 tablet_bootstrap.cc:700] T 314856bf269f4df7aa25f5a30cec0d5b P 89f8f0ee68b940db9ce250c0604f934a: Time spent opening tablet: real 0.122s user 0.000s sys 0.001s
I20250903 20:05:40.410815 27695 tablet_bootstrap.cc:654] T 314856bf269f4df7aa25f5a30cec0d5b P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:40.411882 27183 consensus_peers.cc:489] T de20d18d7f264811a0631aa1144aef2c P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de20d18d7f264811a0631aa1144aef2c. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:40.420966 27684 ts_tablet_manager.cc:1918] T 259bc73ca9df47e9874692725a60bb1c P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:40.421326 27684 log.cc:1199] T 259bc73ca9df47e9874692725a60bb1c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/259bc73ca9df47e9874692725a60bb1c
I20250903 20:05:40.421877 27684 ts_tablet_manager.cc:1939] T 259bc73ca9df47e9874692725a60bb1c P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:40.423135 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1258) took 2354 ms. Trace:
I20250903 20:05:40.424165 27684 rpcz_store.cc:276] 0903 20:05:38.068435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.068494 (+ 59us) service_pool.cc:225] Handling call
0903 20:05:40.391478 (+2322984us) ts_tablet_manager.cc:1145] Deleting tablet 259bc73ca9df47e9874692725a60bb1c
0903 20:05:40.391490 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.395944 (+ 4454us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.420585 (+ 24641us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.423115 (+ 2530us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2321140}
I20250903 20:05:40.425098 27684 tablet_replica.cc:331] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:40.425585 27684 raft_consensus.cc:2241] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:40.428575 27684 raft_consensus.cc:2270] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:40.429481 27684 ts_tablet_manager.cc:1905] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:40.434544 27687 ts_tablet_manager.cc:1918] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:40.442181 27684 ts_tablet_manager.cc:1918] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:40.442984 27687 log.cc:1199] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/39c2ebe4ada5426590ec3769c8662644
I20250903 20:05:40.443398 27684 log.cc:1199] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/4e9a6a6e05464b7b97ae624f51402cf0
I20250903 20:05:40.443454 27687 ts_tablet_manager.cc:1939] T 39c2ebe4ada5426590ec3769c8662644 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:40.443894 29612 raft_consensus.cc:491] T 314856bf269f4df7aa25f5a30cec0d5b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:40.444033 29612 raft_consensus.cc:513] T 314856bf269f4df7aa25f5a30cec0d5b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.444254 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1278) took 2255 ms. Trace:
I20250903 20:05:40.444339 27687 rpcz_store.cc:276] 0903 20:05:38.188897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.189013 (+ 116us) service_pool.cc:225] Handling call
0903 20:05:40.345007 (+2155994us) ts_tablet_manager.cc:1145] Deleting tablet 39c2ebe4ada5426590ec3769c8662644
0903 20:05:40.345024 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.384603 (+ 39579us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.434439 (+ 49836us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.444223 (+ 9784us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2155760}
I20250903 20:05:40.444684 29612 leader_election.cc:290] T 314856bf269f4df7aa25f5a30cec0d5b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:40.445698 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "314856bf269f4df7aa25f5a30cec0d5b" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:40.445778 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 39c2ebe4ada5426590ec3769c8662644 (table default.Table_d84f067d5a924a23964078d360422878 [id=5cb314181f104d63b1aed98c7198d5c3]) successfully deleted
W20250903 20:05:40.446329 27462 leader_election.cc:343] T 314856bf269f4df7aa25f5a30cec0d5b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.446659 27687 tablet_replica.cc:331] T 30cc0d4ba8d444f1b98232f906ea0c7b P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:40.446691 27692 tablet_bootstrap.cc:492] T 112c0dccb60c4c7c9efc164d9c8443ea P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:40.446813 27692 ts_tablet_manager.cc:1397] T 112c0dccb60c4c7c9efc164d9c8443ea P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.000s
I20250903 20:05:40.446614 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "314856bf269f4df7aa25f5a30cec0d5b" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:40.447117 27684 ts_tablet_manager.cc:1939] T 4e9a6a6e05464b7b97ae624f51402cf0 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
W20250903 20:05:40.447352 27462 leader_election.cc:343] T 314856bf269f4df7aa25f5a30cec0d5b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.447587 27462 leader_election.cc:304] T 314856bf269f4df7aa25f5a30cec0d5b P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:40.447638 27687 raft_consensus.cc:2241] T 30cc0d4ba8d444f1b98232f906ea0c7b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:40.448172 29612 raft_consensus.cc:2747] T 314856bf269f4df7aa25f5a30cec0d5b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:40.449052 29411 raft_consensus.cc:491] T 24eadc70e10d4d02bcb643d45c9e0e36 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51578fb7bceb4e119b3ab1e58c4f5528)
I20250903 20:05:40.449201 29411 raft_consensus.cc:513] T 24eadc70e10d4d02bcb643d45c9e0e36 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.452801 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet f27e2aa2d0bf4de48429a030146df3d8
I20250903 20:05:40.452879 27692 raft_consensus.cc:357] T 112c0dccb60c4c7c9efc164d9c8443ea P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.453065 27692 raft_consensus.cc:383] T 112c0dccb60c4c7c9efc164d9c8443ea P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:40.453110 27692 raft_consensus.cc:738] T 112c0dccb60c4c7c9efc164d9c8443ea P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:40.453306 27057 catalog_manager.cc:6652] ReplaceTablet: tablet f27e2aa2d0bf4de48429a030146df3d8 deleted and replaced by tablet 7578beec937e447091ddf7f1d4a793d2
I20250903 20:05:40.453653 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_6ab10618f71c48d09717ebef2a298bc5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:40.454202 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ab10618f71c48d09717ebef2a298bc5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:40.454299 27692 consensus_queue.cc:260] T 112c0dccb60c4c7c9efc164d9c8443ea P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.455474 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 7dc5214512af449bbc537f111a3d8dbc
I20250903 20:05:40.455895 27058 catalog_manager.cc:6652] ReplaceTablet: tablet 7dc5214512af449bbc537f111a3d8dbc deleted and replaced by tablet 9061b68d0e144450b7efa8670f408817
I20250903 20:05:40.456205 27058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_f3ebcbafbb374a6487d0411a0cbbf5c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:40.456722 27058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3ebcbafbb374a6487d0411a0cbbf5c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:40.457293 27692 ts_tablet_manager.cc:1428] T 112c0dccb60c4c7c9efc164d9c8443ea P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20250903 20:05:40.457634 27692 tablet_bootstrap.cc:492] T 7cbbeb8bec5e464b85b8568e916c765f P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:40.457901 27063 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet bf35e907252244118b8ee9e99e0f0415
I20250903 20:05:40.458601 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 4ae0446b76d041c98fa76dfeadab5276
I20250903 20:05:40.458962 27061 catalog_manager.cc:6652] ReplaceTablet: tablet 4ae0446b76d041c98fa76dfeadab5276 deleted and replaced by tablet bba2c6d742b84af088b3422073c89004
I20250903 20:05:40.460906 27361 tablet_service.cc:1515] Processing DeleteTablet for tablet 7dc5214512af449bbc537f111a3d8dbc with delete_type TABLET_DATA_DELETED (replaced by tablet 9061b68d0e144450b7efa8670f408817) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:40.461094 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet bf35e907252244118b8ee9e99e0f0415 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:40 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:40.461334 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ae0446b76d041c98fa76dfeadab5276 with delete_type TABLET_DATA_DELETED (replaced by tablet bba2c6d742b84af088b3422073c89004) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:40.461422 27687 raft_consensus.cc:2270] T 30cc0d4ba8d444f1b98232f906ea0c7b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:40.461930 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 7dc5214512af449bbc537f111a3d8dbc with delete_type TABLET_DATA_DELETED (replaced by tablet 9061b68d0e144450b7efa8670f408817) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:40.462136 27216 tablet_service.cc:1515] Processing DeleteTablet for tablet f27e2aa2d0bf4de48429a030146df3d8 with delete_type TABLET_DATA_DELETED (replaced by tablet 7578beec937e447091ddf7f1d4a793d2) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:40.462409 27210 tablet_service.cc:1515] Processing DeleteTablet for tablet bf35e907252244118b8ee9e99e0f0415 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:40 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:40.462554 27229 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ae0446b76d041c98fa76dfeadab5276 with delete_type TABLET_DATA_DELETED (replaced by tablet bba2c6d742b84af088b3422073c89004) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:40.463658 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 6d614ed6d2fe4217bc2d289c12554efa
I20250903 20:05:40.464156 27687 ts_tablet_manager.cc:1905] T 30cc0d4ba8d444f1b98232f906ea0c7b P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:40.464879 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d614ed6d2fe4217bc2d289c12554efa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:40 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:40.465411 27229 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d614ed6d2fe4217bc2d289c12554efa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:40 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:40.465835 27685 ts_tablet_manager.cc:1918] T 950a6fcd35f64f2d889831fe9cdf503a P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:40.466187 27685 log.cc:1199] T 950a6fcd35f64f2d889831fe9cdf503a P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/950a6fcd35f64f2d889831fe9cdf503a
I20250903 20:05:40.467526 27692 tablet_bootstrap.cc:654] T 7cbbeb8bec5e464b85b8568e916c765f P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:40.468044 27685 ts_tablet_manager.cc:1939] T 950a6fcd35f64f2d889831fe9cdf503a P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:40.469040 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1065) took 5906 ms. Trace:
I20250903 20:05:40.472975 27066 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet e053b613eefc4ea999859847c84b8a3c
I20250903 20:05:40.473177 27685 rpcz_store.cc:276] 0903 20:05:34.562187 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.562283 (+ 96us) service_pool.cc:225] Handling call
0903 20:05:40.396472 (+5834189us) ts_tablet_manager.cc:1145] Deleting tablet 950a6fcd35f64f2d889831fe9cdf503a
0903 20:05:40.396482 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.429821 (+ 33339us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.465715 (+ 35894us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.469008 (+ 3293us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5833990}
I20250903 20:05:40.473941 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1064) took 5912 ms. Trace:
I20250903 20:05:40.473939 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 950a6fcd35f64f2d889831fe9cdf503a (table default.Table_7dae8ff5ca9744708d223312079a4315 [id=7032fe3b789041d380f5db114f241e29]) successfully deleted
I20250903 20:05:40.474100 27685 rpcz_store.cc:276] 0903 20:05:34.561856 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.563698 (+ 1842us) service_pool.cc:225] Handling call
0903 20:05:40.473797 (+5910099us) ts_tablet_manager.cc:1145] Deleting tablet a785adcd0bd943fc97b201b6d29073f4
0903 20:05:40.473809 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.473927 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5909917}
I20250903 20:05:40.474658 27365 tablet_service.cc:1515] Processing DeleteTablet for tablet e053b613eefc4ea999859847c84b8a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:40 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:40.474994 27685 tablet_replica.cc:331] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
W20250903 20:05:40.475761 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet a785adcd0bd943fc97b201b6d29073f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a785adcd0bd943fc97b201b6d29073f4 already in progress: creating tablet
I20250903 20:05:40.475291 27229 tablet_service.cc:1515] Processing DeleteTablet for tablet e053b613eefc4ea999859847c84b8a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:40 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:40.476198 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1259) took 2407 ms. Trace:
I20250903 20:05:40.476454 27685 raft_consensus.cc:2241] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:40.476882 27685 pending_rounds.cc:70] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a: Trying to abort 1 pending ops.
I20250903 20:05:40.476888 27684 rpcz_store.cc:276] 0903 20:05:38.069099 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.069150 (+ 51us) service_pool.cc:225] Handling call
0903 20:05:40.425043 (+2355893us) ts_tablet_manager.cc:1145] Deleting tablet 4e9a6a6e05464b7b97ae624f51402cf0
0903 20:05:40.425056 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.433263 (+ 8207us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.442064 (+ 8801us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.476157 (+ 34093us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2354625}
I20250903 20:05:40.478281 27685 pending_rounds.cc:77] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196385032793444352 op_type: NO_OP noop_request { }
I20250903 20:05:40.478538 27685 raft_consensus.cc:2887] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 20:05:40.478742 27685 raft_consensus.cc:2270] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:40.479446 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet f27e2aa2d0bf4de48429a030146df3d8 with delete_type TABLET_DATA_DELETED (replaced by tablet 7578beec937e447091ddf7f1d4a793d2) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:40.479511 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1260) took 2409 ms. Trace:
I20250903 20:05:40.479583 27684 rpcz_store.cc:276] 0903 20:05:38.069684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.069738 (+ 54us) service_pool.cc:225] Handling call
0903 20:05:40.479313 (+2409575us) ts_tablet_manager.cc:1145] Deleting tablet 4e34d6d5d68e4395aed78821ea077c1b
0903 20:05:40.479327 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.479490 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2408862}
I20250903 20:05:40.480115 27685 ts_tablet_manager.cc:1905] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:40.480521 27064 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9ac48450b5aa448f9f040e88dc2f98bd consensus state has no leader
I20250903 20:05:40.485729 27684 tablet_replica.cc:331] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
W20250903 20:05:40.486030 27184 leader_election.cc:343] T 24eadc70e10d4d02bcb643d45c9e0e36 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 24eadc70e10d4d02bcb643d45c9e0e36
I20250903 20:05:40.486271 29461 raft_consensus.cc:491] T ed096b452ba747f89e91edc15e6e3ac1 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51578fb7bceb4e119b3ab1e58c4f5528)
I20250903 20:05:40.486415 29461 raft_consensus.cc:513] T ed096b452ba747f89e91edc15e6e3ac1 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:40.486622 26890 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 73140.8us
I20250903 20:05:40.487104 29461 leader_election.cc:290] T ed096b452ba747f89e91edc15e6e3ac1 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:40.488926 27229 tablet_service.cc:1468] Processing CreateTablet for tablet f268cdd43a4048ddbfeb7a2ec87fcb03 (DEFAULT_TABLE table=default.Table_7e439005e9d4471e94e67a047e287b3c [id=26e3b9eecf454a87a8159eb79641e807]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:40.489204 27183 leader_election.cc:343] T ed096b452ba747f89e91edc15e6e3ac1 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: ed096b452ba747f89e91edc15e6e3ac1
I20250903 20:05:40.489176 27210 tablet_service.cc:1468] Processing CreateTablet for tablet 3e45128d635f442fbdfb32531be2256b (DEFAULT_TABLE table=default.Table_1dc6317976b84af1b853bbb3825bcf82 [id=3e562137069c499ab6b0a0e3c3a8b14d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.489394 27229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f268cdd43a4048ddbfeb7a2ec87fcb03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.489571 27210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e45128d635f442fbdfb32531be2256b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.491170 27693 tablet_bootstrap.cc:492] T f4dec89fc6bb436ba6542c267eaf9c15 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:40.491320 27693 ts_tablet_manager.cc:1397] T f4dec89fc6bb436ba6542c267eaf9c15 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.291s user 0.014s sys 0.058s
I20250903 20:05:40.492112 27693 raft_consensus.cc:357] T f4dec89fc6bb436ba6542c267eaf9c15 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.492280 27693 raft_consensus.cc:383] T f4dec89fc6bb436ba6542c267eaf9c15 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:40.492529 27693 raft_consensus.cc:738] T f4dec89fc6bb436ba6542c267eaf9c15 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:40.492738 27693 consensus_queue.cc:260] T f4dec89fc6bb436ba6542c267eaf9c15 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.494115 27693 ts_tablet_manager.cc:1428] T f4dec89fc6bb436ba6542c267eaf9c15 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250903 20:05:40.494415 27684 raft_consensus.cc:2241] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:40.495292 29452 raft_consensus.cc:491] T eaa11d17303142ff935705c16f0b9690 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e72ef1dbcc5143cf9f04a808f101c80f)
I20250903 20:05:40.495435 29452 raft_consensus.cc:513] T eaa11d17303142ff935705c16f0b9690 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.495958 29452 leader_election.cc:290] T eaa11d17303142ff935705c16f0b9690 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
W20250903 20:05:40.497128 27183 leader_election.cc:343] T eaa11d17303142ff935705c16f0b9690 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: eaa11d17303142ff935705c16f0b9690
I20250903 20:05:40.499831 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 36e1061915574407b6e7695be8b1e55a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:21 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:40.500568 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 64c4dd2645034a7eaf243f5474603f10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:40.501139 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c42d84a6a5649208b0539a10da21aec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:40.500909 27065 catalog_manager.cc:5582] T 5b46bb34ac214b1ba23fba1a33f746ac P 89f8f0ee68b940db9ce250c0604f934a reported cstate change: term changed from 0 to 1, leader changed from <none> to 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65). New cstate: current_term: 1 leader_uuid: "89f8f0ee68b940db9ce250c0604f934a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:40.501349 27065 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 5b46bb34ac214b1ba23fba1a33f746ac (table default.Table_975e53d37e054938a5b63957252ff50d [id=f261e84232234244824d0232efbb4399]). Expected version 1 got 0
I20250903 20:05:40.501686 27065 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet f4d3e1cfdfab4be1a9143b78bf3bc46d (table default.Table_5e672ed0c8c142029a79b72f799f3f59 [id=386b8d4e7c4042d1947c19ac464c75a0]). Expected version 2 got 0
I20250903 20:05:40.501811 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 659a127405d341c7b074d3445854256f with delete_type TABLET_DATA_DELETED (replaced by tablet 443d77f2593b44afa3e5aae4c5b0a651) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:40.502404 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 3f303233e6c347289650c6cf83f8127f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:31 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:40.502506 29669 consensus_queue.cc:1035] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:40.502971 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 733f8024e1fa401ca9614a97db91a19b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:27 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:40.503458 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet ef02f91c0964446daf62ac1dde0330ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:40.503932 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 4739f10128d64e2c986257b1d9deb90f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:40.504504 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d13884fff8c43b690836c6a341a2d92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:17 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:40.504998 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c77bc0e8bb34712bf8109cadbc0fc06 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:40.505507 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet e2441aa3c2c748bba5360259fb663c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:40.506003 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet f4dec89fc6bb436ba6542c267eaf9c15 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:40.489092 27692 tablet_bootstrap.cc:492] T 7cbbeb8bec5e464b85b8568e916c765f P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:40.506474 27692 ts_tablet_manager.cc:1397] T 7cbbeb8bec5e464b85b8568e916c765f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.000s
I20250903 20:05:40.506713 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 615f42e95bb24c8480f73d371fd63b9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:40.507244 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 5b5be2af29c245c9973fdc2e76304e3f (DEFAULT_TABLE table=default.Table_9b36ebc04fa442909cce88c8e4695c10 [id=48b9220544d647dea63d37a7183e20a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.507436 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 69f8eef52bee431a86f0eef45fd11f9f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:34 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:40.507354 27692 raft_consensus.cc:357] T 7cbbeb8bec5e464b85b8568e916c765f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.507735 27692 raft_consensus.cc:383] T 7cbbeb8bec5e464b85b8568e916c765f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:40.507843 27692 raft_consensus.cc:738] T 7cbbeb8bec5e464b85b8568e916c765f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:40.508137 27692 consensus_queue.cc:260] T 7cbbeb8bec5e464b85b8568e916c765f P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.508613 27210 tablet_service.cc:1468] Processing CreateTablet for tablet 5b5be2af29c245c9973fdc2e76304e3f (DEFAULT_TABLE table=default.Table_9b36ebc04fa442909cce88c8e4695c10 [id=48b9220544d647dea63d37a7183e20a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.509380 27210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b5be2af29c245c9973fdc2e76304e3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.509445 27362 tablet_service.cc:1468] Processing CreateTablet for tablet 3e45128d635f442fbdfb32531be2256b (DEFAULT_TABLE table=default.Table_1dc6317976b84af1b853bbb3825bcf82 [id=3e562137069c499ab6b0a0e3c3a8b14d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.509550 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 39c2ebe4ada5426590ec3769c8662644 with delete_type TABLET_DATA_DELETED (replaced by tablet 2c772ab7c50c4994aacde5a05bb0f08e) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:40.510051 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 5fb2426aa01e47598f268d0a90475271 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:40.510612 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet ed4902c002ef4198802375ced39ef2ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:40.510828 27692 ts_tablet_manager.cc:1428] T 7cbbeb8bec5e464b85b8568e916c765f P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:40.511178 27692 tablet_bootstrap.cc:492] T 878f6ad087614c15bc7e29c784f2c3c9 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:40.512117 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 16549a6d9aa9432eb5cee19f69ce6957 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:32 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:40.512590 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 60c32370a5274347afafc6a7e1e94270 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:40.513862 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet c0343d2226e345669d42375de2545986 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:40 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:40.515704 27692 tablet_bootstrap.cc:654] T 878f6ad087614c15bc7e29c784f2c3c9 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:40.516644 27184 leader_election.cc:343] T ed096b452ba747f89e91edc15e6e3ac1 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: ed096b452ba747f89e91edc15e6e3ac1
I20250903 20:05:40.516793 27184 leader_election.cc:304] T ed096b452ba747f89e91edc15e6e3ac1 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
W20250903 20:05:40.517215 27184 leader_election.cc:343] T eaa11d17303142ff935705c16f0b9690 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: eaa11d17303142ff935705c16f0b9690
I20250903 20:05:40.517318 27184 leader_election.cc:304] T eaa11d17303142ff935705c16f0b9690 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:40.517544 29589 raft_consensus.cc:2747] T ed096b452ba747f89e91edc15e6e3ac1 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:40.517700 29589 raft_consensus.cc:2747] T eaa11d17303142ff935705c16f0b9690 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:40.519304 27693 tablet_bootstrap.cc:492] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:40.521318 27693 tablet_bootstrap.cc:654] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:40.521350 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b5be2af29c245c9973fdc2e76304e3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.525602 27687 ts_tablet_manager.cc:1918] T 30cc0d4ba8d444f1b98232f906ea0c7b P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250903 20:05:40.525663 27183 leader_election.cc:343] T 24eadc70e10d4d02bcb643d45c9e0e36 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 24eadc70e10d4d02bcb643d45c9e0e36
I20250903 20:05:40.525804 27183 leader_election.cc:304] T 24eadc70e10d4d02bcb643d45c9e0e36 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:40.526192 29461 raft_consensus.cc:2747] T 24eadc70e10d4d02bcb643d45c9e0e36 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:40.527204 29411 leader_election.cc:290] T 24eadc70e10d4d02bcb643d45c9e0e36 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:40.519990 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet f27e2aa2d0bf4de48429a030146df3d8 with delete_type TABLET_DATA_DELETED (replaced by tablet 7578beec937e447091ddf7f1d4a793d2) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:40.520412 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 7dc5214512af449bbc537f111a3d8dbc with delete_type TABLET_DATA_DELETED (replaced by tablet 9061b68d0e144450b7efa8670f408817) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:40.521142 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet bf35e907252244118b8ee9e99e0f0415 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:40 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:40.529796 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d614ed6d2fe4217bc2d289c12554efa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:40 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:40.524307 27684 raft_consensus.cc:2270] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:40.527899 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ae0446b76d041c98fa76dfeadab5276 with delete_type TABLET_DATA_DELETED (replaced by tablet bba2c6d742b84af088b3422073c89004) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:40.531279 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet e053b613eefc4ea999859847c84b8a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:40 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:40.531848 27684 ts_tablet_manager.cc:1905] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:40.534770 27695 tablet_bootstrap.cc:492] T 314856bf269f4df7aa25f5a30cec0d5b P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:40.534914 27695 ts_tablet_manager.cc:1397] T 314856bf269f4df7aa25f5a30cec0d5b P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.248s user 0.002s sys 0.001s
I20250903 20:05:40.535696 27695 raft_consensus.cc:357] T 314856bf269f4df7aa25f5a30cec0d5b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.535884 27695 raft_consensus.cc:383] T 314856bf269f4df7aa25f5a30cec0d5b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:40.532964 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e45128d635f442fbdfb32531be2256b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.536042 27687 log.cc:1199] T 30cc0d4ba8d444f1b98232f906ea0c7b P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/30cc0d4ba8d444f1b98232f906ea0c7b
I20250903 20:05:40.536433 27687 ts_tablet_manager.cc:1939] T 30cc0d4ba8d444f1b98232f906ea0c7b P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:40.537178 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1279) took 2346 ms. Trace:
I20250903 20:05:40.537272 27687 rpcz_store.cc:276] 0903 20:05:38.190943 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.191001 (+ 58us) service_pool.cc:225] Handling call
0903 20:05:40.446596 (+2255595us) ts_tablet_manager.cc:1145] Deleting tablet 30cc0d4ba8d444f1b98232f906ea0c7b
0903 20:05:40.446614 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.470900 (+ 24286us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.525506 (+ 54606us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.537157 (+ 11651us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2944,"tablet-delete.queue_time_us":2255421}
I20250903 20:05:40.540508 27497 tablet_service.cc:1468] Processing CreateTablet for tablet f268cdd43a4048ddbfeb7a2ec87fcb03 (DEFAULT_TABLE table=default.Table_7e439005e9d4471e94e67a047e287b3c [id=26e3b9eecf454a87a8159eb79641e807]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.537757 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 30cc0d4ba8d444f1b98232f906ea0c7b (table default.Table_423c3bfc0b8c46bc9f81d7a9013e1468 [id=57668609851c4124b73a3797bb6d2eb8]) successfully deleted
I20250903 20:05:40.540922 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f268cdd43a4048ddbfeb7a2ec87fcb03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.535940 27695 raft_consensus.cc:738] T 314856bf269f4df7aa25f5a30cec0d5b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:40.542228 27359 tablet_service.cc:1468] Processing CreateTablet for tablet a492a9de59844bb5a4c8dfe10bf36e75 (DEFAULT_TABLE table=default.Table_db9c4046409e4dfda92270621470ecd4 [id=68d86c1824fc416ca8bc7e63f7b4b4e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.542614 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a492a9de59844bb5a4c8dfe10bf36e75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.542922 29371 raft_consensus.cc:491] T 5132e6d4288c447e80f0608ba366b69c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:40.543053 29371 raft_consensus.cc:513] T 5132e6d4288c447e80f0608ba366b69c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.543609 29371 leader_election.cc:290] T 5132e6d4288c447e80f0608ba366b69c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:40.544373 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5132e6d4288c447e80f0608ba366b69c" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:40.545153 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5132e6d4288c447e80f0608ba366b69c" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:40.545189 27462 leader_election.cc:343] T 5132e6d4288c447e80f0608ba366b69c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:40.547773 27462 leader_election.cc:343] T 5132e6d4288c447e80f0608ba366b69c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.547940 27462 leader_election.cc:304] T 5132e6d4288c447e80f0608ba366b69c P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:40.549072 29371 raft_consensus.cc:2747] T 5132e6d4288c447e80f0608ba366b69c P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:40.549762 29669 consensus_queue.cc:1035] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:40.554509 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1280) took 2363 ms. Trace:
I20250903 20:05:40.554637 27687 rpcz_store.cc:276] 0903 20:05:38.191426 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.191476 (+ 50us) service_pool.cc:225] Handling call
0903 20:05:40.554387 (+2362911us) ts_tablet_manager.cc:1145] Deleting tablet 238f2f30377b42c2852b016018aa3245
0903 20:05:40.554400 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.554485 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2362745}
I20250903 20:05:40.555234 27229 tablet_service.cc:1468] Processing CreateTablet for tablet 1f1120373c3845d6b8bebd65860ae47a (DEFAULT_TABLE table=default.Table_61fe5d6eecb744fdabbe92ed7320115a [id=de7ce5c3fb4342d182d00f49da5aecc7]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:40.555362 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 4a4198b4841b4c2585e6c2f5c6a655d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a4198b4841b4c2585e6c2f5c6a655d4 already in progress: creating tablet
I20250903 20:05:40.555234 27216 tablet_service.cc:1468] Processing CreateTablet for tablet a492a9de59844bb5a4c8dfe10bf36e75 (DEFAULT_TABLE table=default.Table_db9c4046409e4dfda92270621470ecd4 [id=68d86c1824fc416ca8bc7e63f7b4b4e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.555893 27229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f1120373c3845d6b8bebd65860ae47a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.556166 27216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a492a9de59844bb5a4c8dfe10bf36e75. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:40.556697 27051 catalog_manager.cc:4895] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 09fa32312c9648fbbd6fd327e9738066 because the tablet was not found. No further retry: Not found: Tablet not found: 09fa32312c9648fbbd6fd327e9738066
W20250903 20:05:40.556855 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 503dba32175c4459892c62d3f80693fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 503dba32175c4459892c62d3f80693fb already in progress: creating tablet
W20250903 20:05:40.556984 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet e4ec4cc1ef914cffa2d304cdf701cf33 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4ec4cc1ef914cffa2d304cdf701cf33 already in progress: creating tablet
W20250903 20:05:40.557106 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 74eab6ad2fe64b18bd00c0d5524428c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74eab6ad2fe64b18bd00c0d5524428c6 already in progress: creating tablet
W20250903 20:05:40.557227 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet d7d37849132d4420bb5fcb57c11cfb6b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7d37849132d4420bb5fcb57c11cfb6b already in progress: creating tablet
W20250903 20:05:40.557364 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet e9b5ab312f8744af99d557c0733bf71d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9b5ab312f8744af99d557c0733bf71d already in progress: creating tablet
I20250903 20:05:40.557479 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 259bc73ca9df47e9874692725a60bb1c (table default.Table_8a37ddf23e9041d59958722b82a977f9 [id=60cc385fc1044966b4a12e13fdc7b5bb]) successfully deleted
I20250903 20:05:40.557591 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 4e9a6a6e05464b7b97ae624f51402cf0 (table default.Table_50f6b8731a2c4ccb8bf58597acb5b760 [id=4eecee99ccf149d49f2f95c27b5b05ca]) successfully deleted
W20250903 20:05:40.557705 27051 catalog_manager.cc:4895] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 4e34d6d5d68e4395aed78821ea077c1b because the tablet was not found. No further retry: Not found: Tablet not found: 4e34d6d5d68e4395aed78821ea077c1b
I20250903 20:05:40.565359 27065 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet 6d4b8d4bf29e408cbdf9557c8b0d8a4b (table default.Table_ccdb4130549b4be5888f17097ae35106 [id=cbcfc7067aca4796b39e3eb195f38903]). Expected version 2 got 0
I20250903 20:05:40.566751 27695 consensus_queue.cc:260] T 314856bf269f4df7aa25f5a30cec0d5b P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.567291 27065 catalog_manager.cc:5674] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989) does not have the latest schema for tablet b4b15d77d07c4fa7856d9c198cf15450 (table default.Table_3edadd8954444273b91ea4d39cf3a834 [id=45cd523acd3c45b994d1b9cc6eb569ea]). Expected version 4 got 0
I20250903 20:05:40.568084 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1281) took 2375 ms. Trace:
I20250903 20:05:40.568195 27687 rpcz_store.cc:276] 0903 20:05:38.192296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.192355 (+ 59us) service_pool.cc:225] Handling call
0903 20:05:40.567952 (+2375597us) ts_tablet_manager.cc:1145] Deleting tablet 950a6fcd35f64f2d889831fe9cdf503a
0903 20:05:40.567965 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.568048 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2375439}
I20250903 20:05:40.568550 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1285) took 2250 ms. Trace:
I20250903 20:05:40.568619 27687 rpcz_store.cc:276] 0903 20:05:38.318156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.318263 (+ 107us) service_pool.cc:225] Handling call
0903 20:05:40.568458 (+2250195us) ts_tablet_manager.cc:1145] Deleting tablet a868191630454a1eb3a4301664178a9a
0903 20:05:40.568466 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.568538 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2249986}
I20250903 20:05:40.568960 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1286) took 2177 ms. Trace:
I20250903 20:05:40.568938 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_0f0c9fd203f5428a898af64965508772"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:40.569017 27687 rpcz_store.cc:276] 0903 20:05:38.391282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.391338 (+ 56us) service_pool.cc:225] Handling call
0903 20:05:40.568879 (+2177541us) ts_tablet_manager.cc:1145] Deleting tablet ed73c335b042476fa03f154d23a611ad
0903 20:05:40.568886 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.568949 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2177361}
I20250903 20:05:40.569257 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1287) took 2148 ms. Trace:
I20250903 20:05:40.569307 27687 rpcz_store.cc:276] 0903 20:05:38.420813 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.420877 (+ 64us) service_pool.cc:225] Handling call
0903 20:05:40.569187 (+2148310us) ts_tablet_manager.cc:1145] Deleting tablet ca722dfe38ac4e63b024508a3997cedb
0903 20:05:40.569194 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.569248 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2148114}
I20250903 20:05:40.569536 27687 tablet_replica.cc:331] T 3f303233e6c347289650c6cf83f8127f P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
W20250903 20:05:40.569509 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f0c9fd203f5428a898af64965508772 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:40.569700 27687 raft_consensus.cc:2241] T 3f303233e6c347289650c6cf83f8127f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:40.569840 27687 raft_consensus.cc:2270] T 3f303233e6c347289650c6cf83f8127f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:40.570415 27687 ts_tablet_manager.cc:1905] T 3f303233e6c347289650c6cf83f8127f P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:40.571070 27059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_8a3e1bffc7f340f1b4b6d7dcb66fdcfb" } new_table_name: "default.Table_5737fdf0e3574fc8bccc6a7f8283c755" modify_external_catalogs: true
I20250903 20:05:40.560160 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 1f1120373c3845d6b8bebd65860ae47a (DEFAULT_TABLE table=default.Table_61fe5d6eecb744fdabbe92ed7320115a [id=de7ce5c3fb4342d182d00f49da5aecc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.571871 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f1120373c3845d6b8bebd65860ae47a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.572602 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2b9db406a0804f1c8f25adb358424581" } modify_external_catalogs: true
I20250903 20:05:40.572736 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_2b9db406a0804f1c8f25adb358424581" } modify_external_catalogs: true
I20250903 20:05:40.574044 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_9e2de10a0b504b3897db3979c5631893"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:40.574425 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5fec15415a2b4c22826e4038aaf75aa9" } new_table_name: "default.Table_58d47d6f1f25416982eef2a1869352bf" modify_external_catalogs: true
I20250903 20:05:40.575290 29612 raft_consensus.cc:491] T 6d21f38759d743af972635f8b5b8d8d2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:40.575446 29612 raft_consensus.cc:513] T 6d21f38759d743af972635f8b5b8d8d2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.575670 29371 raft_consensus.cc:491] T 7cbbeb8bec5e464b85b8568e916c765f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:40.575785 29371 raft_consensus.cc:513] T 7cbbeb8bec5e464b85b8568e916c765f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.576020 29612 leader_election.cc:290] T 6d21f38759d743af972635f8b5b8d8d2 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:40.576282 29371 leader_election.cc:290] T 7cbbeb8bec5e464b85b8568e916c765f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:40.576793 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e2de10a0b504b3897db3979c5631893 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:40.576946 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cbbeb8bec5e464b85b8568e916c765f" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:40.577354 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d21f38759d743af972635f8b5b8d8d2" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:40.577484 27462 leader_election.cc:343] T 7cbbeb8bec5e464b85b8568e916c765f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:40.577775 27462 leader_election.cc:343] T 6d21f38759d743af972635f8b5b8d8d2 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.577888 27695 ts_tablet_manager.cc:1428] T 314856bf269f4df7aa25f5a30cec0d5b P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.043s user 0.002s sys 0.000s
I20250903 20:05:40.578620 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cbbeb8bec5e464b85b8568e916c765f" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:40.578903 27695 tablet_bootstrap.cc:492] T 249362a0b8304719863bcb82c47273b3 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:40.578992 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d21f38759d743af972635f8b5b8d8d2" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:40.580163 27462 leader_election.cc:343] T 7cbbeb8bec5e464b85b8568e916c765f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.580202 27685 ts_tablet_manager.cc:1918] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:40.580269 27462 leader_election.cc:304] T 7cbbeb8bec5e464b85b8568e916c765f P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:40.580343 27685 log.cc:1199] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/6800e0b6269045f29682704fd847d5d1
W20250903 20:05:40.580686 27462 leader_election.cc:343] T 6d21f38759d743af972635f8b5b8d8d2 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.580765 27685 ts_tablet_manager.cc:1939] T 6800e0b6269045f29682704fd847d5d1 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:40.580773 27462 leader_election.cc:304] T 6d21f38759d743af972635f8b5b8d8d2 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:40.581127 29612 raft_consensus.cc:2747] T 7cbbeb8bec5e464b85b8568e916c765f P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:40.581115 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1066) took 5952 ms. Trace:
I20250903 20:05:40.581274 29612 raft_consensus.cc:2747] T 6d21f38759d743af972635f8b5b8d8d2 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:40.581195 27685 rpcz_store.cc:276] 0903 20:05:34.628863 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.629176 (+ 313us) service_pool.cc:225] Handling call
0903 20:05:40.474934 (+5845758us) ts_tablet_manager.cc:1145] Deleting tablet 6800e0b6269045f29682704fd847d5d1
0903 20:05:40.474947 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.548399 (+ 73452us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.579989 (+ 31590us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.581098 (+ 1109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5845445}
I20250903 20:05:40.581671 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1067) took 5931 ms. Trace:
I20250903 20:05:40.581755 27685 rpcz_store.cc:276] 0903 20:05:34.649799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.650176 (+ 377us) service_pool.cc:225] Handling call
0903 20:05:40.581526 (+5931350us) ts_tablet_manager.cc:1145] Deleting tablet 8bebea9fa64e459cb24639e1f2d11ac3
0903 20:05:40.581537 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.581656 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5931088}
I20250903 20:05:40.582182 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_867d45b3bc9248559147d26a7af1e231" } new_table_name: "default.Table_55a381059424427295cc1e7dcdfdf1ee" modify_external_catalogs: true
I20250903 20:05:40.582968 27692 tablet_bootstrap.cc:492] T 878f6ad087614c15bc7e29c784f2c3c9 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:40.583101 27692 ts_tablet_manager.cc:1397] T 878f6ad087614c15bc7e29c784f2c3c9 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.072s user 0.006s sys 0.000s
I20250903 20:05:40.584067 27692 raft_consensus.cc:357] T 878f6ad087614c15bc7e29c784f2c3c9 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.584236 27692 raft_consensus.cc:383] T 878f6ad087614c15bc7e29c784f2c3c9 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:40.584304 27692 raft_consensus.cc:738] T 878f6ad087614c15bc7e29c784f2c3c9 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:40.584529 27693 tablet_bootstrap.cc:492] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:40.584506 27692 consensus_queue.cc:260] T 878f6ad087614c15bc7e29c784f2c3c9 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.584789 27693 ts_tablet_manager.cc:1397] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.066s user 0.000s sys 0.004s
I20250903 20:05:40.585062 27692 ts_tablet_manager.cc:1428] T 878f6ad087614c15bc7e29c784f2c3c9 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:40.585352 27692 tablet_bootstrap.cc:492] T 41056c5c5003415993bee74d7621bd14 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:40.586810 27692 tablet_bootstrap.cc:654] T 41056c5c5003415993bee74d7621bd14 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:40.586700 27693 raft_consensus.cc:357] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.586973 27693 raft_consensus.cc:383] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:40.587086 27693 raft_consensus.cc:738] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:40.587373 27693 consensus_queue.cc:260] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.587919 27693 ts_tablet_manager.cc:1428] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250903 20:05:40.588212 27693 tablet_bootstrap.cc:492] T 54f1ce5192f547f595dea23a9eabf8e6 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:40.590430 27693 tablet_bootstrap.cc:654] T 54f1ce5192f547f595dea23a9eabf8e6 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:40.582144 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1070) took 5812 ms. Trace:
I20250903 20:05:40.598377 27685 rpcz_store.cc:276] 0903 20:05:34.769470 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.769535 (+ 65us) service_pool.cc:225] Handling call
0903 20:05:40.582026 (+5812491us) ts_tablet_manager.cc:1145] Deleting tablet 52ee6ba19a734329acde7608327f2a78
0903 20:05:40.582036 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.582130 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5812311}
I20250903 20:05:40.587670 27695 tablet_bootstrap.cc:654] T 249362a0b8304719863bcb82c47273b3 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:40.599663 27693 tablet_bootstrap.cc:492] T 54f1ce5192f547f595dea23a9eabf8e6 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:40.598917 27685 tablet_replica.cc:331] T 24eadc70e10d4d02bcb643d45c9e0e36 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:40.602504 27685 raft_consensus.cc:2241] T 24eadc70e10d4d02bcb643d45c9e0e36 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:40.602749 27685 raft_consensus.cc:2270] T 24eadc70e10d4d02bcb643d45c9e0e36 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:40.603356 27685 ts_tablet_manager.cc:1905] T 24eadc70e10d4d02bcb643d45c9e0e36 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:40.606477 27693 ts_tablet_manager.cc:1397] T 54f1ce5192f547f595dea23a9eabf8e6 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.004s
I20250903 20:05:40.608702 27687 ts_tablet_manager.cc:1918] T 3f303233e6c347289650c6cf83f8127f P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:40.608911 27687 log.cc:1199] T 3f303233e6c347289650c6cf83f8127f P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/3f303233e6c347289650c6cf83f8127f
I20250903 20:05:40.609395 27687 ts_tablet_manager.cc:1939] T 3f303233e6c347289650c6cf83f8127f P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
W20250903 20:05:40.614440 27052 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.26.65.66:38449, user_credentials={real_user=slave}} blocked reactor thread for 58458.1us
I20250903 20:05:40.615203 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 6800e0b6269045f29682704fd847d5d1 (table default.Table_6656b728e12a482fa9e7893ef4842ac7 [id=6788b524a8654e34a4d9ee8cc8fc84d3]) successfully deleted
W20250903 20:05:40.615681 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 8bebea9fa64e459cb24639e1f2d11ac3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8bebea9fa64e459cb24639e1f2d11ac3 already in progress: creating tablet
W20250903 20:05:40.616053 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 52ee6ba19a734329acde7608327f2a78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52ee6ba19a734329acde7608327f2a78 already in progress: creating tablet
W20250903 20:05:40.616684 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 238f2f30377b42c2852b016018aa3245 because the tablet was not found. No further retry: Not found: Tablet not found: 238f2f30377b42c2852b016018aa3245
W20250903 20:05:40.617015 27052 catalog_manager.cc:4895] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 950a6fcd35f64f2d889831fe9cdf503a because the tablet was not found. No further retry: Not found: Tablet not found: 950a6fcd35f64f2d889831fe9cdf503a
W20250903 20:05:40.617380 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet a868191630454a1eb3a4301664178a9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a868191630454a1eb3a4301664178a9a already in progress: creating tablet
W20250903 20:05:40.617682 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet ed73c335b042476fa03f154d23a611ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed73c335b042476fa03f154d23a611ad already in progress: creating tablet
W20250903 20:05:40.617977 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet ca722dfe38ac4e63b024508a3997cedb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca722dfe38ac4e63b024508a3997cedb already in progress: creating tablet
I20250903 20:05:40.609404 27693 raft_consensus.cc:357] T 54f1ce5192f547f595dea23a9eabf8e6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.622653 27693 raft_consensus.cc:383] T 54f1ce5192f547f595dea23a9eabf8e6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:40.622769 27693 raft_consensus.cc:738] T 54f1ce5192f547f595dea23a9eabf8e6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:40.623088 27693 consensus_queue.cc:260] T 54f1ce5192f547f595dea23a9eabf8e6 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.623849 27229 tablet_service.cc:1468] Processing CreateTablet for tablet d224e78b72ae407f8df67f728dff227b (DEFAULT_TABLE table=default.Table_b7cf7b55414f48c7925459bc4b5b8f8a [id=adff559c8dc444b78acaca6965401dc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.624148 27362 tablet_service.cc:1468] Processing CreateTablet for tablet 7335742c1fe4443f98151df37e153026 (DEFAULT_TABLE table=default.Table_46fddb6b15134d34a8a8702787f38259 [id=afe3d13369cf468993ee5b4eb64667aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.623669 27365 tablet_service.cc:1468] Processing CreateTablet for tablet d224e78b72ae407f8df67f728dff227b (DEFAULT_TABLE table=default.Table_b7cf7b55414f48c7925459bc4b5b8f8a [id=adff559c8dc444b78acaca6965401dc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.624440 27229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d224e78b72ae407f8df67f728dff227b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.626202 27216 tablet_service.cc:1468] Processing CreateTablet for tablet 7335742c1fe4443f98151df37e153026 (DEFAULT_TABLE table=default.Table_46fddb6b15134d34a8a8702787f38259 [id=afe3d13369cf468993ee5b4eb64667aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.627094 27216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7335742c1fe4443f98151df37e153026. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.627174 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7335742c1fe4443f98151df37e153026. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.630478 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d224e78b72ae407f8df67f728dff227b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.633205 27693 ts_tablet_manager.cc:1428] T 54f1ce5192f547f595dea23a9eabf8e6 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.027s user 0.001s sys 0.001s
I20250903 20:05:40.633555 27693 tablet_bootstrap.cc:492] T ea3f82f4e4244219ab0919f3e341aead P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:40.635536 27693 tablet_bootstrap.cc:654] T ea3f82f4e4244219ab0919f3e341aead P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:40.637504 27692 tablet_bootstrap.cc:492] T 41056c5c5003415993bee74d7621bd14 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:40.637638 27692 ts_tablet_manager.cc:1397] T 41056c5c5003415993bee74d7621bd14 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.000s
I20250903 20:05:40.638435 27692 raft_consensus.cc:357] T 41056c5c5003415993bee74d7621bd14 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.638648 27692 raft_consensus.cc:383] T 41056c5c5003415993bee74d7621bd14 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:40.638715 27692 raft_consensus.cc:738] T 41056c5c5003415993bee74d7621bd14 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:40.638968 27692 consensus_queue.cc:260] T 41056c5c5003415993bee74d7621bd14 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.639820 27692 ts_tablet_manager.cc:1428] T 41056c5c5003415993bee74d7621bd14 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:40.640187 27692 tablet_bootstrap.cc:492] T b3d9491f777b422da4b9fca803beabff P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:40.642343 27692 tablet_bootstrap.cc:654] T b3d9491f777b422da4b9fca803beabff P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:40.643404 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1290) took 2086 ms. Trace:
I20250903 20:05:40.643563 27687 rpcz_store.cc:276] 0903 20:05:38.556448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.556524 (+ 76us) service_pool.cc:225] Handling call
0903 20:05:40.569494 (+2012970us) ts_tablet_manager.cc:1145] Deleting tablet 3f303233e6c347289650c6cf83f8127f
0903 20:05:40.569503 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.573278 (+ 3775us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.608577 (+ 35299us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.643375 (+ 34798us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1948465}
I20250903 20:05:40.644129 27687 tablet_replica.cc:331] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:40.644366 27687 raft_consensus.cc:2241] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Raft consensus shutting down.
I20250903 20:05:40.644737 27687 raft_consensus.cc:2270] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:40.648330 27685 ts_tablet_manager.cc:1918] T 24eadc70e10d4d02bcb643d45c9e0e36 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:40.648581 27685 log.cc:1199] T 24eadc70e10d4d02bcb643d45c9e0e36 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/24eadc70e10d4d02bcb643d45c9e0e36
I20250903 20:05:40.649034 27685 ts_tablet_manager.cc:1939] T 24eadc70e10d4d02bcb643d45c9e0e36 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:40.655287 27684 ts_tablet_manager.cc:1918] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:40.655808 27684 log.cc:1199] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/3919f7f95a954f59b68f56d0051c0109
I20250903 20:05:40.657160 27684 ts_tablet_manager.cc:1939] T 3919f7f95a954f59b68f56d0051c0109 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:40.663406 27692 tablet_bootstrap.cc:492] T b3d9491f777b422da4b9fca803beabff P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:40.663581 27692 ts_tablet_manager.cc:1397] T b3d9491f777b422da4b9fca803beabff P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.001s
I20250903 20:05:40.664315 27692 raft_consensus.cc:357] T b3d9491f777b422da4b9fca803beabff P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.664494 27692 raft_consensus.cc:383] T b3d9491f777b422da4b9fca803beabff P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:40.664541 27692 raft_consensus.cc:738] T b3d9491f777b422da4b9fca803beabff P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:40.664717 27692 consensus_queue.cc:260] T b3d9491f777b422da4b9fca803beabff P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.669165 27362 tablet_service.cc:1468] Processing CreateTablet for tablet 3af2fef47f9f41a381f0b9af505e3c77 (DEFAULT_TABLE table=default.Table_3110722931cd47ad8e82109243b34fc1 [id=b030619ad6274ddaabf1628d04158648]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.669173 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 3f303233e6c347289650c6cf83f8127f (table default.Table_58569618f9d04fdfbd005b3d0e15c049 [id=1cbfdd865e51404688e08bc60eec2197]) successfully deleted
I20250903 20:05:40.669167 27229 tablet_service.cc:1468] Processing CreateTablet for tablet 3af2fef47f9f41a381f0b9af505e3c77 (DEFAULT_TABLE table=default.Table_3110722931cd47ad8e82109243b34fc1 [id=b030619ad6274ddaabf1628d04158648]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.669750 27229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3af2fef47f9f41a381f0b9af505e3c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.669952 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3af2fef47f9f41a381f0b9af505e3c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.665174 27692 ts_tablet_manager.cc:1428] T b3d9491f777b422da4b9fca803beabff P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:40.674463 27692 tablet_bootstrap.cc:492] T 97d261080b8b479ca12267b6aa2e2ab0 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:40.667155 29371 consensus_queue.cc:1035] T 615f42e95bb24c8480f73d371fd63b9c P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20250903 20:05:40.676894 27692 tablet_bootstrap.cc:654] T 97d261080b8b479ca12267b6aa2e2ab0 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:40.680836 27509 raft_consensus.cc:3058] T 5c42d84a6a5649208b0539a10da21aec P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:40.690682 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_4dbe7e25d163409f83c4836c2158d1e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:40.691603 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4dbe7e25d163409f83c4836c2158d1e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:40.693607 27057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_f985118d9f13472383987d925c566bb4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:40.694432 27057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f985118d9f13472383987d925c566bb4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:40.695155 27693 tablet_bootstrap.cc:492] T ea3f82f4e4244219ab0919f3e341aead P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:40.695374 27693 ts_tablet_manager.cc:1397] T ea3f82f4e4244219ab0919f3e341aead P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.062s user 0.000s sys 0.007s
I20250903 20:05:40.696578 27693 raft_consensus.cc:357] T ea3f82f4e4244219ab0919f3e341aead P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.696986 27693 raft_consensus.cc:383] T ea3f82f4e4244219ab0919f3e341aead P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:40.697144 27693 raft_consensus.cc:738] T ea3f82f4e4244219ab0919f3e341aead P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:40.697445 27693 consensus_queue.cc:260] T ea3f82f4e4244219ab0919f3e341aead P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.698307 27693 ts_tablet_manager.cc:1428] T ea3f82f4e4244219ab0919f3e341aead P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250903 20:05:40.699815 27692 tablet_bootstrap.cc:492] T 97d261080b8b479ca12267b6aa2e2ab0 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
W20250903 20:05:40.705835 27059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6b1fbefaabc540d8a3db2a86f4920044 consensus state has no leader
I20250903 20:05:40.706205 27692 ts_tablet_manager.cc:1397] T 97d261080b8b479ca12267b6aa2e2ab0 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.002s
I20250903 20:05:40.707208 27692 raft_consensus.cc:357] T 97d261080b8b479ca12267b6aa2e2ab0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.707515 27692 raft_consensus.cc:383] T 97d261080b8b479ca12267b6aa2e2ab0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:40.707616 27692 raft_consensus.cc:738] T 97d261080b8b479ca12267b6aa2e2ab0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
W20250903 20:05:40.707909 27051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.125:44059, user_credentials={real_user=slave}} blocked reactor thread for 140267us
I20250903 20:05:40.707891 27692 consensus_queue.cc:260] T 97d261080b8b479ca12267b6aa2e2ab0 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.708425 27692 ts_tablet_manager.cc:1428] T 97d261080b8b479ca12267b6aa2e2ab0 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250903 20:05:40.708725 27692 tablet_bootstrap.cc:492] T 55b2cdee20264ae6b64ebe3607fe1ca6 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:40.708887 27497 tablet_service.cc:1468] Processing CreateTablet for tablet 3e45128d635f442fbdfb32531be2256b (DEFAULT_TABLE table=default.Table_1dc6317976b84af1b853bbb3825bcf82 [id=3e562137069c499ab6b0a0e3c3a8b14d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.709283 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e45128d635f442fbdfb32531be2256b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.711448 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1071) took 5941 ms. Trace:
I20250903 20:05:40.711548 27685 rpcz_store.cc:276] 0903 20:05:34.769567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.769755 (+ 188us) service_pool.cc:225] Handling call
0903 20:05:40.598861 (+5829106us) ts_tablet_manager.cc:1145] Deleting tablet 24eadc70e10d4d02bcb643d45c9e0e36
0903 20:05:40.598876 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.632116 (+ 33240us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.648205 (+ 16089us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.711423 (+ 63218us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5828956}
I20250903 20:05:40.711961 27685 tablet_replica.cc:331] T 1223743918814d88b401782ca4b521dd P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:40.712157 27685 raft_consensus.cc:2241] T 1223743918814d88b401782ca4b521dd P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:40.712349 27685 raft_consensus.cc:2270] T 1223743918814d88b401782ca4b521dd P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:40.704495 27693 tablet_bootstrap.cc:492] T ef02f91c0964446daf62ac1dde0330ff P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:40.712921 27685 ts_tablet_manager.cc:1905] T 1223743918814d88b401782ca4b521dd P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:40.716449 27695 tablet_bootstrap.cc:492] T 249362a0b8304719863bcb82c47273b3 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:40.716624 27695 ts_tablet_manager.cc:1397] T 249362a0b8304719863bcb82c47273b3 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.138s user 0.006s sys 0.006s
I20250903 20:05:40.717368 27695 raft_consensus.cc:357] T 249362a0b8304719863bcb82c47273b3 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.717548 27695 raft_consensus.cc:383] T 249362a0b8304719863bcb82c47273b3 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:40.717600 27695 raft_consensus.cc:738] T 249362a0b8304719863bcb82c47273b3 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:40.719688 27695 consensus_queue.cc:260] T 249362a0b8304719863bcb82c47273b3 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.728837 27497 tablet_service.cc:1468] Processing CreateTablet for tablet 5b5be2af29c245c9973fdc2e76304e3f (DEFAULT_TABLE table=default.Table_9b36ebc04fa442909cce88c8e4695c10 [id=48b9220544d647dea63d37a7183e20a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.727756 27695 ts_tablet_manager.cc:1428] T 249362a0b8304719863bcb82c47273b3 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250903 20:05:40.729597 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b5be2af29c245c9973fdc2e76304e3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.731585 27695 tablet_bootstrap.cc:492] T f4dec89fc6bb436ba6542c267eaf9c15 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:40.732196 27692 tablet_bootstrap.cc:654] T 55b2cdee20264ae6b64ebe3607fe1ca6 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:40.734499 27695 tablet_bootstrap.cc:654] T f4dec89fc6bb436ba6542c267eaf9c15 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:40.734800 29371 raft_consensus.cc:491] T 847d0a0a3d2b45938cec9058c77f6231 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:40.734922 29371 raft_consensus.cc:513] T 847d0a0a3d2b45938cec9058c77f6231 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.735493 29371 leader_election.cc:290] T 847d0a0a3d2b45938cec9058c77f6231 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:40.736495 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "847d0a0a3d2b45938cec9058c77f6231" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:40.736804 27693 tablet_bootstrap.cc:654] T ef02f91c0964446daf62ac1dde0330ff P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:40.737458 27462 leader_election.cc:343] T 847d0a0a3d2b45938cec9058c77f6231 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.737530 27493 tablet_service.cc:1468] Processing CreateTablet for tablet 1f1120373c3845d6b8bebd65860ae47a (DEFAULT_TABLE table=default.Table_61fe5d6eecb744fdabbe92ed7320115a [id=de7ce5c3fb4342d182d00f49da5aecc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.738004 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f1120373c3845d6b8bebd65860ae47a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.743762 27693 tablet_bootstrap.cc:492] T ef02f91c0964446daf62ac1dde0330ff P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:40.744052 27693 ts_tablet_manager.cc:1397] T ef02f91c0964446daf62ac1dde0330ff P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.003s
I20250903 20:05:40.746187 27693 raft_consensus.cc:357] T ef02f91c0964446daf62ac1dde0330ff P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.746512 27693 raft_consensus.cc:383] T ef02f91c0964446daf62ac1dde0330ff P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:40.746740 27693 raft_consensus.cc:738] T ef02f91c0964446daf62ac1dde0330ff P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:40.747036 27693 consensus_queue.cc:260] T ef02f91c0964446daf62ac1dde0330ff P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.750793 29669 raft_consensus.cc:491] T 60c32370a5274347afafc6a7e1e94270 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:40.752871 27493 tablet_service.cc:1468] Processing CreateTablet for tablet a492a9de59844bb5a4c8dfe10bf36e75 (DEFAULT_TABLE table=default.Table_db9c4046409e4dfda92270621470ecd4 [id=68d86c1824fc416ca8bc7e63f7b4b4e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.753484 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a492a9de59844bb5a4c8dfe10bf36e75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.751458 29669 raft_consensus.cc:513] T 60c32370a5274347afafc6a7e1e94270 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.754527 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 24eadc70e10d4d02bcb643d45c9e0e36 (table default.Table_76a88a96cd544cda882c413357a7f8c3 [id=e9a291cff3f4497cad9393ab4c571bf7]) successfully deleted
I20250903 20:05:40.710958 27359 tablet_service.cc:1468] Processing CreateTablet for tablet 98482b5883584922a09e94978468b7fa (DEFAULT_TABLE table=default.Table_6eb8ab17496f4e13906809835d0e8d72 [id=24ffdbd9a54544608fd28172e117f297]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.755131 29669 leader_election.cc:290] T 60c32370a5274347afafc6a7e1e94270 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:40.755357 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98482b5883584922a09e94978468b7fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.755738 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60c32370a5274347afafc6a7e1e94270" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:40.755932 27516 raft_consensus.cc:2466] T 60c32370a5274347afafc6a7e1e94270 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:40.756371 27320 leader_election.cc:304] T 60c32370a5274347afafc6a7e1e94270 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:40.756629 29669 raft_consensus.cc:2802] T 60c32370a5274347afafc6a7e1e94270 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:40.756691 29669 raft_consensus.cc:491] T 60c32370a5274347afafc6a7e1e94270 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:40.756747 29669 raft_consensus.cc:3058] T 60c32370a5274347afafc6a7e1e94270 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:40.706985 27687 ts_tablet_manager.cc:1905] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:40.759294 27506 tablet_service.cc:1468] Processing CreateTablet for tablet d224e78b72ae407f8df67f728dff227b (DEFAULT_TABLE table=default.Table_b7cf7b55414f48c7925459bc4b5b8f8a [id=adff559c8dc444b78acaca6965401dc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.759325 29669 raft_consensus.cc:513] T 60c32370a5274347afafc6a7e1e94270 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.759738 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d224e78b72ae407f8df67f728dff227b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.760921 29669 leader_election.cc:290] T 60c32370a5274347afafc6a7e1e94270 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:40.761183 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60c32370a5274347afafc6a7e1e94270" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:40.761355 27516 raft_consensus.cc:3058] T 60c32370a5274347afafc6a7e1e94270 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:40.764134 27516 raft_consensus.cc:2466] T 60c32370a5274347afafc6a7e1e94270 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:40.764746 27320 leader_election.cc:304] T 60c32370a5274347afafc6a7e1e94270 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:40.765204 29669 raft_consensus.cc:2802] T 60c32370a5274347afafc6a7e1e94270 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:40.765738 29669 raft_consensus.cc:695] T 60c32370a5274347afafc6a7e1e94270 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:40.766597 29669 consensus_queue.cc:237] T 60c32370a5274347afafc6a7e1e94270 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.767206 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 7335742c1fe4443f98151df37e153026 (DEFAULT_TABLE table=default.Table_46fddb6b15134d34a8a8702787f38259 [id=afe3d13369cf468993ee5b4eb64667aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.767853 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7335742c1fe4443f98151df37e153026. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.776968 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 9425f568adf944c2a1910f20ffc05352
I20250903 20:05:40.777323 27693 ts_tablet_manager.cc:1428] T ef02f91c0964446daf62ac1dde0330ff P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.033s user 0.000s sys 0.002s
I20250903 20:05:40.778857 29746 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.65.125:44059 every 8 connection(s)
I20250903 20:05:40.779291 29170 rpc_server.cc:307] RPC server started. Bound to: 127.26.65.125:44059
I20250903 20:05:40.779551 27693 tablet_bootstrap.cc:492] T 733f8024e1fa401ca9614a97db91a19b P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:40.781862 29170 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/master-1/data/info.pb
I20250903 20:05:40.783397 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet 9425f568adf944c2a1910f20ffc05352 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:40 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:40.784637 26885 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskH7eH49/build/asan/bin/kudu as pid 29170
I20250903 20:05:40.784989 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1261) took 2714 ms. Trace:
I20250903 20:05:40.785162 27684 rpcz_store.cc:276] 0903 20:05:38.070397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.070449 (+ 52us) service_pool.cc:225] Handling call
0903 20:05:40.485642 (+2415193us) ts_tablet_manager.cc:1145] Deleting tablet 3919f7f95a954f59b68f56d0051c0109
0903 20:05:40.485657 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.535442 (+ 49785us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.655064 (+119622us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.784957 (+129893us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2415028}
I20250903 20:05:40.785945 27684 tablet_replica.cc:331] T 1bf570855b204b96acd070587c434463 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:40.786181 27684 raft_consensus.cc:2241] T 1bf570855b204b96acd070587c434463 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:40.786432 27684 raft_consensus.cc:2270] T 1bf570855b204b96acd070587c434463 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:40.788504 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 3af2fef47f9f41a381f0b9af505e3c77 (DEFAULT_TABLE table=default.Table_3110722931cd47ad8e82109243b34fc1 [id=b030619ad6274ddaabf1628d04158648]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.789120 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3af2fef47f9f41a381f0b9af505e3c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.790277 29670 raft_consensus.cc:491] T f9375fff5f5945078c4412aa21c06f0d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:40.790438 29670 raft_consensus.cc:513] T f9375fff5f5945078c4412aa21c06f0d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.791203 29670 leader_election.cc:290] T f9375fff5f5945078c4412aa21c06f0d P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:40.793797 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 8d0342a87007454ea108a9dc5b66b3f7 (DEFAULT_TABLE table=default.Table_cb5df7f62a6e49ccaf82d85fde99b264 [id=7710431af7184913b778ea9fefd2b4f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.794196 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d0342a87007454ea108a9dc5b66b3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.795907 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9375fff5f5945078c4412aa21c06f0d" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:40.795989 27359 tablet_service.cc:1468] Processing CreateTablet for tablet 760a0bd9eb924ff7a33b336b057b850a (DEFAULT_TABLE table=default.Table_afd931c89f1b4dcab359c5c7fa189952 [id=6cf04b0c84ec444683a35c2bb240fcd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.796324 27359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 760a0bd9eb924ff7a33b336b057b850a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.796980 27516 raft_consensus.cc:2391] T f9375fff5f5945078c4412aa21c06f0d P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 51578fb7bceb4e119b3ab1e58c4f5528 in current term 1: Already voted for candidate 89f8f0ee68b940db9ce250c0604f934a in this term.
I20250903 20:05:40.798533 27365 tablet_service.cc:1468] Processing CreateTablet for tablet 6638e942adad4f6b8fefe45fe8e03be3 (DEFAULT_TABLE table=default.Table_47fab307e9c2482a8bf34af44051f49c [id=b7595780b7be45db81c634e60f3f5ce1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.798741 27347 tablet_service.cc:1468] Processing CreateTablet for tablet 9db511fd189347a08771584362b7b1ae (DEFAULT_TABLE table=default.Table_e1253db17bbc45d0a1f22c8ba8ea96c6 [id=2468ce4f2c464648b66f3fb18460eefa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.798852 27365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6638e942adad4f6b8fefe45fe8e03be3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.799031 27347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9db511fd189347a08771584362b7b1ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.798492 27357 tablet_service.cc:1468] Processing CreateTablet for tablet 02e86d1f996b4388a383af7610880af0 (DEFAULT_TABLE table=default.Table_99280cacf5a44f0a8818e3e0dedc93f6 [id=245169ef6cb04b6c8ac2e45fdbca7a7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.804158 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02e86d1f996b4388a383af7610880af0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:40.804425 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ddc3f18eeba240cf97986c20f3cf75fa consensus state has no leader
I20250903 20:05:40.808777 29612 raft_consensus.cc:491] T 97d261080b8b479ca12267b6aa2e2ab0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:40.808938 29612 raft_consensus.cc:513] T 97d261080b8b479ca12267b6aa2e2ab0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.809531 29612 leader_election.cc:290] T 97d261080b8b479ca12267b6aa2e2ab0 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:40.810333 27509 raft_consensus.cc:2466] T 5c42d84a6a5649208b0539a10da21aec P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:40.810819 27320 leader_election.cc:304] T 5c42d84a6a5649208b0539a10da21aec P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:40.810818 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "97d261080b8b479ca12267b6aa2e2ab0" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:40.811479 27462 leader_election.cc:343] T 97d261080b8b479ca12267b6aa2e2ab0 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.811703 29670 raft_consensus.cc:2802] T 5c42d84a6a5649208b0539a10da21aec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:40.811837 29670 raft_consensus.cc:695] T 5c42d84a6a5649208b0539a10da21aec P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:40.812096 29670 consensus_queue.cc:237] T 5c42d84a6a5649208b0539a10da21aec P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.815167 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 3919f7f95a954f59b68f56d0051c0109 (table default.Table_e79b97db68cd4a7d827d56a72a6d15e9 [id=147416d6b2ea4002953902e413c5a57e]) successfully deleted
I20250903 20:05:40.815858 27063 master_service.cc:948] ReplaceTablet: received request to replace tablet 16549a6d9aa9432eb5cee19f69ce6957 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:40.816288 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_036776f601aa4311992c12179caf979f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:40.816974 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_036776f601aa4311992c12179caf979f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:40.817221 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 9425f568adf944c2a1910f20ffc05352 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:40 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:40.818241 27684 ts_tablet_manager.cc:1905] T 1bf570855b204b96acd070587c434463 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:40.818451 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7713d2cb69d74ea895a8c86a04ff9c50" } modify_external_catalogs: true
I20250903 20:05:40.818634 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_7713d2cb69d74ea895a8c86a04ff9c50" } modify_external_catalogs: true
I20250903 20:05:40.833402 27684 ts_tablet_manager.cc:1918] T 1bf570855b204b96acd070587c434463 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:40.833549 27684 log.cc:1199] T 1bf570855b204b96acd070587c434463 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/1bf570855b204b96acd070587c434463
I20250903 20:05:40.833954 27684 ts_tablet_manager.cc:1939] T 1bf570855b204b96acd070587c434463 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:40.736912 27243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "847d0a0a3d2b45938cec9058c77f6231" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:40.834440 27062 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_43b40760a8fb4b2c9cd1be14a4a536c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:40.746078 27240 raft_consensus.cc:3058] T 249362a0b8304719863bcb82c47273b3 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:40.834952 27462 leader_election.cc:343] T 847d0a0a3d2b45938cec9058c77f6231 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:40.835069 27062 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43b40760a8fb4b2c9cd1be14a4a536c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:40.835098 27462 leader_election.cc:304] T 847d0a0a3d2b45938cec9058c77f6231 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:40.835536 29612 raft_consensus.cc:2747] T 847d0a0a3d2b45938cec9058c77f6231 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:40.753705 27685 ts_tablet_manager.cc:1918] T 1223743918814d88b401782ca4b521dd P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:40.755748 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60c32370a5274347afafc6a7e1e94270" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:40.760776 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60c32370a5274347afafc6a7e1e94270" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:40.783355 27210 tablet_service.cc:1515] Processing DeleteTablet for tablet 9425f568adf944c2a1910f20ffc05352 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:40 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:40.837585 27685 log.cc:1199] T 1223743918814d88b401782ca4b521dd P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/1223743918814d88b401782ca4b521dd
I20250903 20:05:40.791285 27229 tablet_service.cc:1468] Processing CreateTablet for tablet 98482b5883584922a09e94978468b7fa (DEFAULT_TABLE table=default.Table_6eb8ab17496f4e13906809835d0e8d72 [id=24ffdbd9a54544608fd28172e117f297]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.791548 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 760a0bd9eb924ff7a33b336b057b850a (DEFAULT_TABLE table=default.Table_afd931c89f1b4dcab359c5c7fa189952 [id=6cf04b0c84ec444683a35c2bb240fcd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.792145 27246 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9375fff5f5945078c4412aa21c06f0d" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:40.793051 27211 tablet_service.cc:1468] Processing CreateTablet for tablet 6638e942adad4f6b8fefe45fe8e03be3 (DEFAULT_TABLE table=default.Table_47fab307e9c2482a8bf34af44051f49c [id=b7595780b7be45db81c634e60f3f5ce1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.838055 27229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98482b5883584922a09e94978468b7fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.794147 27217 tablet_service.cc:1468] Processing CreateTablet for tablet 8d0342a87007454ea108a9dc5b66b3f7 (DEFAULT_TABLE table=default.Table_cb5df7f62a6e49ccaf82d85fde99b264 [id=7710431af7184913b778ea9fefd2b4f8]), partition=RANGE (key) PARTITION UNBOUNDED
W20250903 20:05:40.838331 27321 leader_election.cc:343] T 60c32370a5274347afafc6a7e1e94270 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.838394 27217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d0342a87007454ea108a9dc5b66b3f7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:40.838775 27321 leader_election.cc:343] T 60c32370a5274347afafc6a7e1e94270 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.839043 27321 leader_election.cc:304] T f9375fff5f5945078c4412aa21c06f0d P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:40.839355 28567 consensus_queue.cc:1035] T 249362a0b8304719863bcb82c47273b3 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:40.839740 28567 raft_consensus.cc:3058] T f9375fff5f5945078c4412aa21c06f0d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:40.795276 27224 tablet_service.cc:1468] Processing CreateTablet for tablet 02e86d1f996b4388a383af7610880af0 (DEFAULT_TABLE table=default.Table_99280cacf5a44f0a8818e3e0dedc93f6 [id=245169ef6cb04b6c8ac2e45fdbca7a7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.802558 27228 tablet_service.cc:1468] Processing CreateTablet for tablet 9db511fd189347a08771584362b7b1ae (DEFAULT_TABLE table=default.Table_e1253db17bbc45d0a1f22c8ba8ea96c6 [id=2468ce4f2c464648b66f3fb18460eefa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.842501 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 760a0bd9eb924ff7a33b336b057b850a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.842677 27228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9db511fd189347a08771584362b7b1ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.843032 27224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02e86d1f996b4388a383af7610880af0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:40.805474 27184 consensus_peers.cc:489] T 30cc0d4ba8d444f1b98232f906ea0c7b P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30cc0d4ba8d444f1b98232f906ea0c7b. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:40.810335 27244 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "97d261080b8b479ca12267b6aa2e2ab0" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:40.846616 27211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6638e942adad4f6b8fefe45fe8e03be3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:40.846904 27462 leader_election.cc:343] T 97d261080b8b479ca12267b6aa2e2ab0 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.847038 27462 leader_election.cc:304] T 97d261080b8b479ca12267b6aa2e2ab0 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:40.847419 29612 raft_consensus.cc:2747] T 97d261080b8b479ca12267b6aa2e2ab0 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:40.847863 28567 raft_consensus.cc:2747] T f9375fff5f5945078c4412aa21c06f0d P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250903 20:05:40.853266 27183 consensus_peers.cc:489] T 259bc73ca9df47e9874692725a60bb1c P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 259bc73ca9df47e9874692725a60bb1c. This is attempt 1: this message will repeat every 5th retry.
W20250903 20:05:40.855367 27183 consensus_peers.cc:489] T 1bf570855b204b96acd070587c434463 P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1bf570855b204b96acd070587c434463. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:40.860399 27685 ts_tablet_manager.cc:1939] T 1223743918814d88b401782ca4b521dd P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:40.860788 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1073) took 6084 ms. Trace:
I20250903 20:05:40.860888 27685 rpcz_store.cc:276] 0903 20:05:34.776221 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.776283 (+ 62us) service_pool.cc:225] Handling call
0903 20:05:40.711908 (+5935625us) ts_tablet_manager.cc:1145] Deleting tablet 1223743918814d88b401782ca4b521dd
0903 20:05:40.711920 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.717011 (+ 5091us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.753571 (+ 36560us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.860770 (+107199us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5935402}
I20250903 20:05:40.861421 27685 tablet_replica.cc:331] T ed096b452ba747f89e91edc15e6e3ac1 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:40.863090 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 1223743918814d88b401782ca4b521dd (table default.Table_f038423efb3f4fd5bba9911713436b97 [id=22a7d6c72c2e4dbfadd44a229bb175e3]) successfully deleted
I20250903 20:05:40.864560 27685 raft_consensus.cc:2241] T ed096b452ba747f89e91edc15e6e3ac1 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:40.865237 27685 raft_consensus.cc:2270] T ed096b452ba747f89e91edc15e6e3ac1 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:40.866076 29612 raft_consensus.cc:491] T 878f6ad087614c15bc7e29c784f2c3c9 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:40.866364 29612 raft_consensus.cc:513] T 878f6ad087614c15bc7e29c784f2c3c9 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.867009 29612 leader_election.cc:290] T 878f6ad087614c15bc7e29c784f2c3c9 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:40.867107 27692 tablet_bootstrap.cc:492] T 55b2cdee20264ae6b64ebe3607fe1ca6 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:40.867230 27692 ts_tablet_manager.cc:1397] T 55b2cdee20264ae6b64ebe3607fe1ca6 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.159s user 0.005s sys 0.007s
I20250903 20:05:40.867759 27687 ts_tablet_manager.cc:1918] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:40.867745 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "878f6ad087614c15bc7e29c784f2c3c9" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:40.868048 27687 log.cc:1199] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/0fef9aa7153a4902abce522e3ff0df3f
I20250903 20:05:40.868063 27692 raft_consensus.cc:357] T 55b2cdee20264ae6b64ebe3607fe1ca6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.868173 27685 ts_tablet_manager.cc:1905] T ed096b452ba747f89e91edc15e6e3ac1 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:40.868212 27692 raft_consensus.cc:383] T 55b2cdee20264ae6b64ebe3607fe1ca6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250903 20:05:40.868253 27462 leader_election.cc:343] T 878f6ad087614c15bc7e29c784f2c3c9 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.868261 27692 raft_consensus.cc:738] T 55b2cdee20264ae6b64ebe3607fe1ca6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:40.868710 27692 consensus_queue.cc:260] T 55b2cdee20264ae6b64ebe3607fe1ca6 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.869349 27692 ts_tablet_manager.cc:1428] T 55b2cdee20264ae6b64ebe3607fe1ca6 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250903 20:05:40.871979 27462 consensus_peers.cc:489] T 615f42e95bb24c8480f73d371fd63b9c P e72ef1dbcc5143cf9f04a808f101c80f -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:40.874619 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 760a0bd9eb924ff7a33b336b057b850a (DEFAULT_TABLE table=default.Table_afd931c89f1b4dcab359c5c7fa189952 [id=6cf04b0c84ec444683a35c2bb240fcd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.874920 27692 tablet_bootstrap.cc:492] T e9126e4a18c546c28927f22c9f16426e P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:40.875424 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 760a0bd9eb924ff7a33b336b057b850a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.877352 27497 tablet_service.cc:1468] Processing CreateTablet for tablet 6638e942adad4f6b8fefe45fe8e03be3 (DEFAULT_TABLE table=default.Table_47fab307e9c2482a8bf34af44051f49c [id=b7595780b7be45db81c634e60f3f5ce1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.878042 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 8d0342a87007454ea108a9dc5b66b3f7 (DEFAULT_TABLE table=default.Table_cb5df7f62a6e49ccaf82d85fde99b264 [id=7710431af7184913b778ea9fefd2b4f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.878430 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6638e942adad4f6b8fefe45fe8e03be3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.878764 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d0342a87007454ea108a9dc5b66b3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.879756 27695 tablet_bootstrap.cc:492] T f4dec89fc6bb436ba6542c267eaf9c15 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:40.879884 27695 ts_tablet_manager.cc:1397] T f4dec89fc6bb436ba6542c267eaf9c15 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.148s user 0.002s sys 0.000s
I20250903 20:05:40.880609 27695 raft_consensus.cc:357] T f4dec89fc6bb436ba6542c267eaf9c15 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.880777 27695 raft_consensus.cc:383] T f4dec89fc6bb436ba6542c267eaf9c15 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:40.880824 27695 raft_consensus.cc:738] T f4dec89fc6bb436ba6542c267eaf9c15 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
W20250903 20:05:40.885083 27066 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ed90d6ea02ad4a97949e714c8b47a49b consensus state has no leader
I20250903 20:05:40.885492 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_dc6f6f88ae4b4e129dc155a6d583fd9d" } new_table_name: "default.Table_6c4ed9bc01964bcdbd0583345d956073" modify_external_catalogs: true
I20250903 20:05:40.886719 29411 raft_consensus.cc:491] T b4b15d77d07c4fa7856d9c198cf15450 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:40.886868 29411 raft_consensus.cc:513] T b4b15d77d07c4fa7856d9c198cf15450 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.887537 29411 leader_election.cc:290] T b4b15d77d07c4fa7856d9c198cf15450 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:40.894601 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b4b15d77d07c4fa7856d9c198cf15450" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:40.896756 29612 raft_consensus.cc:491] T b84f487773c24735a61d0fd0898b2bcb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:40.896920 29612 raft_consensus.cc:513] T b84f487773c24735a61d0fd0898b2bcb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.897562 29612 leader_election.cc:290] T b84f487773c24735a61d0fd0898b2bcb P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:40.898479 27244 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b84f487773c24735a61d0fd0898b2bcb" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:40.899075 29589 consensus_queue.cc:1035] T ed4902c002ef4198802375ced39ef2ad P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250903 20:05:40.899291 27462 leader_election.cc:343] T b84f487773c24735a61d0fd0898b2bcb P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.899796 27497 tablet_service.cc:1468] Processing CreateTablet for tablet 02e86d1f996b4388a383af7610880af0 (DEFAULT_TABLE table=default.Table_99280cacf5a44f0a8818e3e0dedc93f6 [id=245169ef6cb04b6c8ac2e45fdbca7a7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.900213 27497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02e86d1f996b4388a383af7610880af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.900673 27059 catalog_manager.cc:5582] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 51578fb7bceb4e119b3ab1e58c4f5528 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66). New cstate: current_term: 1 leader_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:40.901134 27059 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet f4d3e1cfdfab4be1a9143b78bf3bc46d (table default.Table_5e672ed0c8c142029a79b72f799f3f59 [id=386b8d4e7c4042d1947c19ac464c75a0]). Expected version 2 got 0
I20250903 20:05:40.901439 27059 catalog_manager.cc:5582] T 6d4b8d4bf29e408cbdf9557c8b0d8a4b P 51578fb7bceb4e119b3ab1e58c4f5528 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66). New cstate: current_term: 1 leader_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:40.901779 27059 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 6d4b8d4bf29e408cbdf9557c8b0d8a4b (table default.Table_ccdb4130549b4be5888f17097ae35106 [id=cbcfc7067aca4796b39e3eb195f38903]). Expected version 2 got 0
I20250903 20:05:40.903622 27493 tablet_service.cc:1468] Processing CreateTablet for tablet 98482b5883584922a09e94978468b7fa (DEFAULT_TABLE table=default.Table_6eb8ab17496f4e13906809835d0e8d72 [id=24ffdbd9a54544608fd28172e117f297]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.902186 27059 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 5b46bb34ac214b1ba23fba1a33f746ac (table default.Table_975e53d37e054938a5b63957252ff50d [id=f261e84232234244824d0232efbb4399]). Expected version 1 got 0
I20250903 20:05:40.904225 27504 tablet_service.cc:1468] Processing CreateTablet for tablet 9db511fd189347a08771584362b7b1ae (DEFAULT_TABLE table=default.Table_e1253db17bbc45d0a1f22c8ba8ea96c6 [id=2468ce4f2c464648b66f3fb18460eefa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.905563 27493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98482b5883584922a09e94978468b7fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.905470 27059 catalog_manager.cc:5582] T 249362a0b8304719863bcb82c47273b3 P 51578fb7bceb4e119b3ab1e58c4f5528 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66). New cstate: current_term: 1 leader_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } health_report { overall_health: UNKNOWN } } }
I20250903 20:05:40.905727 27504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9db511fd189347a08771584362b7b1ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.905906 27059 catalog_manager.cc:5674] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449) does not have the latest schema for tablet 249362a0b8304719863bcb82c47273b3 (table default.Table_1dd1b4a75c914e6a88058266d2a7e98a [id=565dc96c38de4efeb4328c01e030eb8c]). Expected version 1 got 0
I20250903 20:05:40.911947 27495 tablet_service.cc:1468] Processing CreateTablet for tablet 156a02312d6146e8a9ccd797cc8f6d59 (DEFAULT_TABLE table=default.Table_1ee4b42dee0f45299e1beacc5af5b301 [id=79515c0e3b4f4fb499718eee9d8c494b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.912369 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 156a02312d6146e8a9ccd797cc8f6d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.914503 29612 raft_consensus.cc:491] T 5f2eefeebbfc47249e2f72b7c52e905d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:40.914682 29612 raft_consensus.cc:513] T 5f2eefeebbfc47249e2f72b7c52e905d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.915244 27215 tablet_service.cc:1468] Processing CreateTablet for tablet 156a02312d6146e8a9ccd797cc8f6d59 (DEFAULT_TABLE table=default.Table_1ee4b42dee0f45299e1beacc5af5b301 [id=79515c0e3b4f4fb499718eee9d8c494b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.915479 29612 leader_election.cc:290] T 5f2eefeebbfc47249e2f72b7c52e905d P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:40.915716 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 156a02312d6146e8a9ccd797cc8f6d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.916036 27695 consensus_queue.cc:260] T f4dec89fc6bb436ba6542c267eaf9c15 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:40.921707 27495 tablet_service.cc:1468] Processing CreateTablet for tablet f3ab9ca3a11d4713899ca846d2fc3ef5 (DEFAULT_TABLE table=default.Table_0b669e82cd084404b53110469f908cd0 [id=0100a58caae04cb4bfdaa418b195ff4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.922255 27495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3ab9ca3a11d4713899ca846d2fc3ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.930841 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1262) took 2842 ms. Trace:
I20250903 20:05:40.930960 27684 rpcz_store.cc:276] 0903 20:05:38.088627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.088686 (+ 59us) service_pool.cc:225] Handling call
0903 20:05:40.785890 (+2697204us) ts_tablet_manager.cc:1145] Deleting tablet 1bf570855b204b96acd070587c434463
0903 20:05:40.785901 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.828241 (+ 42340us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.833294 (+ 5053us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.930818 (+ 97524us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2696987}
I20250903 20:05:40.931387 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1263) took 2803 ms. Trace:
I20250903 20:05:40.931468 27684 rpcz_store.cc:276] 0903 20:05:38.127600 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.127666 (+ 66us) service_pool.cc:225] Handling call
0903 20:05:40.931264 (+2803598us) ts_tablet_manager.cc:1145] Deleting tablet a7c8657489734199b346033bd4afb02e
0903 20:05:40.931274 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.931371 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2803360}
I20250903 20:05:40.931730 27684 tablet_replica.cc:331] T 06a79706f1184fa49d10182ba669b0ab P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:40.931931 27684 raft_consensus.cc:2241] T 06a79706f1184fa49d10182ba669b0ab P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:40.932127 27684 raft_consensus.cc:2270] T 06a79706f1184fa49d10182ba669b0ab P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:40.932650 27684 ts_tablet_manager.cc:1905] T 06a79706f1184fa49d10182ba669b0ab P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:40.920135 27184 consensus_peers.cc:489] T ed4902c002ef4198802375ced39ef2ad P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:40.920845 27244 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f2eefeebbfc47249e2f72b7c52e905d" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:40.934952 27462 leader_election.cc:343] T 5f2eefeebbfc47249e2f72b7c52e905d P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.935544 27695 ts_tablet_manager.cc:1428] T f4dec89fc6bb436ba6542c267eaf9c15 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.056s user 0.002s sys 0.000s
I20250903 20:05:40.935912 27695 tablet_bootstrap.cc:492] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:40.935920 27211 tablet_service.cc:1468] Processing CreateTablet for tablet 858106cce6c24ff7a44c497f65b8caa8 (DEFAULT_TABLE table=default.Table_76c9c4d442e7410990e7b588771507b5 [id=510f59b11dc84effb23ed8965e399dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.938954 27506 tablet_service.cc:1468] Processing CreateTablet for tablet 858106cce6c24ff7a44c497f65b8caa8 (DEFAULT_TABLE table=default.Table_76c9c4d442e7410990e7b588771507b5 [id=510f59b11dc84effb23ed8965e399dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.943455 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 1bf570855b204b96acd070587c434463 (table default.Table_3dee346162e54d5190c8e658e9483c9a [id=5ff9845a7d804adeaaee6199ac7faee6]) successfully deleted
I20250903 20:05:40.944111 27244 raft_consensus.cc:3058] T b4b15d77d07c4fa7856d9c198cf15450 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:40.944386 27685 ts_tablet_manager.cc:1918] T ed096b452ba747f89e91edc15e6e3ac1 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:40.944579 27685 log.cc:1199] T ed096b452ba747f89e91edc15e6e3ac1 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/ed096b452ba747f89e91edc15e6e3ac1
I20250903 20:05:40.945019 27685 ts_tablet_manager.cc:1939] T ed096b452ba747f89e91edc15e6e3ac1 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:40.945385 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1074) took 6167 ms. Trace:
I20250903 20:05:40.945539 27685 rpcz_store.cc:276] 0903 20:05:34.777963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.778022 (+ 59us) service_pool.cc:225] Handling call
0903 20:05:40.861362 (+6083340us) ts_tablet_manager.cc:1145] Deleting tablet ed096b452ba747f89e91edc15e6e3ac1
0903 20:05:40.861375 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.865110 (+ 3735us) spinlock_profiling.cc:243] Waited 2.83 ms on lock 0x6160000ca054. stack: 00007f589abe0dc6 00007f589abe0b80 00007f5899bc83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f58a666a600 00007f58a797db13 00007f58aa4602b7 00007f58aa492344 00007f58aa473da0 00007f58aa473b2d 0000000000a774dc 00007f589ac2714f 00007f589ac2a41b 00007f589ac2a298
0903 20:05:40.912334 (+ 47224us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.944273 (+ 31939us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.945366 (+ 1093us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6501248,"tablet-delete.queue_time_us":6083135}
I20250903 20:05:40.944674 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_cfb651c1215b4325bcd5237fb8c1d7d4" } modify_external_catalogs: true
I20250903 20:05:40.948066 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_cfb651c1215b4325bcd5237fb8c1d7d4" } modify_external_catalogs: true
I20250903 20:05:40.949853 27506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 858106cce6c24ff7a44c497f65b8caa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.952364 27296 heartbeater.cc:344] Connected to a master server at 127.26.65.125:44059
I20250903 20:05:40.956717 29748 sys_catalog.cc:263] Verifying existing consensus state
I20250903 20:05:40.946426 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet ed096b452ba747f89e91edc15e6e3ac1 (table default.Table_5bc4b150c12940db90c7800f5d07b9ed [id=27548347ece844e3bb55ba894da5c8e0]) successfully deleted
W20250903 20:05:40.948148 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet a7c8657489734199b346033bd4afb02e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7c8657489734199b346033bd4afb02e already in progress: creating tablet
I20250903 20:05:40.963681 27684 ts_tablet_manager.cc:1918] T 06a79706f1184fa49d10182ba669b0ab P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250903 20:05:40.963833 27684 log.cc:1199] T 06a79706f1184fa49d10182ba669b0ab P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/06a79706f1184fa49d10182ba669b0ab
I20250903 20:05:40.965528 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1075) took 6147 ms. Trace:
I20250903 20:05:40.965721 27685 rpcz_store.cc:276] 0903 20:05:34.817848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.817915 (+ 67us) service_pool.cc:225] Handling call
0903 20:05:40.965274 (+6147359us) ts_tablet_manager.cc:1145] Deleting tablet 7203b4ba9047430ab782d4c0498466cc
0903 20:05:40.965287 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.965499 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6147138}
W20250903 20:05:40.966451 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 7203b4ba9047430ab782d4c0498466cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7203b4ba9047430ab782d4c0498466cc already in progress: creating tablet
I20250903 20:05:40.966958 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1076) took 6124 ms. Trace:
I20250903 20:05:40.967103 27685 rpcz_store.cc:276] 0903 20:05:34.842386 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:34.842436 (+ 50us) service_pool.cc:225] Handling call
0903 20:05:40.966834 (+6124398us) ts_tablet_manager.cc:1145] Deleting tablet baac5b97f7eb4d18a0f294ca1595a383
0903 20:05:40.966845 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.966937 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6124159}
I20250903 20:05:40.967811 29748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35dd69f3fa7a4fd8843b97f8a5d68296: Bootstrap starting.
W20250903 20:05:40.967862 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet baac5b97f7eb4d18a0f294ca1595a383 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet baac5b97f7eb4d18a0f294ca1595a383 already in progress: creating tablet
I20250903 20:05:40.868011 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "878f6ad087614c15bc7e29c784f2c3c9" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:40.890650 27386 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b4b15d77d07c4fa7856d9c198cf15450" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:40.968473 27685 tablet_replica.cc:331] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:40.898900 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b84f487773c24735a61d0fd0898b2bcb" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:40.968672 27685 raft_consensus.cc:2241] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus shutting down.
W20250903 20:05:40.968946 27462 leader_election.cc:343] T 878f6ad087614c15bc7e29c784f2c3c9 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.969105 27462 leader_election.cc:304] T 878f6ad087614c15bc7e29c784f2c3c9 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:40.969501 29612 raft_consensus.cc:2747] T 878f6ad087614c15bc7e29c784f2c3c9 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:40.970110 27184 leader_election.cc:304] T b4b15d77d07c4fa7856d9c198cf15450 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:40.970577 27695 tablet_bootstrap.cc:654] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:40.904852 29670 consensus_queue.cc:1035] T 249362a0b8304719863bcb82c47273b3 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:40.910805 27693 tablet_bootstrap.cc:654] T 733f8024e1fa401ca9614a97db91a19b P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:40.915246 27362 tablet_service.cc:1468] Processing CreateTablet for tablet 156a02312d6146e8a9ccd797cc8f6d59 (DEFAULT_TABLE table=default.Table_1ee4b42dee0f45299e1beacc5af5b301 [id=79515c0e3b4f4fb499718eee9d8c494b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.920588 27376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f2eefeebbfc47249e2f72b7c52e905d" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:40.924655 27381 raft_consensus.cc:3058] T ea3f82f4e4244219ab0919f3e341aead P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:40.925045 27357 tablet_service.cc:1468] Processing CreateTablet for tablet f3ab9ca3a11d4713899ca846d2fc3ef5 (DEFAULT_TABLE table=default.Table_0b669e82cd084404b53110469f908cd0 [id=0100a58caae04cb4bfdaa418b195ff4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.972432 27357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3ab9ca3a11d4713899ca846d2fc3ef5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250903 20:05:40.973847 27462 leader_election.cc:343] T b84f487773c24735a61d0fd0898b2bcb P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.973986 27462 leader_election.cc:304] T b84f487773c24735a61d0fd0898b2bcb P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
W20250903 20:05:40.974378 27462 leader_election.cc:343] T 5f2eefeebbfc47249e2f72b7c52e905d P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:40.974463 27462 leader_election.cc:304] T 5f2eefeebbfc47249e2f72b7c52e905d P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:40.974752 29612 raft_consensus.cc:2747] T b84f487773c24735a61d0fd0898b2bcb P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:40.974901 29612 raft_consensus.cc:2747] T 5f2eefeebbfc47249e2f72b7c52e905d P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:40.926405 27361 tablet_service.cc:1468] Processing CreateTablet for tablet 858106cce6c24ff7a44c497f65b8caa8 (DEFAULT_TABLE table=default.Table_76c9c4d442e7410990e7b588771507b5 [id=510f59b11dc84effb23ed8965e399dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.975569 27361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 858106cce6c24ff7a44c497f65b8caa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.976351 27211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 858106cce6c24ff7a44c497f65b8caa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:40.968829 27685 raft_consensus.cc:2270] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:40.983290 27687 ts_tablet_manager.cc:1939] T 0fef9aa7153a4902abce522e3ff0df3f P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:40.983708 27685 ts_tablet_manager.cc:1905] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:40.984166 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1289) took 2428 ms. Trace:
I20250903 20:05:40.991781 27057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_781c5fddf2e945b5bf4c68483f03dde5" } modify_external_catalogs: true
I20250903 20:05:40.992046 27057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_781c5fddf2e945b5bf4c68483f03dde5" } modify_external_catalogs: true
I20250903 20:05:40.995882 27215 tablet_service.cc:1468] Processing CreateTablet for tablet f3ab9ca3a11d4713899ca846d2fc3ef5 (DEFAULT_TABLE table=default.Table_0b669e82cd084404b53110469f908cd0 [id=0100a58caae04cb4bfdaa418b195ff4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 20:05:40.984673 27362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 156a02312d6146e8a9ccd797cc8f6d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:41.003757 29765 consensus_queue.cc:1035] T ea3f82f4e4244219ab0919f3e341aead P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:41.000684 29669 consensus_queue.cc:1035] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20250903 20:05:40.998741 27693 tablet_bootstrap.cc:492] T 733f8024e1fa401ca9614a97db91a19b P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:41.012209 27693 ts_tablet_manager.cc:1397] T 733f8024e1fa401ca9614a97db91a19b P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.233s user 0.002s sys 0.003s
I20250903 20:05:41.013332 27693 raft_consensus.cc:357] T 733f8024e1fa401ca9614a97db91a19b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.013700 27693 raft_consensus.cc:383] T 733f8024e1fa401ca9614a97db91a19b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:41.013811 27693 raft_consensus.cc:738] T 733f8024e1fa401ca9614a97db91a19b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:41.014128 27693 consensus_queue.cc:260] T 733f8024e1fa401ca9614a97db91a19b P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.014854 27693 ts_tablet_manager.cc:1428] T 733f8024e1fa401ca9614a97db91a19b P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:41.015209 27693 tablet_bootstrap.cc:492] T ed4902c002ef4198802375ced39ef2ad P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:41.017030 27693 tablet_bootstrap.cc:654] T ed4902c002ef4198802375ced39ef2ad P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:41.018414 27692 tablet_bootstrap.cc:654] T e9126e4a18c546c28927f22c9f16426e P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:41.018479 27684 ts_tablet_manager.cc:1939] T 06a79706f1184fa49d10182ba669b0ab P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:41.019635 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1264) took 2882 ms. Trace:
I20250903 20:05:41.019759 27684 rpcz_store.cc:276] 0903 20:05:38.137690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.137762 (+ 72us) service_pool.cc:225] Handling call
0903 20:05:40.931690 (+2793928us) ts_tablet_manager.cc:1145] Deleting tablet 06a79706f1184fa49d10182ba669b0ab
0903 20:05:40.931698 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.959858 (+ 28160us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.963562 (+ 3704us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:41.019601 (+ 56039us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2793702}
I20250903 20:05:41.020304 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1269) took 2701 ms. Trace:
I20250903 20:05:41.020340 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 06a79706f1184fa49d10182ba669b0ab (table default.Table_1153bf74b285448b88b364efe2169784 [id=f321a32c1ac043238c2fc92bfd5ec1ae]) successfully deleted
I20250903 20:05:41.020390 27684 rpcz_store.cc:276] 0903 20:05:38.318820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.318884 (+ 64us) service_pool.cc:225] Handling call
0903 20:05:41.020141 (+2701257us) ts_tablet_manager.cc:1145] Deleting tablet a868191630454a1eb3a4301664178a9a
0903 20:05:41.020154 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:41.020286 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2701044}
I20250903 20:05:41.020761 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1270) took 2628 ms. Trace:
W20250903 20:05:41.020813 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet a868191630454a1eb3a4301664178a9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a868191630454a1eb3a4301664178a9a already in progress: creating tablet
I20250903 20:05:41.020839 27684 rpcz_store.cc:276] 0903 20:05:38.392349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.392538 (+ 189us) service_pool.cc:225] Handling call
0903 20:05:41.020661 (+2628123us) ts_tablet_manager.cc:1145] Deleting tablet ed73c335b042476fa03f154d23a611ad
0903 20:05:41.020669 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:41.020744 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2627871}
I20250903 20:05:41.021193 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1271) took 2598 ms. Trace:
W20250903 20:05:41.021214 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet ed73c335b042476fa03f154d23a611ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed73c335b042476fa03f154d23a611ad already in progress: creating tablet
I20250903 20:05:41.021246 27684 rpcz_store.cc:276] 0903 20:05:38.422521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.423022 (+ 501us) service_pool.cc:225] Handling call
0903 20:05:41.021101 (+2598079us) ts_tablet_manager.cc:1145] Deleting tablet ca722dfe38ac4e63b024508a3997cedb
0903 20:05:41.021111 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:41.021181 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2597839}
I20250903 20:05:41.021538 27684 tablet_replica.cc:331] T 793d6157927143d4b9d8dbafd7cf0fcb P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
W20250903 20:05:41.021591 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet ca722dfe38ac4e63b024508a3997cedb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca722dfe38ac4e63b024508a3997cedb already in progress: creating tablet
I20250903 20:05:41.021785 27684 raft_consensus.cc:2241] T 793d6157927143d4b9d8dbafd7cf0fcb P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:41.022030 27684 raft_consensus.cc:2270] T 793d6157927143d4b9d8dbafd7cf0fcb P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:41.023759 27684 ts_tablet_manager.cc:1905] T 793d6157927143d4b9d8dbafd7cf0fcb P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:41.030975 27684 ts_tablet_manager.cc:1918] T 793d6157927143d4b9d8dbafd7cf0fcb P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:41.031212 27684 log.cc:1199] T 793d6157927143d4b9d8dbafd7cf0fcb P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/793d6157927143d4b9d8dbafd7cf0fcb
I20250903 20:05:41.031749 27684 ts_tablet_manager.cc:1939] T 793d6157927143d4b9d8dbafd7cf0fcb P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:41.032645 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1274) took 2339 ms. Trace:
I20250903 20:05:40.984268 27687 rpcz_store.cc:276] 0903 20:05:38.555659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.555738 (+ 79us) service_pool.cc:225] Handling call
0903 20:05:40.644074 (+2088336us) ts_tablet_manager.cc:1145] Deleting tablet 0fef9aa7153a4902abce522e3ff0df3f
0903 20:05:40.644088 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.761322 (+117234us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.867629 (+106307us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:40.984141 (+116512us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2022683}
I20250903 20:05:41.033179 27687 tablet_replica.cc:331] T 4e9a6a6e05464b7b97ae624f51402cf0 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:41.033445 27687 raft_consensus.cc:2241] T 4e9a6a6e05464b7b97ae624f51402cf0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:41.033730 27687 raft_consensus.cc:2270] T 4e9a6a6e05464b7b97ae624f51402cf0 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:41.034365 27687 ts_tablet_manager.cc:1905] T 4e9a6a6e05464b7b97ae624f51402cf0 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:41.038034 28567 consensus_queue.cc:1035] T f4d3e1cfdfab4be1a9143b78bf3bc46d P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:41.038389 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 0fef9aa7153a4902abce522e3ff0df3f (table default.Table_c304f6eaf4014caba10a5b376c13972d [id=f8544f28af82478aa3762f84b040a6d4]) successfully deleted
I20250903 20:05:41.011395 27685 ts_tablet_manager.cc:1918] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:41.038707 27685 log.cc:1199] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/45f7068bafaf48d290ded9d1272c0e46
I20250903 20:05:41.039068 27685 ts_tablet_manager.cc:1939] T 45f7068bafaf48d290ded9d1272c0e46 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:41.039371 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1085) took 5974 ms. Trace:
I20250903 20:05:41.039443 27685 rpcz_store.cc:276] 0903 20:05:35.064775 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.070027 (+ 5252us) service_pool.cc:225] Handling call
0903 20:05:40.968425 (+5898398us) ts_tablet_manager.cc:1145] Deleting tablet 45f7068bafaf48d290ded9d1272c0e46
0903 20:05:40.968435 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:40.987266 (+ 18831us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:41.011267 (+ 24001us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:41.039355 (+ 28088us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5897846}
I20250903 20:05:41.039875 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1088) took 5912 ms. Trace:
I20250903 20:05:41.039966 27685 rpcz_store.cc:276] 0903 20:05:35.127377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.145542 (+ 18165us) service_pool.cc:225] Handling call
0903 20:05:41.039716 (+5894174us) ts_tablet_manager.cc:1145] Deleting tablet 20d826f1b1d64746bd8e409a12b74eb1
0903 20:05:41.039724 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:41.039855 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5893936}
I20250903 20:05:41.040339 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1089) took 5880 ms. Trace:
I20250903 20:05:41.040426 27685 rpcz_store.cc:276] 0903 20:05:35.160284 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.161996 (+ 1712us) service_pool.cc:225] Handling call
0903 20:05:41.040221 (+5878225us) ts_tablet_manager.cc:1145] Deleting tablet f28fd3e1e0e14cf48a244580b4f355d0
0903 20:05:41.040232 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:41.040322 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5877890}
I20250903 20:05:41.040802 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1090) took 5831 ms. Trace:
I20250903 20:05:41.040879 27685 rpcz_store.cc:276] 0903 20:05:35.209000 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.230292 (+ 21292us) service_pool.cc:225] Handling call
0903 20:05:41.040690 (+5810398us) ts_tablet_manager.cc:1145] Deleting tablet 69f8eef52bee431a86f0eef45fd11f9f
0903 20:05:41.040701 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:41.040787 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5801409}
I20250903 20:05:41.041199 27685 tablet_replica.cc:331] T eaa11d17303142ff935705c16f0b9690 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:41.041460 27685 raft_consensus.cc:2241] T eaa11d17303142ff935705c16f0b9690 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:41.041651 27685 raft_consensus.cc:2270] T eaa11d17303142ff935705c16f0b9690 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:41.025383 29411 raft_consensus.cc:491] T 238f2f30377b42c2852b016018aa3245 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51578fb7bceb4e119b3ab1e58c4f5528)
I20250903 20:05:41.042330 29411 raft_consensus.cc:513] T 238f2f30377b42c2852b016018aa3245 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.043161 29411 leader_election.cc:290] T 238f2f30377b42c2852b016018aa3245 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
W20250903 20:05:41.044472 27053 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 1ed519e2e9c94fa3a134dcfc8d5364e6 (table default.Table_2329720f2aaa431fb0a3ad5faf6d62ac [id=6c8ffb4fd82644f6a75a57bb196c6d6d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 1ed519e2e9c94fa3a134dcfc8d5364e6 consensus state has no leader
W20250903 20:05:41.044972 27183 leader_election.cc:343] T 238f2f30377b42c2852b016018aa3245 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 238f2f30377b42c2852b016018aa3245
W20250903 20:05:41.046567 27184 leader_election.cc:343] T 238f2f30377b42c2852b016018aa3245 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 238f2f30377b42c2852b016018aa3245
I20250903 20:05:41.046682 27184 leader_election.cc:304] T 238f2f30377b42c2852b016018aa3245 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:41.047058 29411 raft_consensus.cc:2747] T 238f2f30377b42c2852b016018aa3245 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:41.047837 27685 ts_tablet_manager.cc:1905] T eaa11d17303142ff935705c16f0b9690 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:41.054373 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 45f7068bafaf48d290ded9d1272c0e46 (table default.Table_7179a3a012834e34b703156b9c834722 [id=e660a48b356a4b0784016e527303f1f6]) successfully deleted
W20250903 20:05:41.054680 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 20d826f1b1d64746bd8e409a12b74eb1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20d826f1b1d64746bd8e409a12b74eb1 already in progress: creating tablet
W20250903 20:05:41.054863 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet f28fd3e1e0e14cf48a244580b4f355d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f28fd3e1e0e14cf48a244580b4f355d0 already in progress: creating tablet
W20250903 20:05:41.055083 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 69f8eef52bee431a86f0eef45fd11f9f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69f8eef52bee431a86f0eef45fd11f9f already in progress: creating tablet
I20250903 20:05:41.055645 27215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3ab9ca3a11d4713899ca846d2fc3ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 20:05:41.078074 29411 raft_consensus.cc:491] T 314856bf269f4df7aa25f5a30cec0d5b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:41.078311 29411 raft_consensus.cc:513] T 314856bf269f4df7aa25f5a30cec0d5b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.079176 29411 leader_election.cc:290] T 314856bf269f4df7aa25f5a30cec0d5b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:41.080073 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "314856bf269f4df7aa25f5a30cec0d5b" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
W20250903 20:05:41.080776 27184 leader_election.cc:343] T 314856bf269f4df7aa25f5a30cec0d5b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:41.083595 27685 ts_tablet_manager.cc:1918] T eaa11d17303142ff935705c16f0b9690 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:41.094254 27433 heartbeater.cc:344] Connected to a master server at 127.26.65.125:44059
I20250903 20:05:41.095189 29669 raft_consensus.cc:491] T 7d5a3fbe93e44c228222ab1b1e78c187 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:41.095319 29669 raft_consensus.cc:513] T 7d5a3fbe93e44c228222ab1b1e78c187 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.095932 29669 leader_election.cc:290] T 7d5a3fbe93e44c228222ab1b1e78c187 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:41.120898 27685 log.cc:1199] T eaa11d17303142ff935705c16f0b9690 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/eaa11d17303142ff935705c16f0b9690
I20250903 20:05:41.097147 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d5a3fbe93e44c228222ab1b1e78c187" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:41.112211 29461 consensus_queue.cc:1035] T f9375fff5f5945078c4412aa21c06f0d P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:41.121490 27685 ts_tablet_manager.cc:1939] T eaa11d17303142ff935705c16f0b9690 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:41.091948 27695 tablet_bootstrap.cc:492] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:41.121747 27695 ts_tablet_manager.cc:1397] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.186s user 0.003s sys 0.007s
I20250903 20:05:41.121795 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1091) took 5841 ms. Trace:
W20250903 20:05:41.121774 27321 leader_election.cc:343] T 7d5a3fbe93e44c228222ab1b1e78c187 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:41.121863 27685 rpcz_store.cc:276] 0903 20:05:35.280194 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.286278 (+ 6084us) service_pool.cc:225] Handling call
0903 20:05:41.041121 (+5754843us) ts_tablet_manager.cc:1145] Deleting tablet eaa11d17303142ff935705c16f0b9690
0903 20:05:41.041155 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:41.052290 (+ 11135us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:41.083476 (+ 31186us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:41.121781 (+ 38305us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5753167}
I20250903 20:05:41.122303 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1092) took 5777 ms. Trace:
I20250903 20:05:41.122385 27685 rpcz_store.cc:276] 0903 20:05:35.345067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.356868 (+ 11801us) service_pool.cc:225] Handling call
0903 20:05:41.122160 (+5765292us) ts_tablet_manager.cc:1145] Deleting tablet eb878263eda9447e870a782907fe6a8d
0903 20:05:41.122170 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:41.122288 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5765073}
I20250903 20:05:41.122553 27695 raft_consensus.cc:357] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.122674 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1093) took 5670 ms. Trace:
I20250903 20:05:41.122705 27695 raft_consensus.cc:383] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:41.122747 27695 raft_consensus.cc:738] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:41.122910 27695 consensus_queue.cc:260] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.123359 27695 ts_tablet_manager.cc:1428] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:41.123596 27695 tablet_bootstrap.cc:492] T 6d13884fff8c43b690836c6a341a2d92 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:41.125592 27695 tablet_bootstrap.cc:654] T 6d13884fff8c43b690836c6a341a2d92 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:41.126861 27058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 5175dbfc0c8a4c438524cff8556dded0
I20250903 20:05:41.128314 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b7d2f6c7831140dd9e0d3f49c3c3f7ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:41.128870 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7d2f6c7831140dd9e0d3f49c3c3f7ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:41.129189 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5d2480e7b40245e488dedd3c6a5add50" } modify_external_catalogs: true
I20250903 20:05:41.129328 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5d2480e7b40245e488dedd3c6a5add50" } modify_external_catalogs: true
W20250903 20:05:41.131130 27066 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 31edb50c250f42739f82cdaaf1525266 consensus state has no leader
I20250903 20:05:41.131482 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_476f1a4abacf42c5b39d8ea938f9babc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:41.131963 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_476f1a4abacf42c5b39d8ea938f9babc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:41.132006 27695 tablet_bootstrap.cc:492] T 6d13884fff8c43b690836c6a341a2d92 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:41.132121 27695 ts_tablet_manager.cc:1397] T 6d13884fff8c43b690836c6a341a2d92 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20250903 20:05:41.132854 27695 raft_consensus.cc:357] T 6d13884fff8c43b690836c6a341a2d92 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.133015 27695 raft_consensus.cc:383] T 6d13884fff8c43b690836c6a341a2d92 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:41.133070 27695 raft_consensus.cc:738] T 6d13884fff8c43b690836c6a341a2d92 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:41.133288 27695 consensus_queue.cc:260] T 6d13884fff8c43b690836c6a341a2d92 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.133994 27695 ts_tablet_manager.cc:1428] T 6d13884fff8c43b690836c6a341a2d92 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:41.133931 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_bcd1ecf3ed1543db8c20807203abf4f6" } new_table_name: "default.Table_6e0953d2909a4e38bc92a9c44a3ac979" modify_external_catalogs: true
I20250903 20:05:41.134362 27695 tablet_bootstrap.cc:492] T 733f8024e1fa401ca9614a97db91a19b P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:41.136085 27695 tablet_bootstrap.cc:654] T 733f8024e1fa401ca9614a97db91a19b P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:41.136844 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet eaa11d17303142ff935705c16f0b9690 (table default.Table_857612f25ca54f8b8d0ea31485d96a5d [id=ddebc54a9f9849478f0d1606dfe3bc96]) successfully deleted
W20250903 20:05:41.137194 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet eb878263eda9447e870a782907fe6a8d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb878263eda9447e870a782907fe6a8d already in progress: creating tablet
I20250903 20:05:41.137710 27211 tablet_service.cc:1515] Processing DeleteTablet for tablet 5175dbfc0c8a4c438524cff8556dded0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:40 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:41.140128 29669 raft_consensus.cc:491] T 54f1ce5192f547f595dea23a9eabf8e6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:41.140291 29669 raft_consensus.cc:513] T 54f1ce5192f547f595dea23a9eabf8e6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.140537 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 5175dbfc0c8a4c438524cff8556dded0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:40 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:41.140924 29669 leader_election.cc:290] T 54f1ce5192f547f595dea23a9eabf8e6 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:41.144774 27321 leader_election.cc:343] T 54f1ce5192f547f595dea23a9eabf8e6 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 54f1ce5192f547f595dea23a9eabf8e6
I20250903 20:05:41.122735 27685 rpcz_store.cc:276] 0903 20:05:35.452106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.452191 (+ 85us) service_pool.cc:225] Handling call
0903 20:05:41.122584 (+5670393us) ts_tablet_manager.cc:1145] Deleting tablet a7f25f00dff7426195ac51214a14022b
0903 20:05:41.122592 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:41.122660 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5670156}
W20250903 20:05:41.152105 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet a7f25f00dff7426195ac51214a14022b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7f25f00dff7426195ac51214a14022b already in progress: creating tablet
I20250903 20:05:41.152905 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1094) took 5663 ms. Trace:
I20250903 20:05:41.153625 27685 rpcz_store.cc:276] 0903 20:05:35.489321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.489398 (+ 77us) service_pool.cc:225] Handling call
0903 20:05:41.152714 (+5663316us) ts_tablet_manager.cc:1145] Deleting tablet c9d09ecd8fe347d79283fc9a3056e844
0903 20:05:41.152730 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:41.152882 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5646071}
W20250903 20:05:41.154434 27052 catalog_manager.cc:4908] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet c9d09ecd8fe347d79283fc9a3056e844 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9d09ecd8fe347d79283fc9a3056e844 already in progress: creating tablet
I20250903 20:05:41.155174 27685 tablet_replica.cc:331] T 46f49fd718be4a2996d781ab31268217 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:41.159127 27695 tablet_bootstrap.cc:492] T 733f8024e1fa401ca9614a97db91a19b P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:41.159277 27695 ts_tablet_manager.cc:1397] T 733f8024e1fa401ca9614a97db91a19b P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.002s
I20250903 20:05:41.160027 27695 raft_consensus.cc:357] T 733f8024e1fa401ca9614a97db91a19b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.160202 27695 raft_consensus.cc:383] T 733f8024e1fa401ca9614a97db91a19b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:41.160257 27695 raft_consensus.cc:738] T 733f8024e1fa401ca9614a97db91a19b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:41.160429 27695 consensus_queue.cc:260] T 733f8024e1fa401ca9614a97db91a19b P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.160550 29411 consensus_queue.cc:1035] T f9375fff5f5945078c4412aa21c06f0d P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:41.160905 27695 ts_tablet_manager.cc:1428] T 733f8024e1fa401ca9614a97db91a19b P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250903 20:05:41.161196 27695 tablet_bootstrap.cc:492] T 60c32370a5274347afafc6a7e1e94270 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
W20250903 20:05:41.162508 27184 consensus_peers.cc:489] T 3f303233e6c347289650c6cf83f8127f P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f303233e6c347289650c6cf83f8127f. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:41.052187 29371 consensus_queue.cc:1035] T ea3f82f4e4244219ab0919f3e341aead P e72ef1dbcc5143cf9f04a808f101c80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:41.067957 29765 raft_consensus.cc:491] T 41056c5c5003415993bee74d7621bd14 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:41.163614 29765 raft_consensus.cc:513] T 41056c5c5003415993bee74d7621bd14 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.164050 27684 rpcz_store.cc:276] 0903 20:05:38.692671 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.692719 (+ 48us) service_pool.cc:225] Handling call
0903 20:05:41.021487 (+2328768us) ts_tablet_manager.cc:1145] Deleting tablet 793d6157927143d4b9d8dbafd7cf0fcb
0903 20:05:41.021497 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:41.027152 (+ 5655us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:41.030852 (+ 3700us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:41.032624 (+ 1772us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2328545}
I20250903 20:05:41.081441 27509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "314856bf269f4df7aa25f5a30cec0d5b" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:41.164634 27509 raft_consensus.cc:2466] T 314856bf269f4df7aa25f5a30cec0d5b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:41.093922 27574 heartbeater.cc:344] Connected to a master server at 127.26.65.125:44059
I20250903 20:05:41.165284 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 793d6157927143d4b9d8dbafd7cf0fcb (table default.Table_e79f52615fe142b9823a2a99b8b9c406 [id=3432f6ac076b4e49b9fe63bf5fce8591]) successfully deleted
I20250903 20:05:41.096913 27516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d5a3fbe93e44c228222ab1b1e78c187" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:41.115085 27692 tablet_bootstrap.cc:492] T e9126e4a18c546c28927f22c9f16426e P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:41.165598 27516 raft_consensus.cc:2466] T 7d5a3fbe93e44c228222ab1b1e78c187 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:41.165661 27692 ts_tablet_manager.cc:1397] T e9126e4a18c546c28927f22c9f16426e P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.291s user 0.000s sys 0.004s
I20250903 20:05:41.165652 29774 tablet.cc:1722] T b4b15d77d07c4fa7856d9c198cf15450 P 89f8f0ee68b940db9ce250c0604f934a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250903 20:05:41.166465 27692 raft_consensus.cc:357] T e9126e4a18c546c28927f22c9f16426e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:41.166621 27692 raft_consensus.cc:383] T e9126e4a18c546c28927f22c9f16426e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:41.166683 27692 raft_consensus.cc:738] T e9126e4a18c546c28927f22c9f16426e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:41.166870 27692 consensus_queue.cc:260] T e9126e4a18c546c28927f22c9f16426e P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:41.167023 27183 leader_election.cc:304] T 314856bf269f4df7aa25f5a30cec0d5b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528
I20250903 20:05:41.167084 27693 tablet_bootstrap.cc:492] T ed4902c002ef4198802375ced39ef2ad P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:41.167222 27693 ts_tablet_manager.cc:1397] T ed4902c002ef4198802375ced39ef2ad P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.152s user 0.003s sys 0.000s
I20250903 20:05:41.167266 27692 ts_tablet_manager.cc:1428] T e9126e4a18c546c28927f22c9f16426e P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:41.127651 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet 5175dbfc0c8a4c438524cff8556dded0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:40 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:41.167917 27693 raft_consensus.cc:357] T ed4902c002ef4198802375ced39ef2ad P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.168097 27693 raft_consensus.cc:383] T ed4902c002ef4198802375ced39ef2ad P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:41.168154 27693 raft_consensus.cc:738] T ed4902c002ef4198802375ced39ef2ad P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:41.168352 27693 consensus_queue.cc:260] T ed4902c002ef4198802375ced39ef2ad P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.168983 29411 raft_consensus.cc:2802] T 314856bf269f4df7aa25f5a30cec0d5b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:41.169078 29411 raft_consensus.cc:491] T 314856bf269f4df7aa25f5a30cec0d5b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:41.169137 29411 raft_consensus.cc:3058] T 314856bf269f4df7aa25f5a30cec0d5b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:41.169281 27693 ts_tablet_manager.cc:1428] T ed4902c002ef4198802375ced39ef2ad P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250903 20:05:41.169456 27695 tablet_bootstrap.cc:654] T 60c32370a5274347afafc6a7e1e94270 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:41.169567 27693 tablet_bootstrap.cc:492] T 43fb27c857d543b7b78ea6108c406085 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:41.169597 29589 consensus_queue.cc:1035] T 13496efe34944b7280e7a684f180e6e0 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:41.160137 27510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f1ce5192f547f595dea23a9eabf8e6" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:41.170176 27510 raft_consensus.cc:2466] T 54f1ce5192f547f595dea23a9eabf8e6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:41.171059 27693 tablet_bootstrap.cc:654] T 43fb27c857d543b7b78ea6108c406085 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:41.171828 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1275) took 2413 ms. Trace:
I20250903 20:05:41.171893 29411 raft_consensus.cc:513] T 314856bf269f4df7aa25f5a30cec0d5b P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.171914 27684 rpcz_store.cc:276] 0903 20:05:38.758820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.758908 (+ 88us) service_pool.cc:225] Handling call
0903 20:05:41.165126 (+2406218us) ts_tablet_manager.cc:1145] Deleting tablet 3f08c7edbcf54c16bc222f59773f72b1
0903 20:05:41.171645 (+ 6519us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:41.171802 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1408,"tablet-delete.queue_time_us":2405954}
W20250903 20:05:41.172495 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 3f08c7edbcf54c16bc222f59773f72b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f08c7edbcf54c16bc222f59773f72b1 already in progress: creating tablet
I20250903 20:05:41.173046 27510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "314856bf269f4df7aa25f5a30cec0d5b" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:41.173226 27510 raft_consensus.cc:3058] T 314856bf269f4df7aa25f5a30cec0d5b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:41.173595 29452 raft_consensus.cc:2747] T b4b15d77d07c4fa7856d9c198cf15450 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:41.164208 29765 leader_election.cc:290] T 41056c5c5003415993bee74d7621bd14 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:41.077303 29772 raft_consensus.cc:491] T 54602d7f723d4c10af12408a43ed1f02 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:41.176082 29772 raft_consensus.cc:513] T 54602d7f723d4c10af12408a43ed1f02 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.176702 29772 leader_election.cc:290] T 54602d7f723d4c10af12408a43ed1f02 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:41.176808 27510 raft_consensus.cc:2466] T 314856bf269f4df7aa25f5a30cec0d5b P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:41.179159 29411 leader_election.cc:290] T 314856bf269f4df7aa25f5a30cec0d5b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:41.185179 27692 tablet_bootstrap.cc:492] T 4fa886f5be0f489290921e61dcaa4916 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:41.187011 27692 tablet_bootstrap.cc:654] T 4fa886f5be0f489290921e61dcaa4916 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:41.187150 27060 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 7e7390249fd045f8aec9a8e4e9573c7a
I20250903 20:05:41.188935 27183 leader_election.cc:304] T 314856bf269f4df7aa25f5a30cec0d5b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:41.189265 29411 raft_consensus.cc:2802] T 314856bf269f4df7aa25f5a30cec0d5b P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:41.189499 29411 raft_consensus.cc:695] T 314856bf269f4df7aa25f5a30cec0d5b P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:41.189742 29411 consensus_queue.cc:237] T 314856bf269f4df7aa25f5a30cec0d5b P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:41.190770 27183 consensus_peers.cc:489] T 13496efe34944b7280e7a684f180e6e0 P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 13496efe34944b7280e7a684f180e6e0. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:41.191877 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e7390249fd045f8aec9a8e4e9573c7a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:40 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:41.194795 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "314856bf269f4df7aa25f5a30cec0d5b" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:41.195108 27064 catalog_manager.cc:5674] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231) does not have the latest schema for tablet 43fb27c857d543b7b78ea6108c406085 (table default.Table_61979f075b5b4555b56d523ab4931364 [id=95e5cb3083814ab996fce9dc666c405e]). Expected version 2 got 0
I20250903 20:05:41.195545 27065 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: default.Table_aabc7101f3e24a189aee5e863a629263 [id=52f969e50b9d466cbd14de6f252b7706] alter complete (version 2)
W20250903 20:05:41.197691 27065 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.26.65.65:44885 (request call id 200) took 827 ms (client timeout 1000 ms). Trace:
W20250903 20:05:41.197834 27184 leader_election.cc:343] T 314856bf269f4df7aa25f5a30cec0d5b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:41.197808 27065 rpcz_store.cc:269] 0903 20:05:40.370523 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:40.376934 (+ 6411us) service_pool.cc:225] Handling call
0903 20:05:41.197666 (+820732us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0903 20:05:40.877467 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:40.877582 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:40.877590 (+ 8us) write_op.cc:435] Acquired schema lock
0903 20:05:40.877594 (+ 4us) tablet.cc:660] Decoding operations
0903 20:05:40.877713 (+ 119us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:40.877724 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:40.877729 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0903 20:05:40.877799 (+ 70us) tablet.cc:699] Row locks acquired
0903 20:05:40.877803 (+ 4us) write_op.cc:260] PREPARE: finished
0903 20:05:40.877851 (+ 48us) write_op.cc:270] Start()
0903 20:05:40.877901 (+ 50us) write_op.cc:276] Timestamp: P: 1756929940877792 usec, L: 0
0903 20:05:40.877906 (+ 5us) op_driver.cc:348] REPLICATION: starting
0903 20:05:40.878253 (+ 347us) log.cc:844] Serialized 530 byte log entry
0903 20:05:40.894305 (+ 16052us) op_driver.cc:464] REPLICATION: finished
0903 20:05:40.894459 (+ 154us) write_op.cc:301] APPLY: starting
0903 20:05:40.894482 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:40.894537 (+ 55us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:40.894567 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:40.894646 (+ 79us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:40.894689 (+ 43us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0903 20:05:40.894697 (+ 8us) write_op.cc:312] APPLY: finished
0903 20:05:40.894818 (+ 121us) log.cc:844] Serialized 23 byte log entry
0903 20:05:40.894892 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:40.894917 (+ 25us) write_op.cc:454] Released schema lock
0903 20:05:40.894924 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0903 20:05:40.925077 (+ 30153us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:40.925088 (+ 11us) write_op.cc:454] Released schema lock
Related trace 'op':
0903 20:05:41.135311 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0903 20:05:41.135416 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0903 20:05:41.135421 (+ 5us) write_op.cc:435] Acquired schema lock
0903 20:05:41.135423 (+ 2us) tablet.cc:660] Decoding operations
0903 20:05:41.135528 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
0903 20:05:41.135543 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0903 20:05:41.135547 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0903 20:05:41.135607 (+ 60us) tablet.cc:699] Row locks acquired
0903 20:05:41.135610 (+ 3us) write_op.cc:260] PREPARE: finished
0903 20:05:41.135638 (+ 28us) write_op.cc:270] Start()
0903 20:05:41.135667 (+ 29us) write_op.cc:276] Timestamp: P: 1756929941135601 usec, L: 0
0903 20:05:41.135670 (+ 3us) op_driver.cc:348] REPLICATION: starting
0903 20:05:41.135828 (+ 158us) log.cc:844] Serialized 356 byte log entry
0903 20:05:41.151285 (+ 15457us) op_driver.cc:464] REPLICATION: finished
0903 20:05:41.152824 (+ 1539us) write_op.cc:301] APPLY: starting
0903 20:05:41.152840 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0903 20:05:41.152880 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0903 20:05:41.152909 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0903 20:05:41.152964 (+ 55us) tablet.cc:1381] finished ApplyRowOperation cycle
0903 20:05:41.152994 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0903 20:05:41.153001 (+ 7us) write_op.cc:312] APPLY: finished
0903 20:05:41.153079 (+ 78us) log.cc:844] Serialized 23 byte log entry
0903 20:05:41.154010 (+ 931us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:41.154030 (+ 20us) write_op.cc:454] Released schema lock
0903 20:05:41.154039 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0903 20:05:41.154789 (+ 750us) write_op.cc:489] Releasing partition, row and schema locks
0903 20:05:41.154797 (+ 8us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":19,"child_traces":[["op",{"apply.queue_time_us":91,"apply.run_cpu_time_us":553,"apply.run_wall_time_us":552,"num_ops":1,"prepare.queue_time_us":209798,"prepare.run_cpu_time_us":1033,"prepare.run_wall_time_us":1520,"replication_time_us":16354}],["op",{"apply.queue_time_us":1463,"apply.run_cpu_time_us":457,"apply.run_wall_time_us":1301,"num_ops":1,"prepare.queue_time_us":230112,"prepare.run_cpu_time_us":656,"prepare.run_wall_time_us":19466,"raft.queue_time_us":6996,"raft.run_cpu_time_us":319,"raft.run_wall_time_us":319,"replication_time_us":15569}]]}
I20250903 20:05:41.213491 27692 tablet_bootstrap.cc:492] T 4fa886f5be0f489290921e61dcaa4916 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:41.213933 27692 ts_tablet_manager.cc:1397] T 4fa886f5be0f489290921e61dcaa4916 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20250903 20:05:41.214109 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet 3919f7f95a954f59b68f56d0051c0109 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:19 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:41.214882 27692 raft_consensus.cc:357] T 4fa886f5be0f489290921e61dcaa4916 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.215090 27692 raft_consensus.cc:383] T 4fa886f5be0f489290921e61dcaa4916 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:41.215150 27692 raft_consensus.cc:738] T 4fa886f5be0f489290921e61dcaa4916 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:41.215353 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e7390249fd045f8aec9a8e4e9573c7a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:40 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:41.215405 27692 consensus_queue.cc:260] T 4fa886f5be0f489290921e61dcaa4916 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.215637 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet 6800e0b6269045f29682704fd847d5d1 with delete_type TABLET_DATA_DELETED (replaced by tablet a3cf18a9a8c94a58bb739ee2f81d19c0) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:41.215996 27217 tablet_service.cc:1515] Processing DeleteTablet for tablet f9375fff5f5945078c4412aa21c06f0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:41.216117 27224 tablet_service.cc:1515] Processing DeleteTablet for tablet ea3f82f4e4244219ab0919f3e341aead with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:41.216261 27211 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e7390249fd045f8aec9a8e4e9573c7a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:40 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:41.216400 27224 tablet_service.cc:1515] Processing DeleteTablet for tablet 13496efe34944b7280e7a684f180e6e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:41.216699 27224 tablet_service.cc:1515] Processing DeleteTablet for tablet ed4902c002ef4198802375ced39ef2ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:41.217536 27692 ts_tablet_manager.cc:1428] T 4fa886f5be0f489290921e61dcaa4916 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250903 20:05:41.217909 27692 tablet_bootstrap.cc:492] T 64d53f2071b0449ca61caa6c264a3641 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:41.219403 29589 raft_consensus.cc:491] T 733f8024e1fa401ca9614a97db91a19b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:41.219533 29589 raft_consensus.cc:513] T 733f8024e1fa401ca9614a97db91a19b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.220084 29589 leader_election.cc:290] T 733f8024e1fa401ca9614a97db91a19b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:41.220312 27692 tablet_bootstrap.cc:654] T 64d53f2071b0449ca61caa6c264a3641 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:41.220880 27510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "733f8024e1fa401ca9614a97db91a19b" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:41.221076 27510 raft_consensus.cc:2466] T 733f8024e1fa401ca9614a97db91a19b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:41.221359 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c42d84a6a5649208b0539a10da21aec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:24 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:41.221532 27183 leader_election.cc:304] T 733f8024e1fa401ca9614a97db91a19b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:41.221838 29589 raft_consensus.cc:2802] T 733f8024e1fa401ca9614a97db91a19b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:41.221909 29589 raft_consensus.cc:491] T 733f8024e1fa401ca9614a97db91a19b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:41.221904 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bf570855b204b96acd070587c434463 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:41.221993 29589 raft_consensus.cc:3058] T 733f8024e1fa401ca9614a97db91a19b P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:41.222361 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet 60c32370a5274347afafc6a7e1e94270 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:16 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:41.222784 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet 13496efe34944b7280e7a684f180e6e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:28 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:41.223196 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet f4dec89fc6bb436ba6542c267eaf9c15 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:41.223770 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1276) took 2365 ms. Trace:
I20250903 20:05:41.223856 27684 rpcz_store.cc:276] 0903 20:05:38.858295 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.858353 (+ 58us) service_pool.cc:225] Handling call
0903 20:05:41.172273 (+2313920us) ts_tablet_manager.cc:1145] Deleting tablet 421dc8af40c94a89bc372e8779872a4e
0903 20:05:41.223471 (+ 51198us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:41.223615 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2313726}
W20250903 20:05:41.224380 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet 421dc8af40c94a89bc372e8779872a4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 421dc8af40c94a89bc372e8779872a4e already in progress: creating tablet
I20250903 20:05:41.224818 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1277) took 2320 ms. Trace:
I20250903 20:05:41.224890 27684 rpcz_store.cc:276] 0903 20:05:38.904680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.904750 (+ 70us) service_pool.cc:225] Handling call
0903 20:05:41.224686 (+2319936us) ts_tablet_manager.cc:1145] Deleting tablet d41de82304bf453fab8b15d96cf32a74
0903 20:05:41.224696 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:41.224800 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2319720}
W20250903 20:05:41.225310 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet d41de82304bf453fab8b15d96cf32a74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d41de82304bf453fab8b15d96cf32a74 already in progress: creating tablet
I20250903 20:05:41.225857 27684 tablet_replica.cc:331] T 8e12f4c420874ab382d433982bddbf05 P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:41.226048 27684 raft_consensus.cc:2241] T 8e12f4c420874ab382d433982bddbf05 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:41.226197 27684 raft_consensus.cc:2270] T 8e12f4c420874ab382d433982bddbf05 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:41.226172 29589 raft_consensus.cc:513] T 733f8024e1fa401ca9614a97db91a19b P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.226789 27684 ts_tablet_manager.cc:1905] T 8e12f4c420874ab382d433982bddbf05 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:41.227595 29589 leader_election.cc:290] T 733f8024e1fa401ca9614a97db91a19b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:41.228415 27510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "733f8024e1fa401ca9614a97db91a19b" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:41.228585 27510 raft_consensus.cc:3058] T 733f8024e1fa401ca9614a97db91a19b P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:41.229111 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet f9375fff5f5945078c4412aa21c06f0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:22 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:41.229770 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet 0fef9aa7153a4902abce522e3ff0df3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:41.230281 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet 17f16e9c8f794e03a2f8b3b6b35f8e84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:36 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:41.230949 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "733f8024e1fa401ca9614a97db91a19b" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:41.231182 27379 raft_consensus.cc:2466] T 733f8024e1fa401ca9614a97db91a19b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:41.231734 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "733f8024e1fa401ca9614a97db91a19b" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:41.231902 27379 raft_consensus.cc:3058] T 733f8024e1fa401ca9614a97db91a19b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:41.235436 27379 raft_consensus.cc:2466] T 733f8024e1fa401ca9614a97db91a19b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:41.236322 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_2b93be051e10447e94101bca27b9c6f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:41.237056 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b93be051e10447e94101bca27b9c6f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:41.238562 27184 leader_election.cc:304] T 733f8024e1fa401ca9614a97db91a19b P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:41.238603 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_6f2040bdf8524136baca3a90d5635bd3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:41.239132 29452 raft_consensus.cc:2802] T 733f8024e1fa401ca9614a97db91a19b P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
W20250903 20:05:41.239225 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f2040bdf8524136baca3a90d5635bd3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:41.239264 29452 raft_consensus.cc:695] T 733f8024e1fa401ca9614a97db91a19b P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:41.239512 29452 consensus_queue.cc:237] T 733f8024e1fa401ca9614a97db91a19b P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.240485 27063 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_3110722931cd47ad8e82109243b34fc1" } new_table_name: "default.Table_b8f3dad456054b9b99d38963ebe132e4" modify_external_catalogs: true
I20250903 20:05:41.242403 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_285fd2ca2ca84683a5dc8f9dac2fdf43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:41.242544 29461 consensus_queue.cc:1035] T 5b46bb34ac214b1ba23fba1a33f746ac P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250903 20:05:41.243189 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_285fd2ca2ca84683a5dc8f9dac2fdf43 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:41.251363 29461 consensus_queue.cc:1035] T 5b46bb34ac214b1ba23fba1a33f746ac P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:41.252346 27692 tablet_bootstrap.cc:492] T 64d53f2071b0449ca61caa6c264a3641 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:41.252501 27692 ts_tablet_manager.cc:1397] T 64d53f2071b0449ca61caa6c264a3641 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.001s
I20250903 20:05:41.253449 27692 raft_consensus.cc:357] T 64d53f2071b0449ca61caa6c264a3641 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.253628 27692 raft_consensus.cc:383] T 64d53f2071b0449ca61caa6c264a3641 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:41.253686 27692 raft_consensus.cc:738] T 64d53f2071b0449ca61caa6c264a3641 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:41.253872 27692 consensus_queue.cc:260] T 64d53f2071b0449ca61caa6c264a3641 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.257540 27687 ts_tablet_manager.cc:1918] T 4e9a6a6e05464b7b97ae624f51402cf0 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:41.259791 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e9a6a6e05464b7b97ae624f51402cf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:41.260843 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d21f38759d743af972635f8b5b8d8d2 with delete_type TABLET_DATA_DELETED (replaced by tablet fa2014bccb5e4f0a91cbc9845df4634d) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:41.261561 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet 8400e23873fd4be18de2998afc64a689 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:18 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:41.262501 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet 0ca6e2165a9d41299c1fe0bfa9475915 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:17 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:41.263126 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet bc98762790f441038972366b1cc2c55b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:19 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:41.263760 27497 tablet_service.cc:1515] Processing DeleteTablet for tablet 8218868ff40b405387d560f614427c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:17 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:41.265112 27687 log.cc:1199] T 4e9a6a6e05464b7b97ae624f51402cf0 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/4e9a6a6e05464b7b97ae624f51402cf0
I20250903 20:05:41.265529 27687 ts_tablet_manager.cc:1939] T 4e9a6a6e05464b7b97ae624f51402cf0 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:41.265865 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1292) took 2707 ms. Trace:
I20250903 20:05:41.265933 27687 rpcz_store.cc:276] 0903 20:05:38.557995 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.558082 (+ 87us) service_pool.cc:225] Handling call
0903 20:05:41.033128 (+2475046us) ts_tablet_manager.cc:1145] Deleting tablet 4e9a6a6e05464b7b97ae624f51402cf0
0903 20:05:41.033138 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:41.037651 (+ 4513us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:41.257401 (+219750us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:41.265848 (+ 8447us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2411725}
I20250903 20:05:41.266479 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 4e9a6a6e05464b7b97ae624f51402cf0 (table default.Table_50f6b8731a2c4ccb8bf58597acb5b760 [id=4eecee99ccf149d49f2f95c27b5b05ca]) successfully deleted
I20250903 20:05:41.238128 27510 raft_consensus.cc:2466] T 733f8024e1fa401ca9614a97db91a19b P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
W20250903 20:05:41.266758 27462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.65.65:42989, user_credentials={real_user=slave}} blocked reactor thread for 104717us
I20250903 20:05:41.267951 27320 leader_election.cc:304] T 7d5a3fbe93e44c228222ab1b1e78c187 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:41.268203 29669 raft_consensus.cc:2802] T 7d5a3fbe93e44c228222ab1b1e78c187 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:41.268280 29669 raft_consensus.cc:491] T 7d5a3fbe93e44c228222ab1b1e78c187 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:41.268335 29669 raft_consensus.cc:3058] T 7d5a3fbe93e44c228222ab1b1e78c187 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:41.268575 27687 tablet_replica.cc:331] T 64c4dd2645034a7eaf243f5474603f10 P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:41.268802 27320 leader_election.cc:304] T 54f1ce5192f547f595dea23a9eabf8e6 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:41.269395 27385 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54602d7f723d4c10af12408a43ed1f02" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:41.269610 29760 raft_consensus.cc:2802] T 54f1ce5192f547f595dea23a9eabf8e6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:41.269681 29760 raft_consensus.cc:491] T 54f1ce5192f547f595dea23a9eabf8e6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:41.269753 29760 raft_consensus.cc:3058] T 54f1ce5192f547f595dea23a9eabf8e6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:41.271674 29669 raft_consensus.cc:513] T 7d5a3fbe93e44c228222ab1b1e78c187 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.272125 29760 raft_consensus.cc:513] T 54f1ce5192f547f595dea23a9eabf8e6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.272620 29760 leader_election.cc:290] T 54f1ce5192f547f595dea23a9eabf8e6 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:41.273501 27321 leader_election.cc:343] T 54f1ce5192f547f595dea23a9eabf8e6 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Not found: Tablet not found: 54f1ce5192f547f595dea23a9eabf8e6
I20250903 20:05:41.273599 27687 raft_consensus.cc:2241] T 64c4dd2645034a7eaf243f5474603f10 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:41.273767 29669 leader_election.cc:290] T 7d5a3fbe93e44c228222ab1b1e78c187 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:41.273978 27692 ts_tablet_manager.cc:1428] T 64d53f2071b0449ca61caa6c264a3641 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20250903 20:05:41.274108 27244 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d5a3fbe93e44c228222ab1b1e78c187" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:41.274340 27692 tablet_bootstrap.cc:492] T cf99a74acfdb4a17bbbce3a07467e9c1 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
W20250903 20:05:41.274825 27321 leader_election.cc:343] T 7d5a3fbe93e44c228222ab1b1e78c187 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:41.275835 27692 tablet_bootstrap.cc:654] T cf99a74acfdb4a17bbbce3a07467e9c1 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:41.277995 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "41056c5c5003415993bee74d7621bd14" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:41.278573 27248 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "41056c5c5003415993bee74d7621bd14" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:41.279402 27249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54602d7f723d4c10af12408a43ed1f02" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:41.279461 27509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f1ce5192f547f595dea23a9eabf8e6" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:41.279626 27509 raft_consensus.cc:3058] T 54f1ce5192f547f595dea23a9eabf8e6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
W20250903 20:05:41.282174 27184 consensus_peers.cc:489] T 64c4dd2645034a7eaf243f5474603f10 P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:41.283349 29779 raft_consensus.cc:491] T 55b2cdee20264ae6b64ebe3607fe1ca6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:41.283459 27509 raft_consensus.cc:2466] T 54f1ce5192f547f595dea23a9eabf8e6 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:41.283579 29779 raft_consensus.cc:513] T 55b2cdee20264ae6b64ebe3607fe1ca6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:41.284053 27462 leader_election.cc:343] T 54602d7f723d4c10af12408a43ed1f02 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:41.284402 29779 leader_election.cc:290] T 55b2cdee20264ae6b64ebe3607fe1ca6 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:41.284584 27510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d5a3fbe93e44c228222ab1b1e78c187" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
W20250903 20:05:41.284417 27462 leader_election.cc:343] T 41056c5c5003415993bee74d7621bd14 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:41.285313 27320 leader_election.cc:304] T 54f1ce5192f547f595dea23a9eabf8e6 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:41.285704 27244 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "55b2cdee20264ae6b64ebe3607fe1ca6" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:41.285851 27462 leader_election.cc:343] T 41056c5c5003415993bee74d7621bd14 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:41.286052 29669 raft_consensus.cc:2802] T 54f1ce5192f547f595dea23a9eabf8e6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:41.286244 29669 raft_consensus.cc:695] T 54f1ce5192f547f595dea23a9eabf8e6 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:41.286559 29669 consensus_queue.cc:237] T 54f1ce5192f547f595dea23a9eabf8e6 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.287748 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "55b2cdee20264ae6b64ebe3607fe1ca6" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:41.289364 27057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet 6b0c7097d19a412bbe5720126621b79d
I20250903 20:05:41.289716 27462 leader_election.cc:304] T 41056c5c5003415993bee74d7621bd14 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:41.290057 29779 raft_consensus.cc:2747] T 41056c5c5003415993bee74d7621bd14 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:41.284839 27510 raft_consensus.cc:3058] T 7d5a3fbe93e44c228222ab1b1e78c187 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:41.291173 27062 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet cab87148f7b84222915bf8eefc78a5f1
I20250903 20:05:41.292459 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b0c7097d19a412bbe5720126621b79d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:40 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:41.292843 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet cab87148f7b84222915bf8eefc78a5f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:41 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:41.293398 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet cab87148f7b84222915bf8eefc78a5f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:41 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:41.293426 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b0c7097d19a412bbe5720126621b79d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:40 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:41.293829 27359 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b0c7097d19a412bbe5720126621b79d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:40 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:41.293838 27183 consensus_peers.cc:489] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 89f8f0ee68b940db9ce250c0604f934a -> Peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Couldn't send request to peer e72ef1dbcc5143cf9f04a808f101c80f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 17f16e9c8f794e03a2f8b3b6b35f8e84. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:41.294153 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet cab87148f7b84222915bf8eefc78a5f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:41 UTC) from {username='slave'} at 127.0.0.1:54902
W20250903 20:05:41.295851 27462 leader_election.cc:343] T 54602d7f723d4c10af12408a43ed1f02 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:41.296074 27462 leader_election.cc:304] T 54602d7f723d4c10af12408a43ed1f02 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:41.296375 29779 raft_consensus.cc:2747] T 54602d7f723d4c10af12408a43ed1f02 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:41.273790 27687 raft_consensus.cc:2270] T 64c4dd2645034a7eaf243f5474603f10 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:41.297829 27687 ts_tablet_manager.cc:1905] T 64c4dd2645034a7eaf243f5474603f10 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250903 20:05:41.305837 27462 leader_election.cc:343] T 55b2cdee20264ae6b64ebe3607fe1ca6 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:41.307030 27693 tablet_bootstrap.cc:492] T 43fb27c857d543b7b78ea6108c406085 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:41.309793 29411 consensus_queue.cc:1035] T 17f16e9c8f794e03a2f8b3b6b35f8e84 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 20:05:41.319659 27693 ts_tablet_manager.cc:1397] T 43fb27c857d543b7b78ea6108c406085 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.150s user 0.003s sys 0.000s
I20250903 20:05:41.320950 27693 raft_consensus.cc:357] T 43fb27c857d543b7b78ea6108c406085 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.308713 29779 raft_consensus.cc:491] T b3d9491f777b422da4b9fca803beabff P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:41.320171 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bf570855b204b96acd070587c434463 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
W20250903 20:05:41.306196 27462 leader_election.cc:343] T 55b2cdee20264ae6b64ebe3607fe1ca6 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:41.326325 27462 leader_election.cc:304] T 55b2cdee20264ae6b64ebe3607fe1ca6 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:41.322384 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet 314856bf269f4df7aa25f5a30cec0d5b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:19 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:41.321189 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet b84f487773c24735a61d0fd0898b2bcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:17 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:41.326843 29371 raft_consensus.cc:2747] T 55b2cdee20264ae6b64ebe3607fe1ca6 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:41.326089 29779 raft_consensus.cc:513] T b3d9491f777b422da4b9fca803beabff P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.327956 29779 leader_election.cc:290] T b3d9491f777b422da4b9fca803beabff P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:41.328101 27693 raft_consensus.cc:383] T 43fb27c857d543b7b78ea6108c406085 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:41.328177 27693 raft_consensus.cc:738] T 43fb27c857d543b7b78ea6108c406085 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:41.328644 27693 consensus_queue.cc:260] T 43fb27c857d543b7b78ea6108c406085 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.332077 27693 ts_tablet_manager.cc:1428] T 43fb27c857d543b7b78ea6108c406085 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20250903 20:05:41.333249 27693 tablet_bootstrap.cc:492] T 314856bf269f4df7aa25f5a30cec0d5b P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:41.335397 29760 raft_consensus.cc:491] T f4dec89fc6bb436ba6542c267eaf9c15 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:41.335613 29760 raft_consensus.cc:513] T f4dec89fc6bb436ba6542c267eaf9c15 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.336376 29760 leader_election.cc:290] T f4dec89fc6bb436ba6542c267eaf9c15 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:41.336431 27693 tablet_bootstrap.cc:654] T 314856bf269f4df7aa25f5a30cec0d5b P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:41.337617 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4dec89fc6bb436ba6542c267eaf9c15" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:41.337888 27239 raft_consensus.cc:2466] T f4dec89fc6bb436ba6542c267eaf9c15 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:41.338407 27321 leader_election.cc:304] T f4dec89fc6bb436ba6542c267eaf9c15 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:41.338752 29760 raft_consensus.cc:2802] T f4dec89fc6bb436ba6542c267eaf9c15 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:41.338831 29760 raft_consensus.cc:491] T f4dec89fc6bb436ba6542c267eaf9c15 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:41.338907 29760 raft_consensus.cc:3058] T f4dec89fc6bb436ba6542c267eaf9c15 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:41.341915 29760 raft_consensus.cc:513] T f4dec89fc6bb436ba6542c267eaf9c15 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.342664 27692 tablet_bootstrap.cc:492] T cf99a74acfdb4a17bbbce3a07467e9c1 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:41.343008 27692 ts_tablet_manager.cc:1397] T cf99a74acfdb4a17bbbce3a07467e9c1 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.069s user 0.004s sys 0.000s
I20250903 20:05:41.343814 27692 raft_consensus.cc:357] T cf99a74acfdb4a17bbbce3a07467e9c1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.343930 29760 leader_election.cc:290] T f4dec89fc6bb436ba6542c267eaf9c15 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:41.344012 27692 raft_consensus.cc:383] T cf99a74acfdb4a17bbbce3a07467e9c1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:41.344170 27692 raft_consensus.cc:738] T cf99a74acfdb4a17bbbce3a07467e9c1 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:41.344476 27692 consensus_queue.cc:260] T cf99a74acfdb4a17bbbce3a07467e9c1 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.344709 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4dec89fc6bb436ba6542c267eaf9c15" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:41.344949 27239 raft_consensus.cc:3058] T f4dec89fc6bb436ba6542c267eaf9c15 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:41.344977 27692 ts_tablet_manager.cc:1428] T cf99a74acfdb4a17bbbce3a07467e9c1 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:41.345239 27692 tablet_bootstrap.cc:492] T 849eb453d3e04b16bbb5df9017bfb2fd P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:41.346736 27692 tablet_bootstrap.cc:654] T 849eb453d3e04b16bbb5df9017bfb2fd P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:41.348088 27239 raft_consensus.cc:2466] T f4dec89fc6bb436ba6542c267eaf9c15 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:41.348944 27321 leader_election.cc:304] T f4dec89fc6bb436ba6542c267eaf9c15 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:41.349318 29669 raft_consensus.cc:2802] T f4dec89fc6bb436ba6542c267eaf9c15 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:41.349416 27061 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_14a46b39c912447db13c3d3d84cad475" } modify_external_catalogs: true
I20250903 20:05:41.349506 29669 raft_consensus.cc:695] T f4dec89fc6bb436ba6542c267eaf9c15 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:41.349617 27061 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_14a46b39c912447db13c3d3d84cad475" } modify_external_catalogs: true
I20250903 20:05:41.349771 29669 consensus_queue.cc:237] T f4dec89fc6bb436ba6542c267eaf9c15 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.350442 27685 raft_consensus.cc:2241] T 46f49fd718be4a2996d781ab31268217 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:41.350656 27685 raft_consensus.cc:2270] T 46f49fd718be4a2996d781ab31268217 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:41.351209 27685 ts_tablet_manager.cc:1905] T 46f49fd718be4a2996d781ab31268217 P 89f8f0ee68b940db9ce250c0604f934a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:41.351176 27062 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_58f40fbfe6a946eab6b621584922f4a6" } new_table_name: "default.Table_4b1549571976435493df5449979f5a46" modify_external_catalogs: true
W20250903 20:05:41.353055 27064 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.26.65.67:58355 (request call id 179) took 804 ms (client timeout 1000 ms). Trace:
W20250903 20:05:41.353152 27064 rpcz_store.cc:269] 0903 20:05:40.548549 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:40.556051 (+ 7502us) service_pool.cc:225] Handling call
0903 20:05:40.583875 (+ 27824us) spinlock_profiling.cc:243] Waited 7.61 ms on lock 0x616000040f54. stack: 00007f4b9d6c2dc6 00007f4b9d6c2b80 00007f4b9c6aa3ad 0000000000972d94 0000000000972d5d 00007f4bac49014d 00007f4bac490243 00007f4ba914f941 00007f4bb0cdaa7d 00007f4bb0e5e6f8 00007f4ba7c98c86 00007f4ba7c989b5 00007f4ba44730be 00007f4ba4471f9e 00007f4ba44781d5 00007f4ba4478aff
0903 20:05:41.353029 (+769154us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":43,"spinlock_wait_cycles":17512832}
I20250903 20:05:41.369783 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b3d9491f777b422da4b9fca803beabff" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:41.371080 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b3d9491f777b422da4b9fca803beabff" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:41.371918 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3dd25f75f0744e2ca56bd526a5af59b9 consensus state has no leader
I20250903 20:05:41.347837 27504 tablet_service.cc:1515] Processing DeleteTablet for tablet 847d0a0a3d2b45938cec9058c77f6231 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:41.348480 27493 tablet_service.cc:1515] Processing DeleteTablet for tablet ea3f82f4e4244219ab0919f3e341aead with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:26 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:41.349143 27506 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f2eefeebbfc47249e2f72b7c52e905d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:41.352627 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet 259bc73ca9df47e9874692725a60bb1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:20 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:41.375355 27695 tablet_bootstrap.cc:492] T 60c32370a5274347afafc6a7e1e94270 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:41.375584 27695 ts_tablet_manager.cc:1397] T 60c32370a5274347afafc6a7e1e94270 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.214s user 0.021s sys 0.033s
I20250903 20:05:41.378593 27695 raft_consensus.cc:357] T 60c32370a5274347afafc6a7e1e94270 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.378876 27695 raft_consensus.cc:383] T 60c32370a5274347afafc6a7e1e94270 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:41.378983 27695 raft_consensus.cc:738] T 60c32370a5274347afafc6a7e1e94270 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:41.368543 29779 raft_consensus.cc:491] T 64d53f2071b0449ca61caa6c264a3641 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:41.379271 27695 consensus_queue.cc:260] T 60c32370a5274347afafc6a7e1e94270 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.369246 27509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4dec89fc6bb436ba6542c267eaf9c15" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:41.379738 27509 raft_consensus.cc:3058] T f4dec89fc6bb436ba6542c267eaf9c15 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:41.379815 27695 ts_tablet_manager.cc:1428] T 60c32370a5274347afafc6a7e1e94270 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250903 20:05:41.369526 27511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4dec89fc6bb436ba6542c267eaf9c15" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:41.325785 27510 raft_consensus.cc:2466] T 7d5a3fbe93e44c228222ab1b1e78c187 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:41.380098 27695 tablet_bootstrap.cc:492] T 5fb2426aa01e47598f268d0a90475271 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:41.381788 27695 tablet_bootstrap.cc:654] T 5fb2426aa01e47598f268d0a90475271 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:41.373746 27462 leader_election.cc:343] T b3d9491f777b422da4b9fca803beabff P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:41.382848 29782 raft_consensus.cc:491] T 112c0dccb60c4c7c9efc164d9c8443ea P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:41.383009 29782 raft_consensus.cc:513] T 112c0dccb60c4c7c9efc164d9c8443ea P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:41.383236 27462 leader_election.cc:343] T b3d9491f777b422da4b9fca803beabff P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:41.383338 27462 leader_election.cc:304] T b3d9491f777b422da4b9fca803beabff P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:41.383592 29782 leader_election.cc:290] T 112c0dccb60c4c7c9efc164d9c8443ea P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:41.383715 29782 raft_consensus.cc:2747] T b3d9491f777b422da4b9fca803beabff P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:41.384438 27320 leader_election.cc:304] T 7d5a3fbe93e44c228222ab1b1e78c187 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:41.384817 29760 raft_consensus.cc:2802] T 7d5a3fbe93e44c228222ab1b1e78c187 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:41.384931 29760 raft_consensus.cc:695] T 7d5a3fbe93e44c228222ab1b1e78c187 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:41.385156 29760 consensus_queue.cc:237] T 7d5a3fbe93e44c228222ab1b1e78c187 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.387477 27509 raft_consensus.cc:2466] T f4dec89fc6bb436ba6542c267eaf9c15 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:41.388024 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "112c0dccb60c4c7c9efc164d9c8443ea" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:41.388096 29779 raft_consensus.cc:513] T 64d53f2071b0449ca61caa6c264a3641 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.389724 29779 leader_election.cc:290] T 64d53f2071b0449ca61caa6c264a3641 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:41.395381 27244 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "112c0dccb60c4c7c9efc164d9c8443ea" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:41.397660 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "64d53f2071b0449ca61caa6c264a3641" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:41.398742 27244 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "64d53f2071b0449ca61caa6c264a3641" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:41.400202 27695 tablet_bootstrap.cc:492] T 5fb2426aa01e47598f268d0a90475271 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:41.400406 27695 ts_tablet_manager.cc:1397] T 5fb2426aa01e47598f268d0a90475271 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20250903 20:05:41.401139 29779 raft_consensus.cc:491] T e9126e4a18c546c28927f22c9f16426e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:41.401273 27695 raft_consensus.cc:357] T 5fb2426aa01e47598f268d0a90475271 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.401351 29779 raft_consensus.cc:513] T e9126e4a18c546c28927f22c9f16426e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:41.401547 27695 raft_consensus.cc:383] T 5fb2426aa01e47598f268d0a90475271 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:41.401656 27695 raft_consensus.cc:738] T 5fb2426aa01e47598f268d0a90475271 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:41.401947 27695 consensus_queue.cc:260] T 5fb2426aa01e47598f268d0a90475271 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.402302 29779 leader_election.cc:290] T e9126e4a18c546c28927f22c9f16426e P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:41.402576 27695 ts_tablet_manager.cc:1428] T 5fb2426aa01e47598f268d0a90475271 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:41.403025 27695 tablet_bootstrap.cc:492] T 7d5a3fbe93e44c228222ab1b1e78c187 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
W20250903 20:05:41.403753 27462 leader_election.cc:343] T 112c0dccb60c4c7c9efc164d9c8443ea P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:41.404086 27462 leader_election.cc:343] T 64d53f2071b0449ca61caa6c264a3641 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:41.404332 27462 leader_election.cc:343] T 112c0dccb60c4c7c9efc164d9c8443ea P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:41.404459 27462 leader_election.cc:304] T 112c0dccb60c4c7c9efc164d9c8443ea P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
W20250903 20:05:41.405121 27462 leader_election.cc:343] T 64d53f2071b0449ca61caa6c264a3641 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:41.405306 27462 leader_election.cc:304] T 64d53f2071b0449ca61caa6c264a3641 P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:41.405597 29371 raft_consensus.cc:2747] T 112c0dccb60c4c7c9efc164d9c8443ea P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:41.405638 29782 raft_consensus.cc:2747] T 64d53f2071b0449ca61caa6c264a3641 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:41.407184 27695 tablet_bootstrap.cc:654] T 7d5a3fbe93e44c228222ab1b1e78c187 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:41.408304 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9126e4a18c546c28927f22c9f16426e" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:41.412601 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9126e4a18c546c28927f22c9f16426e" candidate_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:41.414460 29760 raft_consensus.cc:491] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:41.414726 29760 raft_consensus.cc:513] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.415990 29760 leader_election.cc:290] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:41.422082 27693 tablet_bootstrap.cc:492] T 314856bf269f4df7aa25f5a30cec0d5b P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:41.422367 27693 ts_tablet_manager.cc:1397] T 314856bf269f4df7aa25f5a30cec0d5b P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.089s user 0.002s sys 0.002s
I20250903 20:05:41.423247 27693 raft_consensus.cc:357] T 314856bf269f4df7aa25f5a30cec0d5b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.423516 27693 raft_consensus.cc:383] T 314856bf269f4df7aa25f5a30cec0d5b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:41.423615 27693 raft_consensus.cc:738] T 314856bf269f4df7aa25f5a30cec0d5b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:41.424075 27693 consensus_queue.cc:260] T 314856bf269f4df7aa25f5a30cec0d5b P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.424860 27693 ts_tablet_manager.cc:1428] T 314856bf269f4df7aa25f5a30cec0d5b P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:41.425202 27693 tablet_bootstrap.cc:492] T 351208baafc94f5eab531101d52e3b3f P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:41.426795 27693 tablet_bootstrap.cc:654] T 351208baafc94f5eab531101d52e3b3f P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:41.430480 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c77bc0e8bb34712bf8109cadbc0fc06" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:41.430889 27239 raft_consensus.cc:2466] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:41.432924 27692 tablet_bootstrap.cc:492] T 849eb453d3e04b16bbb5df9017bfb2fd P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:41.433171 27692 ts_tablet_manager.cc:1397] T 849eb453d3e04b16bbb5df9017bfb2fd P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.088s user 0.003s sys 0.000s
I20250903 20:05:41.434010 27692 raft_consensus.cc:357] T 849eb453d3e04b16bbb5df9017bfb2fd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.434286 27692 raft_consensus.cc:383] T 849eb453d3e04b16bbb5df9017bfb2fd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:41.434094 27066 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e151b7f4451b4c68b3237b08382c5efc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:41.434350 27692 raft_consensus.cc:738] T 849eb453d3e04b16bbb5df9017bfb2fd P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
W20250903 20:05:41.434903 27066 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e151b7f4451b4c68b3237b08382c5efc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:41.434602 27692 consensus_queue.cc:260] T 849eb453d3e04b16bbb5df9017bfb2fd P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.435917 27692 ts_tablet_manager.cc:1428] T 849eb453d3e04b16bbb5df9017bfb2fd P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250903 20:05:41.436262 27692 tablet_bootstrap.cc:492] T 47500175a642457687aa4226195fce47 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:41.437254 27684 ts_tablet_manager.cc:1918] T 8e12f4c420874ab382d433982bddbf05 P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:41.438324 27684 log.cc:1199] T 8e12f4c420874ab382d433982bddbf05 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/8e12f4c420874ab382d433982bddbf05
I20250903 20:05:41.439193 27058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_ec4b7aba6eaa402586095c0237bdb4d4" } new_table_name: "default.Table_5ec412e6385440aeba984e14ab0c4082" modify_external_catalogs: true
I20250903 20:05:41.440678 27060 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_5496e1e153d54f8b925de9f4052ede74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:41.441254 27060 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5496e1e153d54f8b925de9f4052ede74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:41.443115 27065 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_04a9700c1b2245169517c0f1bf78347c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:41.443683 27065 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_04a9700c1b2245169517c0f1bf78347c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:41.449661 29461 raft_consensus.cc:491] T 09fa32312c9648fbbd6fd327e9738066 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51578fb7bceb4e119b3ab1e58c4f5528)
I20250903 20:05:41.449916 29461 raft_consensus.cc:513] T 09fa32312c9648fbbd6fd327e9738066 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:41.454131 27063 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3af2fef47f9f41a381f0b9af505e3c77 consensus state has no leader
I20250903 20:05:41.431442 27321 leader_election.cc:304] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:41.454597 29760 raft_consensus.cc:2802] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:41.454685 29760 raft_consensus.cc:491] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:41.454746 29760 raft_consensus.cc:3058] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:41.456566 27059 master_service.cc:948] ReplaceTablet: received request to replace tablet fa544dfdfff04a1e97676e562aeb0f40 from {username='slave'} at 127.0.0.1:45936
I20250903 20:05:41.456954 27061 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 35bdb0090d8747059b5ee103e786e979: Sending DeleteTablet for 3 replicas of tablet cc374781309042a394d517734fdd6918
I20250903 20:05:41.457844 27362 tablet_service.cc:1515] Processing DeleteTablet for tablet cc374781309042a394d517734fdd6918 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:41 UTC) from {username='slave'} at 127.0.0.1:54902
I20250903 20:05:41.458716 29760 raft_consensus.cc:513] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.459820 29760 leader_election.cc:290] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:41.460158 27062 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c432d5a0363f4d9b9cb23bfb6348382b consensus state has no leader
I20250903 20:05:41.460301 27215 tablet_service.cc:1515] Processing DeleteTablet for tablet cc374781309042a394d517734fdd6918 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:41 UTC) from {username='slave'} at 127.0.0.1:56534
I20250903 20:05:41.460805 27687 ts_tablet_manager.cc:1918] T 64c4dd2645034a7eaf243f5474603f10 P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:41.460906 27687 log.cc:1199] T 64c4dd2645034a7eaf243f5474603f10 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/64c4dd2645034a7eaf243f5474603f10
I20250903 20:05:41.461328 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c77bc0e8bb34712bf8109cadbc0fc06" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:41.461525 27239 raft_consensus.cc:3058] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:41.461635 27687 ts_tablet_manager.cc:1939] T 64c4dd2645034a7eaf243f5474603f10 P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
W20250903 20:05:41.463181 27184 leader_election.cc:343] T 09fa32312c9648fbbd6fd327e9738066 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: 09fa32312c9648fbbd6fd327e9738066
I20250903 20:05:41.464314 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1291) took 2907 ms. Trace:
I20250903 20:05:41.464884 27239 raft_consensus.cc:2466] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:41.465322 27321 leader_election.cc:304] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:41.465683 29760 raft_consensus.cc:2802] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:41.464749 27687 rpcz_store.cc:276] 0903 20:05:38.557165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.557214 (+ 49us) service_pool.cc:225] Handling call
0903 20:05:41.268519 (+2711305us) ts_tablet_manager.cc:1145] Deleting tablet 64c4dd2645034a7eaf243f5474603f10
0903 20:05:41.268530 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:41.318737 (+ 50207us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:41.460679 (+141942us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:41.464287 (+ 3608us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2645055}
W20250903 20:05:41.467342 27183 leader_election.cc:343] T 09fa32312c9648fbbd6fd327e9738066 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: 09fa32312c9648fbbd6fd327e9738066
I20250903 20:05:41.467509 27183 leader_election.cc:304] T 09fa32312c9648fbbd6fd327e9738066 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:41.467796 29452 raft_consensus.cc:2747] T 09fa32312c9648fbbd6fd327e9738066 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:41.468150 29461 leader_election.cc:290] T 09fa32312c9648fbbd6fd327e9738066 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:41.465863 29760 raft_consensus.cc:695] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:41.469235 29760 consensus_queue.cc:237] T 9c77bc0e8bb34712bf8109cadbc0fc06 P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.470992 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 64c4dd2645034a7eaf243f5474603f10 (table default.Table_d9878f04c3404885815ae53a9054de14 [id=a98d20608a054ad29a951bb47cf61711]) successfully deleted
I20250903 20:05:41.471153 27687 tablet_replica.cc:331] T 259bc73ca9df47e9874692725a60bb1c P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:41.471535 27687 raft_consensus.cc:2241] T 259bc73ca9df47e9874692725a60bb1c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:41.471604 27693 tablet_bootstrap.cc:492] T 351208baafc94f5eab531101d52e3b3f P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:41.471815 27693 ts_tablet_manager.cc:1397] T 351208baafc94f5eab531101d52e3b3f P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.000s
I20250903 20:05:41.471873 27687 raft_consensus.cc:2270] T 259bc73ca9df47e9874692725a60bb1c P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:41.472682 27693 raft_consensus.cc:357] T 351208baafc94f5eab531101d52e3b3f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.477025 27684 ts_tablet_manager.cc:1939] T 8e12f4c420874ab382d433982bddbf05 P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:41.458489 27495 tablet_service.cc:1515] Processing DeleteTablet for tablet cc374781309042a394d517734fdd6918 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 20:05:41 UTC) from {username='slave'} at 127.0.0.1:44666
I20250903 20:05:41.478474 27693 raft_consensus.cc:383] T 351208baafc94f5eab531101d52e3b3f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:41.467653 27692 tablet_bootstrap.cc:654] T 47500175a642457687aa4226195fce47 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:41.478982 27693 raft_consensus.cc:738] T 351208baafc94f5eab531101d52e3b3f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
W20250903 20:05:41.456614 27462 leader_election.cc:343] T e9126e4a18c546c28927f22c9f16426e P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
W20250903 20:05:41.479543 27462 leader_election.cc:343] T e9126e4a18c546c28927f22c9f16426e P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:41.479717 27462 leader_election.cc:304] T e9126e4a18c546c28927f22c9f16426e P e72ef1dbcc5143cf9f04a808f101c80f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e72ef1dbcc5143cf9f04a808f101c80f; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:41.480294 29371 raft_consensus.cc:2747] T e9126e4a18c546c28927f22c9f16426e P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:41.481812 27509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c77bc0e8bb34712bf8109cadbc0fc06" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:41.482100 27509 raft_consensus.cc:2466] T 9c77bc0e8bb34712bf8109cadbc0fc06 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:41.482651 27693 consensus_queue.cc:260] T 351208baafc94f5eab531101d52e3b3f P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.483850 27693 ts_tablet_manager.cc:1428] T 351208baafc94f5eab531101d52e3b3f P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250903 20:05:41.484658 27693 tablet_bootstrap.cc:492] T 69f8eef52bee431a86f0eef45fd11f9f P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:41.487380 27693 tablet_bootstrap.cc:654] T 69f8eef52bee431a86f0eef45fd11f9f P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:41.487432 27687 ts_tablet_manager.cc:1905] T 259bc73ca9df47e9874692725a60bb1c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:41.492146 29760 raft_consensus.cc:491] T ef02f91c0964446daf62ac1dde0330ff P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:41.498446 29760 raft_consensus.cc:513] T ef02f91c0964446daf62ac1dde0330ff P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.485265 27509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c77bc0e8bb34712bf8109cadbc0fc06" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:41.498874 27509 raft_consensus.cc:3058] T 9c77bc0e8bb34712bf8109cadbc0fc06 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:41.499661 29760 leader_election.cc:290] T ef02f91c0964446daf62ac1dde0330ff P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:41.499732 27244 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef02f91c0964446daf62ac1dde0330ff" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
W20250903 20:05:41.500228 27321 leader_election.cc:343] T ef02f91c0964446daf62ac1dde0330ff P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:41.501538 27509 raft_consensus.cc:2466] T 9c77bc0e8bb34712bf8109cadbc0fc06 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:41.484866 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1279) took 2466 ms. Trace:
I20250903 20:05:41.501825 27684 rpcz_store.cc:276] 0903 20:05:39.018682 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:39.019020 (+ 338us) service_pool.cc:225] Handling call
0903 20:05:41.225806 (+2206786us) ts_tablet_manager.cc:1145] Deleting tablet 8e12f4c420874ab382d433982bddbf05
0903 20:05:41.225817 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:41.316147 (+ 90330us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:41.437113 (+120966us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:41.484843 (+ 47730us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2175253}
I20250903 20:05:41.502298 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1280) took 2409 ms. Trace:
I20250903 20:05:41.502377 27684 rpcz_store.cc:276] 0903 20:05:39.092943 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:39.093005 (+ 62us) service_pool.cc:225] Handling call
0903 20:05:41.502131 (+2409126us) ts_tablet_manager.cc:1145] Deleting tablet d1e89ad94adb4de8b3738e30a84a8f79
0903 20:05:41.502143 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:41.502283 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2408931}
I20250903 20:05:41.502439 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet 8e12f4c420874ab382d433982bddbf05 (table default.Table_42f95b6c82f949a082559298f94e747c [id=430a979602e1430eaab642507035c4f0]) successfully deleted
I20250903 20:05:41.502624 27684 tablet_replica.cc:331] T f9375fff5f5945078c4412aa21c06f0d P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:41.502908 27684 raft_consensus.cc:2241] T f9375fff5f5945078c4412aa21c06f0d P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus shutting down.
W20250903 20:05:41.502979 27051 catalog_manager.cc:4908] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): delete failed for tablet d1e89ad94adb4de8b3738e30a84a8f79 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1e89ad94adb4de8b3738e30a84a8f79 already in progress: creating tablet
I20250903 20:05:41.503082 27684 raft_consensus.cc:2270] T f9375fff5f5945078c4412aa21c06f0d P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:41.503875 27684 ts_tablet_manager.cc:1905] T f9375fff5f5945078c4412aa21c06f0d P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:41.507644 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_b3a06d244cf147caaec9c053167fc413"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:41.508303 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3a06d244cf147caaec9c053167fc413 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:41.509771 27687 ts_tablet_manager.cc:1918] T 259bc73ca9df47e9874692725a60bb1c P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:41.509871 27687 log.cc:1199] T 259bc73ca9df47e9874692725a60bb1c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/259bc73ca9df47e9874692725a60bb1c
I20250903 20:05:41.510196 27687 ts_tablet_manager.cc:1939] T 259bc73ca9df47e9874692725a60bb1c P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:41.510046 27059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_a5dfcc8d322b4346a37a9d43614a84f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 20:05:41.510545 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1293) took 2951 ms. Trace:
W20250903 20:05:41.510650 27059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5dfcc8d322b4346a37a9d43614a84f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 20:05:41.510618 27687 rpcz_store.cc:276] 0903 20:05:38.558896 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.558945 (+ 49us) service_pool.cc:225] Handling call
0903 20:05:41.471024 (+2912079us) ts_tablet_manager.cc:1145] Deleting tablet 259bc73ca9df47e9874692725a60bb1c
0903 20:05:41.471072 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:41.506472 (+ 35400us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:41.509660 (+ 3188us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:41.510527 (+ 867us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2847468}
I20250903 20:05:41.511083 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 259bc73ca9df47e9874692725a60bb1c (table default.Table_8a37ddf23e9041d59958722b82a977f9 [id=60cc385fc1044966b4a12e13fdc7b5bb]) successfully deleted
I20250903 20:05:41.511548 27687 tablet_replica.cc:331] T 793d6157927143d4b9d8dbafd7cf0fcb P 51578fb7bceb4e119b3ab1e58c4f5528: stopping tablet replica
I20250903 20:05:41.511734 27687 raft_consensus.cc:2241] T 793d6157927143d4b9d8dbafd7cf0fcb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:41.511907 27687 raft_consensus.cc:2270] T 793d6157927143d4b9d8dbafd7cf0fcb P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:41.511932 27062 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5fbfb89f12b1422688807d10f3aaaabf" } modify_external_catalogs: true
I20250903 20:05:41.512094 27062 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_5fbfb89f12b1422688807d10f3aaaabf" } modify_external_catalogs: true
I20250903 20:05:41.512205 27509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef02f91c0964446daf62ac1dde0330ff" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:41.512393 27509 raft_consensus.cc:2466] T ef02f91c0964446daf62ac1dde0330ff P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 0.
I20250903 20:05:41.512526 27687 ts_tablet_manager.cc:1905] T 793d6157927143d4b9d8dbafd7cf0fcb P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:41.513713 27695 tablet_bootstrap.cc:492] T 7d5a3fbe93e44c228222ab1b1e78c187 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:41.513855 27695 ts_tablet_manager.cc:1397] T 7d5a3fbe93e44c228222ab1b1e78c187 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.111s user 0.011s sys 0.005s
I20250903 20:05:41.514637 27695 raft_consensus.cc:357] T 7d5a3fbe93e44c228222ab1b1e78c187 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.514799 27695 raft_consensus.cc:383] T 7d5a3fbe93e44c228222ab1b1e78c187 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:41.514851 27695 raft_consensus.cc:738] T 7d5a3fbe93e44c228222ab1b1e78c187 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:41.515045 27695 consensus_queue.cc:260] T 7d5a3fbe93e44c228222ab1b1e78c187 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.518391 27320 leader_election.cc:304] T ef02f91c0964446daf62ac1dde0330ff P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:41.519071 29760 raft_consensus.cc:2802] T ef02f91c0964446daf62ac1dde0330ff P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:41.519165 29760 raft_consensus.cc:491] T ef02f91c0964446daf62ac1dde0330ff P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:41.519236 29760 raft_consensus.cc:3058] T ef02f91c0964446daf62ac1dde0330ff P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:41.522094 29760 raft_consensus.cc:513] T ef02f91c0964446daf62ac1dde0330ff P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.523943 27687 ts_tablet_manager.cc:1918] T 793d6157927143d4b9d8dbafd7cf0fcb P 51578fb7bceb4e119b3ab1e58c4f5528: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:41.524886 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef02f91c0964446daf62ac1dde0330ff" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a"
I20250903 20:05:41.530751 27509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef02f91c0964446daf62ac1dde0330ff" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:41.531255 27509 raft_consensus.cc:3058] T ef02f91c0964446daf62ac1dde0330ff P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:41.534065 27509 raft_consensus.cc:2466] T ef02f91c0964446daf62ac1dde0330ff P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51578fb7bceb4e119b3ab1e58c4f5528 in term 1.
I20250903 20:05:41.535214 27695 ts_tablet_manager.cc:1428] T 7d5a3fbe93e44c228222ab1b1e78c187 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20250903 20:05:41.536334 27685 ts_tablet_manager.cc:1918] T 46f49fd718be4a2996d781ab31268217 P 89f8f0ee68b940db9ce250c0604f934a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 20:05:41.536437 27685 log.cc:1199] T 46f49fd718be4a2996d781ab31268217 P 89f8f0ee68b940db9ce250c0604f934a: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-0/wal/wals/46f49fd718be4a2996d781ab31268217
I20250903 20:05:41.536731 27685 ts_tablet_manager.cc:1939] T 46f49fd718be4a2996d781ab31268217 P 89f8f0ee68b940db9ce250c0604f934a: Deleting consensus metadata
I20250903 20:05:41.537359 27695 tablet_bootstrap.cc:492] T 43fb27c857d543b7b78ea6108c406085 P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:41.524523 29760 leader_election.cc:290] T ef02f91c0964446daf62ac1dde0330ff P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
W20250903 20:05:41.525528 27321 leader_election.cc:343] T ef02f91c0964446daf62ac1dde0330ff P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:41.542944 27320 leader_election.cc:304] T ef02f91c0964446daf62ac1dde0330ff P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f; no voters: 89f8f0ee68b940db9ce250c0604f934a
I20250903 20:05:41.543494 29760 raft_consensus.cc:2802] T ef02f91c0964446daf62ac1dde0330ff P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:41.543749 29760 raft_consensus.cc:695] T ef02f91c0964446daf62ac1dde0330ff P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 LEADER]: Becoming Leader. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Running, Role: LEADER
I20250903 20:05:41.544066 29760 consensus_queue.cc:237] T ef02f91c0964446daf62ac1dde0330ff P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.529417 27693 tablet_bootstrap.cc:492] T 69f8eef52bee431a86f0eef45fd11f9f P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:41.545322 27693 ts_tablet_manager.cc:1397] T 69f8eef52bee431a86f0eef45fd11f9f P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.000s
I20250903 20:05:41.546165 27695 tablet_bootstrap.cc:654] T 43fb27c857d543b7b78ea6108c406085 P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:41.546103 27693 raft_consensus.cc:357] T 69f8eef52bee431a86f0eef45fd11f9f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.546298 27693 raft_consensus.cc:383] T 69f8eef52bee431a86f0eef45fd11f9f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:41.546352 27693 raft_consensus.cc:738] T 69f8eef52bee431a86f0eef45fd11f9f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:41.546571 27693 consensus_queue.cc:260] T 69f8eef52bee431a86f0eef45fd11f9f P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.547766 29669 raft_consensus.cc:491] T ed4902c002ef4198802375ced39ef2ad P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:41.547894 29669 raft_consensus.cc:513] T ed4902c002ef4198802375ced39ef2ad P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.548532 29669 leader_election.cc:290] T ed4902c002ef4198802375ced39ef2ad P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:41.547626 27693 ts_tablet_manager.cc:1428] T 69f8eef52bee431a86f0eef45fd11f9f P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 20:05:41.549760 27693 tablet_bootstrap.cc:492] T 6d13884fff8c43b690836c6a341a2d92 P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:41.551579 27693 tablet_bootstrap.cc:654] T 6d13884fff8c43b690836c6a341a2d92 P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:41.551885 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed4902c002ef4198802375ced39ef2ad" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:41.555938 27509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed4902c002ef4198802375ced39ef2ad" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:41.557480 27692 tablet_bootstrap.cc:492] T 47500175a642457687aa4226195fce47 P e72ef1dbcc5143cf9f04a808f101c80f: No bootstrap required, opened a new log
I20250903 20:05:41.557644 27692 ts_tablet_manager.cc:1397] T 47500175a642457687aa4226195fce47 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent bootstrapping tablet: real 0.121s user 0.004s sys 0.000s
I20250903 20:05:41.558552 27692 raft_consensus.cc:357] T 47500175a642457687aa4226195fce47 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.558729 27692 raft_consensus.cc:383] T 47500175a642457687aa4226195fce47 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:41.558791 27692 raft_consensus.cc:738] T 47500175a642457687aa4226195fce47 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72ef1dbcc5143cf9f04a808f101c80f, State: Initialized, Role: FOLLOWER
I20250903 20:05:41.560410 27692 consensus_queue.cc:260] T 47500175a642457687aa4226195fce47 P e72ef1dbcc5143cf9f04a808f101c80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.558391 27320 leader_election.cc:304] T ed4902c002ef4198802375ced39ef2ad P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:41.568914 27692 ts_tablet_manager.cc:1428] T 47500175a642457687aa4226195fce47 P e72ef1dbcc5143cf9f04a808f101c80f: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250903 20:05:41.569283 27692 tablet_bootstrap.cc:492] T 1453a8f6bd7b4b26b75d7fc08578a351 P e72ef1dbcc5143cf9f04a808f101c80f: Bootstrap starting.
I20250903 20:05:41.571156 27692 tablet_bootstrap.cc:654] T 1453a8f6bd7b4b26b75d7fc08578a351 P e72ef1dbcc5143cf9f04a808f101c80f: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:41.572106 28567 raft_consensus.cc:2747] T ed4902c002ef4198802375ced39ef2ad P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:41.559249 29669 raft_consensus.cc:491] T 314856bf269f4df7aa25f5a30cec0d5b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:41.573184 29669 raft_consensus.cc:513] T 314856bf269f4df7aa25f5a30cec0d5b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.573771 29669 leader_election.cc:290] T 314856bf269f4df7aa25f5a30cec0d5b P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:41.575317 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "314856bf269f4df7aa25f5a30cec0d5b" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:41.578991 27517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "314856bf269f4df7aa25f5a30cec0d5b" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:41.579236 27517 raft_consensus.cc:2391] T 314856bf269f4df7aa25f5a30cec0d5b P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 51578fb7bceb4e119b3ab1e58c4f5528 in current term 1: Already voted for candidate 89f8f0ee68b940db9ce250c0604f934a in this term.
I20250903 20:05:41.580155 27320 leader_election.cc:304] T 314856bf269f4df7aa25f5a30cec0d5b P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528; no voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:41.580686 29760 raft_consensus.cc:3058] T 314856bf269f4df7aa25f5a30cec0d5b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:41.583927 29760 raft_consensus.cc:2747] T 314856bf269f4df7aa25f5a30cec0d5b P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 20:05:41.585186 27684 ts_tablet_manager.cc:1918] T f9375fff5f5945078c4412aa21c06f0d P e72ef1dbcc5143cf9f04a808f101c80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 20:05:41.589373 27684 log.cc:1199] T f9375fff5f5945078c4412aa21c06f0d P e72ef1dbcc5143cf9f04a808f101c80f: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-2/wal/wals/f9375fff5f5945078c4412aa21c06f0d
I20250903 20:05:41.589973 27684 ts_tablet_manager.cc:1939] T f9375fff5f5945078c4412aa21c06f0d P e72ef1dbcc5143cf9f04a808f101c80f: Deleting consensus metadata
I20250903 20:05:41.590312 27687 log.cc:1199] T 793d6157927143d4b9d8dbafd7cf0fcb P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting WAL directory at /tmp/dist-test-taskH7eH49/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756929904118321-26885-0/minicluster-data/ts-1/wal/wals/793d6157927143d4b9d8dbafd7cf0fcb
I20250903 20:05:41.591025 27687 ts_tablet_manager.cc:1939] T 793d6157927143d4b9d8dbafd7cf0fcb P 51578fb7bceb4e119b3ab1e58c4f5528: Deleting consensus metadata
I20250903 20:05:41.592119 27066 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_566a0ec0444945f4933ae71fbd4866e0" } new_table_name: "default.Table_d979cd8007544f0d89edeb9c2080b43b" modify_external_catalogs: true
I20250903 20:05:41.594094 27695 tablet_bootstrap.cc:492] T 43fb27c857d543b7b78ea6108c406085 P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:41.594267 27695 ts_tablet_manager.cc:1397] T 43fb27c857d543b7b78ea6108c406085 P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.057s user 0.007s sys 0.001s
I20250903 20:05:41.595057 27695 raft_consensus.cc:357] T 43fb27c857d543b7b78ea6108c406085 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.595263 27695 raft_consensus.cc:383] T 43fb27c857d543b7b78ea6108c406085 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:41.595726 27695 raft_consensus.cc:738] T 43fb27c857d543b7b78ea6108c406085 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:41.596462 27695 consensus_queue.cc:260] T 43fb27c857d543b7b78ea6108c406085 P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.598243 27695 ts_tablet_manager.cc:1428] T 43fb27c857d543b7b78ea6108c406085 P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250903 20:05:41.598750 27693 tablet_bootstrap.cc:492] T 6d13884fff8c43b690836c6a341a2d92 P 51578fb7bceb4e119b3ab1e58c4f5528: No bootstrap required, opened a new log
I20250903 20:05:41.598971 27693 ts_tablet_manager.cc:1397] T 6d13884fff8c43b690836c6a341a2d92 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.000s
I20250903 20:05:41.599810 27693 raft_consensus.cc:357] T 6d13884fff8c43b690836c6a341a2d92 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.600080 27693 raft_consensus.cc:383] T 6d13884fff8c43b690836c6a341a2d92 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250903 20:05:41.600196 27058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8f3116ad51fd48cea9f2321401e94802 has no consensus state
I20250903 20:05:41.600188 27693 raft_consensus.cc:738] T 6d13884fff8c43b690836c6a341a2d92 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51578fb7bceb4e119b3ab1e58c4f5528, State: Initialized, Role: FOLLOWER
I20250903 20:05:41.600572 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1294) took 2894 ms. Trace:
I20250903 20:05:41.600740 27687 rpcz_store.cc:276] 0903 20:05:38.706521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.706615 (+ 94us) service_pool.cc:225] Handling call
0903 20:05:41.511491 (+2804876us) ts_tablet_manager.cc:1145] Deleting tablet 793d6157927143d4b9d8dbafd7cf0fcb
0903 20:05:41.511504 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:41.520813 (+ 9309us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:41.523824 (+ 3011us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:41.600552 (+ 76728us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2804657}
I20250903 20:05:41.601253 27052 catalog_manager.cc:4928] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): tablet 793d6157927143d4b9d8dbafd7cf0fcb (table default.Table_e79f52615fe142b9823a2a99b8b9c406 [id=3432f6ac076b4e49b9fe63bf5fce8591]) successfully deleted
I20250903 20:05:41.601733 27687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54902 (request call id 1295) took 2842 ms. Trace:
I20250903 20:05:41.601871 27687 rpcz_store.cc:276] 0903 20:05:38.759728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:38.759784 (+ 56us) service_pool.cc:225] Handling call
0903 20:05:41.601591 (+2841807us) ts_tablet_manager.cc:1145] Deleting tablet 3f08c7edbcf54c16bc222f59773f72b1
0903 20:05:41.601602 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:41.601712 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2841615}
W20250903 20:05:41.604049 27052 catalog_manager.cc:4908] TS 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): delete failed for tablet 3f08c7edbcf54c16bc222f59773f72b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f08c7edbcf54c16bc222f59773f72b1 already in progress: creating tablet
I20250903 20:05:41.600544 27693 consensus_queue.cc:260] T 6d13884fff8c43b690836c6a341a2d92 P 51578fb7bceb4e119b3ab1e58c4f5528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.604497 27058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_43b6194256e1461cb34be4e7642889f2" } modify_external_catalogs: true
I20250903 20:05:41.604699 27058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_43b6194256e1461cb34be4e7642889f2" } modify_external_catalogs: true
I20250903 20:05:41.604825 27693 ts_tablet_manager.cc:1428] T 6d13884fff8c43b690836c6a341a2d92 P 51578fb7bceb4e119b3ab1e58c4f5528: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250903 20:05:41.605144 27693 tablet_bootstrap.cc:492] T 615f42e95bb24c8480f73d371fd63b9c P 51578fb7bceb4e119b3ab1e58c4f5528: Bootstrap starting.
I20250903 20:05:41.606840 27693 tablet_bootstrap.cc:654] T 615f42e95bb24c8480f73d371fd63b9c P 51578fb7bceb4e119b3ab1e58c4f5528: Neither blocks nor log segments found. Creating new log.
I20250903 20:05:41.607328 29461 raft_consensus.cc:491] T 6d13884fff8c43b690836c6a341a2d92 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:41.607474 29461 raft_consensus.cc:513] T 6d13884fff8c43b690836c6a341a2d92 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.608080 29461 leader_election.cc:290] T 6d13884fff8c43b690836c6a341a2d92 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:41.609442 27517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d13884fff8c43b690836c6a341a2d92" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:41.609661 27517 raft_consensus.cc:2466] T 6d13884fff8c43b690836c6a341a2d92 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:41.610090 27183 leader_election.cc:304] T 6d13884fff8c43b690836c6a341a2d92 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:41.610450 29461 raft_consensus.cc:2802] T 6d13884fff8c43b690836c6a341a2d92 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 20:05:41.610523 29461 raft_consensus.cc:491] T 6d13884fff8c43b690836c6a341a2d92 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 20:05:41.610579 29461 raft_consensus.cc:3058] T 6d13884fff8c43b690836c6a341a2d92 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:41.613234 29461 raft_consensus.cc:513] T 6d13884fff8c43b690836c6a341a2d92 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.608700 27379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d13884fff8c43b690836c6a341a2d92" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" is_pre_election: true
I20250903 20:05:41.614697 27379 raft_consensus.cc:2466] T 6d13884fff8c43b690836c6a341a2d92 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
I20250903 20:05:41.614979 27381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d13884fff8c43b690836c6a341a2d92" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51578fb7bceb4e119b3ab1e58c4f5528"
I20250903 20:05:41.615170 27381 raft_consensus.cc:3058] T 6d13884fff8c43b690836c6a341a2d92 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:41.617887 27381 raft_consensus.cc:2466] T 6d13884fff8c43b690836c6a341a2d92 P 51578fb7bceb4e119b3ab1e58c4f5528 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:41.618343 27184 leader_election.cc:304] T 6d13884fff8c43b690836c6a341a2d92 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51578fb7bceb4e119b3ab1e58c4f5528, 89f8f0ee68b940db9ce250c0604f934a; no voters:
I20250903 20:05:41.618628 29452 raft_consensus.cc:2802] T 6d13884fff8c43b690836c6a341a2d92 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader election won for term 1
I20250903 20:05:41.618745 29452 raft_consensus.cc:695] T 6d13884fff8c43b690836c6a341a2d92 P 89f8f0ee68b940db9ce250c0604f934a [term 1 LEADER]: Becoming Leader. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Running, Role: LEADER
I20250903 20:05:41.618691 27060 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_8a742d84a1dd47c1a154c8e4bd43dc79" } modify_external_catalogs: true
I20250903 20:05:41.618885 27060 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_8a742d84a1dd47c1a154c8e4bd43dc79" } modify_external_catalogs: true
I20250903 20:05:41.619001 29452 consensus_queue.cc:237] T 6d13884fff8c43b690836c6a341a2d92 P 89f8f0ee68b940db9ce250c0604f934a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.620704 29461 leader_election.cc:290] T 6d13884fff8c43b690836c6a341a2d92 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 election: Requested vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449)
I20250903 20:05:41.623162 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1095) took 5998 ms. Trace:
I20250903 20:05:41.623360 27685 rpcz_store.cc:276] 0903 20:05:35.625056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.625367 (+ 311us) service_pool.cc:225] Handling call
0903 20:05:41.155115 (+5529748us) ts_tablet_manager.cc:1145] Deleting tablet 46f49fd718be4a2996d781ab31268217
0903 20:05:41.155126 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:41.412641 (+257515us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:41.536209 (+123568us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:41.623073 (+ 86864us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5465041}
I20250903 20:05:41.624097 27052 catalog_manager.cc:4928] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): tablet 46f49fd718be4a2996d781ab31268217 (table default.Table_b5110c69d0364c50a92330c1824b8416 [id=889259ae7ac44e97aaff581d77e05009]) successfully deleted
I20250903 20:05:41.624493 27517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d13884fff8c43b690836c6a341a2d92" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"
I20250903 20:05:41.624670 27517 raft_consensus.cc:3058] T 6d13884fff8c43b690836c6a341a2d92 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Advancing to term 1
I20250903 20:05:41.627410 27517 raft_consensus.cc:2466] T 6d13884fff8c43b690836c6a341a2d92 P e72ef1dbcc5143cf9f04a808f101c80f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 1.
I20250903 20:05:41.627735 27685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56534 (request call id 1096) took 6002 ms. Trace:
I20250903 20:05:41.627875 27685 rpcz_store.cc:276] 0903 20:05:35.625189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:35.625889 (+ 700us) service_pool.cc:225] Handling call
0903 20:05:41.624479 (+5998590us) ts_tablet_manager.cc:1145] Deleting tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8
0903 20:05:41.627604 (+ 3125us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:41.627710 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5934311}
I20250903 20:05:41.628063 29760 consensus_queue.cc:1035] T 5c42d84a6a5649208b0539a10da21aec P 51578fb7bceb4e119b3ab1e58c4f5528 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250903 20:05:41.628391 27052 catalog_manager.cc:4895] TS 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): delete failed for tablet 1f6cd4d475f147fe8dfcc7e3a73dedf8 because the tablet was not found. No further retry: Not found: Tablet not found: 1f6cd4d475f147fe8dfcc7e3a73dedf8
I20250903 20:05:41.628688 27685 tablet_replica.cc:331] T 238f2f30377b42c2852b016018aa3245 P 89f8f0ee68b940db9ce250c0604f934a: stopping tablet replica
I20250903 20:05:41.628897 27685 raft_consensus.cc:2241] T 238f2f30377b42c2852b016018aa3245 P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:41.632503 27065 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f9124043f4d84586858e6bdeb679df55" } modify_external_catalogs: true
I20250903 20:05:41.632701 27065 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45936:
table { table_name: "default.Table_f9124043f4d84586858e6bdeb679df55" } modify_external_catalogs: true
I20250903 20:05:41.633170 27684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44666 (request call id 1281) took 2323 ms. Trace:
I20250903 20:05:41.633323 27684 rpcz_store.cc:276] 0903 20:05:39.309858 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 20:05:39.310411 (+ 553us) service_pool.cc:225] Handling call
0903 20:05:41.502583 (+2192172us) ts_tablet_manager.cc:1145] Deleting tablet f9375fff5f5945078c4412aa21c06f0d
0903 20:05:41.502591 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0903 20:05:41.530090 (+ 27499us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:41.585057 (+ 54967us) tablet_metadata.cc:641] Metadata flushed
0903 20:05:41.624011 (+ 38954us) spinlock_profiling.cc:243] Waited 9.1 ms on lock 0x61800000d518. stack: 00007f71bc6f7dc6 00007f71bc6f7b80 00007f71bb6df3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f71bc6e1a07 00007f71bc6e1786 00007f71cfda98ed 00007f71cfd2b1d3 00007f71cbf77723 00007f71cbfa9344 00007f71cbf8ada0 00007f71cbf8ab2d 0000000000a774dc 00007f71bc73e14f
0903 20:05:41.633145 (+ 9134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20920576,"tablet-delete.queue_time_us":2191981}
I20250903 20:05:41.633913 27684 tablet_replica.cc:331] T a686ea49a29a4dd7b6e5dce5e1f9935a P e72ef1dbcc5143cf9f04a808f101c80f: stopping tablet replica
I20250903 20:05:41.634053 27051 catalog_manager.cc:4928] TS e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): tablet f9375fff5f5945078c4412aa21c06f0d (table default.Table_98cd4cb4df914678b7bb584d97bd0ca9 [id=51d8d2e93e264e74948b83bbef771ef1]) successfully deleted
I20250903 20:05:41.634160 27684 raft_consensus.cc:2241] T a686ea49a29a4dd7b6e5dce5e1f9935a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250903 20:05:41.634342 27684 raft_consensus.cc:2270] T a686ea49a29a4dd7b6e5dce5e1f9935a P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250903 20:05:41.634514 27695 tablet_bootstrap.cc:492] T 5c42d84a6a5649208b0539a10da21aec P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
I20250903 20:05:41.635169 27684 ts_tablet_manager.cc:1905] T a686ea49a29a4dd7b6e5dce5e1f9935a P e72ef1dbcc5143cf9f04a808f101c80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 20:05:41.637264 27695 tablet_bootstrap.cc:654] T 5c42d84a6a5649208b0539a10da21aec P 89f8f0ee68b940db9ce250c0604f934a: Neither blocks nor log segments found. Creating new log.
W20250903 20:05:41.640491 27184 consensus_peers.cc:489] T 793d6157927143d4b9d8dbafd7cf0fcb P 89f8f0ee68b940db9ce250c0604f934a -> Peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Couldn't send request to peer 51578fb7bceb4e119b3ab1e58c4f5528. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 793d6157927143d4b9d8dbafd7cf0fcb. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:41.648599 27061 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "default.Table_e510da40d4f944e0be3bb38e775b22c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 20:05:41.649430 27061 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e510da40d4f944e0be3bb38e775b22c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250903 20:05:41.654027 27321 consensus_peers.cc:489] T 5c42d84a6a5649208b0539a10da21aec P 51578fb7bceb4e119b3ab1e58c4f5528 -> Peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Couldn't send request to peer 89f8f0ee68b940db9ce250c0604f934a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250903 20:05:41.658936 27695 tablet_bootstrap.cc:492] T 5c42d84a6a5649208b0539a10da21aec P 89f8f0ee68b940db9ce250c0604f934a: No bootstrap required, opened a new log
I20250903 20:05:41.659220 27695 ts_tablet_manager.cc:1397] T 5c42d84a6a5649208b0539a10da21aec P 89f8f0ee68b940db9ce250c0604f934a: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.002s
I20250903 20:05:41.659780 29452 raft_consensus.cc:491] T cb2f50683e904dd39ee6eb128717d38c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e72ef1dbcc5143cf9f04a808f101c80f)
I20250903 20:05:41.659940 29452 raft_consensus.cc:513] T cb2f50683e904dd39ee6eb128717d38c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } }
I20250903 20:05:41.660313 27695 raft_consensus.cc:357] T 5c42d84a6a5649208b0539a10da21aec P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.660549 27695 raft_consensus.cc:383] T 5c42d84a6a5649208b0539a10da21aec P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 20:05:41.660661 27695 raft_consensus.cc:738] T 5c42d84a6a5649208b0539a10da21aec P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f8f0ee68b940db9ce250c0604f934a, State: Initialized, Role: FOLLOWER
I20250903 20:05:41.660969 27695 consensus_queue.cc:260] T 5c42d84a6a5649208b0539a10da21aec P 89f8f0ee68b940db9ce250c0604f934a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
W20250903 20:05:41.661417 27184 leader_election.cc:343] T cb2f50683e904dd39ee6eb128717d38c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449): Not found: Tablet not found: cb2f50683e904dd39ee6eb128717d38c
I20250903 20:05:41.661727 27695 ts_tablet_manager.cc:1428] T 5c42d84a6a5649208b0539a10da21aec P 89f8f0ee68b940db9ce250c0604f934a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250903 20:05:41.662137 27695 tablet_bootstrap.cc:492] T 69f8eef52bee431a86f0eef45fd11f9f P 89f8f0ee68b940db9ce250c0604f934a: Bootstrap starting.
W20250903 20:05:41.662649 27183 leader_election.cc:343] T cb2f50683e904dd39ee6eb128717d38c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231): Not found: Tablet not found: cb2f50683e904dd39ee6eb128717d38c
I20250903 20:05:41.662758 27183 leader_election.cc:304] T cb2f50683e904dd39ee6eb128717d38c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a; no voters: 51578fb7bceb4e119b3ab1e58c4f5528, e72ef1dbcc5143cf9f04a808f101c80f
I20250903 20:05:41.662961 29461 raft_consensus.cc:2747] T cb2f50683e904dd39ee6eb128717d38c P 89f8f0ee68b940db9ce250c0604f934a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 20:05:41.663031 29452 leader_election.cc:290] T cb2f50683e904dd39ee6eb128717d38c P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51578fb7bceb4e119b3ab1e58c4f5528 (127.26.65.66:38449), e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231)
I20250903 20:05:41.668254 29452 raft_consensus.cc:491] T 43fb27c857d543b7b78ea6108c406085 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:41.670125 29760 raft_consensus.cc:491] T 69f8eef52bee431a86f0eef45fd11f9f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 20:05:41.670418 29760 raft_consensus.cc:513] T 69f8eef52bee431a86f0eef45fd11f9f P 51578fb7bceb4e119b3ab1e58c4f5528 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.668581 29452 raft_consensus.cc:513] T 43fb27c857d543b7b78ea6108c406085 P 89f8f0ee68b940db9ce250c0604f934a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" member_type: VOTER last_known_addr { host: "127.26.65.67" port: 46231 } } peers { permanent_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" member_type: VOTER last_known_addr { host: "127.26.65.66" port: 38449 } } peers { permanent_uuid: "89f8f0ee68b940db9ce250c0604f934a" member_type: VOTER last_known_addr { host: "127.26.65.65" port: 42989 } }
I20250903 20:05:41.671743 27239 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f8eef52bee431a86f0eef45fd11f9f" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89f8f0ee68b940db9ce250c0604f934a" is_pre_election: true
I20250903 20:05:41.672056 27509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "43fb27c857d543b7b78ea6108c406085" candidate_uuid: "89f8f0ee68b940db9ce250c0604f934a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f" is_pre_election: true
I20250903 20:05:41.672263 27509 raft_consensus.cc:2466] T 43fb27c857d543b7b78ea6108c406085 P e72ef1dbcc5143cf9f04a808f101c80f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f8f0ee68b940db9ce250c0604f934a in term 0.
W20250903 20:05:41.672245 27321 leader_election.cc:343] T 69f8eef52bee431a86f0eef45fd11f9f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989): Illegal state: must be running to vote when last-logged opid is not known
I20250903 20:05:41.672403 29760 leader_election.cc:290] T 69f8eef52bee431a86f0eef45fd11f9f P 51578fb7bceb4e119b3ab1e58c4f5528 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72ef1dbcc5143cf9f04a808f101c80f (127.26.65.67:46231), 89f8f0ee68b940db9ce250c0604f934a (127.26.65.65:42989)
I20250903 20:05:41.672647 27183 leader_election.cc:304] T 43fb27c857d543b7b78ea6108c406085 P 89f8f0ee68b940db9ce250c0604f934a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89f8f0ee68b940db9ce250c0604f934a, e72ef1dbcc5143cf9f04a808f101c80f; no voters:
I20250903 20:05:41.673100 27509 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f8eef52bee431a86f0eef45fd11f9f" candidate_uuid: "51578fb7bceb4e119b3ab1e58c4f5528" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72ef1dbcc5143cf9f04a808f101c80f"